Meeting_2006-01-30

Meeting setup

Date: 30.01.2006
Time: 09.30 Norw. time
Place: Wherever we are : -)
Tools: iChat, SubEthaEdit

Agenda

Opening, agenda review
Reviewing the task list from two weeks ago
Documentation - divvun.no
Corpus gathering
Corpus infrastructure
Linguistics
name lexicon infrastructure
Other issues
Summary, task lists
Closing

1. Opening, agenda review, participants

Opened at 09: 48.

Present: Børre, Sjur, Tomi, Trond

Absent: Maaren, Saara, Thomas

Main secretary: Børre

Agenda accepted as is.

2. Reviewing the task list from the last meeting

Børre

send out contracts with accompanying letter
- Not done
Gather public texts, preferrably also parallel ones
- Not done
Contact Odin editor (Ove Sæth) to ask for source (and parallel) documents
- Done
Continue converting text from input format to our xml
- Not done
review code and documentation for corpus xsl files under version control
- Not done
fix bugs!
- Not done

Maaren

work with risten.no
discuss with relevant people regarding seminar on proofing tools, normativity and SGL in February/March, including place.

Saara

Convert the name lexicon from present format to xml for testing; final conversion pending the editing fascilities
- done
Refine language detection for Finnish
- not done
Finish the review of the hyphenation detection.
- not done
Review the handling of xsl-files in corpus infrastructure, including version control
- in progress
Do some testing for bug #211
- not done
optimize the preprocess script
- not done
Write/update user documentation for the corpus usage in preparation for the review in the project meetings next week.
- done
finalize an improved working version of the CGI and command line scripts for corpus additions
- in progress
xml2lexc update to handle complex names: construct entries like we have now from the different parts of a complex name entry
- tomi will do the xml2lexc-script(?)
update conversion from lexc to xml (proper names) with the latest refinements
- not done
fix bugs!

Sjur

Follow up the lawyer treatment of the contracts
- not done
Project seminar
- plan and make schedule with Trond
  - done
- check which hotels SD has an agreement with
  - done
- plan XQuery/XSLT training session
  - done
  - the whole seminar is done
Lule Sámi twol problems, with Thomas and Trond
- not done
follow up on voice group-chat not working to Sámediggi
- Test Marratech when the new Marratech server is in place
  - not done
project planning with Trond, continued
- also look at the development processes - specification and testing
  - not done
Follow up on place names from Norge Digitalt
- write an e-mail to or call Bjørn Olav Megard
  - not done
Evaluate SFST as speller (and analyzer) lexicon
- more thorough analysis than was possible in Guovdageaidnu
  - not done
write a background document on the corpus contracts
- not done
continue proper name lexicon work and discussion
- done a lot at the seminar
public tender:
- review offer from Finnut Consult AS
  - done, as well as asked for two other offers, then picked one (Finnut), and had a meeting with them, planning the whole PT process
smj G3 issue with Thomas and Trond
- not done
sme G3 issue with Thomas and Trond
- not done
call EDD/ Christian Emil Ore about national place name lexicon
- not done
risten.no/name lexicon development: fix bugs, continue development
- done some, backed up and restarted the server; the backup isn't completely valid, due to some strange bug/conflict between the version on the server and the version of the client. The XML files are ok though.
fix bugs!

Thomas

On sick leave

Tomi

Aspell: Continue working on the affix file & aspell
- Contact aspell author (UTF-8 thing)
  - Not done
corpus infrastructure:
- dtd location (both public and internal)
  - Not done
- cgi-admin script for adding xsl-files
  - Not done
Document aspell and corpus infrastructure
- Not done
Specification for new catxml in C++
- install and announce new ccat tool
  - Done
new proper name lexicon
- remove last part of complex names not used as simplex names
  - Not done
- start looking at conversion of the name lexicon from present format to xml
  - Not done
- discuss the new lexicon format in the newsgroup
  - Not done
- Look into synchronisation of proper names with risten.no
  - Not done
- meeting to arrive at final xml format
  - Participated the meeting
- new version of xml2lexc (based on catxml, now ccat)
  - Not done
hyphenation in corpus docs
- Not done, but has Saara done it? Sjur: yes.
comment review template made by Saara
- Not done
fix bugs!
- Not done
pick up backpacks after Xmas
- Done

Trond

Contact Odin editor (Ove Sæth) immediately to reopen contacts
- Done.
Project seminar
- plan and make schedule with Sjur
- Done.
- check with Linda and Ilona whether we can start on Monday after lunch
- Done.
sign contract with Bibelselskapet for Norwegian parallel texts
- Done.
document corpus infrastructure, your part
review corpus usage documentation (ccat)
- Done.
discuss the new lexicon format in the newsgroup
- Done, but not extensively.
smj G3 issue with Sjur and Thomas
- Not done
sme G3 issue with Sjur and Thomas
- Not done
fix bugs!
- I think we made some progress on some of the number bugs, but we haven't looked at the bugzilla base yet.

3. Documentation

Reviews

ccat review

Saara, Linda, Ilona, Trond

Conducted the review as part of the seminar, although Thomas and Maaren weren't there. Outcome:

It works mostly as documented, a few glitches were found and corrected. The documentation is terse but concise (simple is beautiful, that is, in full accordance to KISS). ccat -h gives:

        -a              Print all text elements.
        -p              Print plain paragraphs. (default)
        -T              Print paragraphs with title type.
        -L              Print paragraphs with list type.
        -t              Print paragraphs with table type.
        -r <dir>        Recursively process directory dir and subdirs enountered.
        -h              Print this help message.

(Trond: ) I thought that -T printed titles, but it gave the same output as -a.
(Tomi: ) No, it gave you all regular paragraphs + titles, but no tables or lists. The latest version will give you only titles, lists, etc according to the specified option.
(Sjur: ) I would like a -v option, so we are able to identify which version to send a bug report for : -) The version number should be autoincremented on each CVS check-in/build, or manually set?

The [xml-based documentation|/ling/catxml.html] was not completely up-to-date with the latest changes, fixed in the meeting.

4. Corpus gathering

Discussed briefly how the formalities should be implemented: signature, Websak, posting etc.

Collecting

See the previous meeting memo for what's to be done.

TODO: Still a lot for Børre!

Odin

DONE: Trond, and then Børre to call Ove Sæth to re-establish contact.

Sæth to discuss with colleagues about how to implement the cooperation.

Bible texts

ccat -t zcorp/gt/sme/bible/ot/1Mos_09-01.doc.xml | less

This gives everything. What we want is to make a file-specific version of testament.xml, with these properties:

The first column should be suppressed
The second column should be marked number or something
The third column should be marked header if the typographic code in the fist column indicates it is a header (1 or 3) and it should be marked text if the typographic code indicates it is text (is 5).

sme$ls zcorp/orig/sme/bible/ot/
1Mos_09-01.doc  Salmmat-_garvasat_0203.doc

There are two new books in the paratext format waiting in the nob orig and nno orig directories.

TODO:

write a paratext2xml converter
convert smj NT to paratext
ask to get fin and swe NT and OT in paratext format

We already have an embryonic converter: gt/script/testament.xsl Usage: xsltproc /path/to/testament.xsl bible-text.xml > converted-text.xml format.

5. Corpus infrastructure

Task list:

Include the xsl files under version control
1. RCS version control is almost finished, but an issue with access control is still open. Discussed a bit in the meeting, but nothing conclusive. We'll continue the discussion in the newsgroup.
Incorporate language detection as part of the corpus processing (Saara)
1. Almost finished. Needs improved Finnish language model - presently it isn't able to distinguish Finnish from Sámi (proving the family bonds: -)
we need to review whether only automatic hyphen detection is good enough, or whether manual post-processing in some form is needed. Delayed until we have some results to base the review on.
1. Acceptable results: 90% of all real hyphens correctly tagged.
CGI-admin script to add xsl-file to a corpus file that doesn't have one ( Saara)

Things are moving forward, but still more work to do. The list is left as is.

6. Linguistics

Nothing today, our linguists are on sick leave or not participating. For the tasks and their status, see the previous meeting memo.

7. Name lexicon infrastructure

Complex names

Task list for this issue:

make sure xml2lexc can handle complex names in ways compatible with our present tool chain (=reconstruct the lexc format we have now) (Tomi)
- the resulting file format should be identical to our present prop-name file (=lexc), that can then be converted to our new xml format using the same script as for the regular names (Tomi or Saara, but only when the technical details are settled)
Saara has added the analyzer as part of the preprocess, but it is slow, and needs to be optimized.

XML format

Tasks:

make a test lexicon for evaluating the format, set up the editing, and test it ( Saara)
1. Done
update conversion from lexc to xml to reflect new xml format (Saara)
1. mostly done, some open questions left
testing of conversion
eXist as editor:
1. develop the needed XQueries and interface
2. synchronisation between risten.no and
3. test whether eXist as editor is actually working well

8. Other

SGL Seminar

SGL/normativity seminar
- all members = potentially/likely all languages
  - not all languages, only North Sámi
- date? As early as possible, end of February/beginning of March
- place? Maaren will investigate

Technical issues

The mac os / perl bug (at least Trond and Sjur has it, Bugzilla #211):
- utf8 "\xC4" does not map to Unicode at /Users/trond/gt/script/preprocess line 82. This msg did not show up in 10.3 (perl 5.8.1), but does so in 10.4 (perl 5.8.6). It is probably a perl - OS mismatch. (Trond, Saara, Tomi)
  - 10.4 introduced support for locales in the shell (10.3 and earlier didn't know about locales)
- Test: the result of the last line should indicate whether this is a problem in cat or a Perl/OS mismatch.
- Is this a problem with ccat?
  - It doesn't seem so (3 min and still counting)
  - In the end, the bug turned up with ccat as well. I gave the command:
  - zcorp/gt/sme/*/*xml

preprocess --abbr=bin/abbr.txt

lookup -flags

mbTT -utf8 bin/sme.fst

lookup2cg

vislcg --grammar=src/sme-dis.rle

--minimal

sort

less

1729 constraint rules
utf8 "\xA1" does not map to Unicode at /home/trond/gt/script/preprocess line 109, <> chunk 12.

To ccat's defence I must say that cat, in a similar situation, would have given far more error messages (hold on, testing still under way).

   preprocess file_name.txt - OK
   cat file_name.txt | preprocess - bug!!
   catxml file_name.xml | preprocess - ??
   ccat filename | preprocess - bug !!

This bug isn't a high priority any more, because ccat behaves differently than cat, and because there is the possibility of avoiding cat when working locally.

BUG: close as Won't fix.

Bug fixing

30 open bugs (and 25 risten.no bugs)

Norwegian ispell press release

The i18n section of Skolelinux plans a press release including a paragraph about our project. We will ask them to reformulate a couple of things, and remove the links they've included. Text submitted to them:

Dette er den eneste kilden til retteprogram for norsk uavhengig av Microsoft, bl.a. det som brukes av Apple.

Et separat prosjekt ved Sametinget er i gang for å utvikle samiske retteprogram, bl a for Linux. Se http: //www.divvun.no for mer info.

9. Summary, task list