Meeting_2006-05-29

• Date: 29.05.2006
• Time: 09.30 Norw. time
• Place: Where we are
• Tools: SubEthaEdit, iChat

1. Opening, agenda review
2. Reviewing the task list from two weeks ago
3. Documentation - divvun.no
4. Corpus gathering
5. Corpus infrastructure
6. Infrastructure
7. Linguistics
8. name lexicon infrastructure
9. Spellers
10. Other issues
11. Summary, task lists
12. Closing

Opened at 09: 37.

Present: Børre, Saara, Sjur, Thomas, Trond, Tomi

Absent: Maaren

Main secretary: the whole concept dropped for now - working collaboratively makes it obsolete.

Agenda accepted as is.

Børre

• corpus collection:
  • send out contracts with accompanying letter
  • Gather public texts, preferrably also parallel ones
    • Began cron job on http: //galdu.org/samegillii
  • Continue converting text from input format to our xml
    • Done by convert2xml.pl, updated some .xsl files by hand
  • convert nob and nno bible texts to be used as part of a parallel corpus
    • Not done
  • review the paratext2xml converter
    • Not done
  • convert smj NT to paratext
    • Not done
  • Send out letters to the rest of the Iđut authors
    • Contact with Kurt Tore Andersen, writer of schoolbooks. Was positive.
  • call Brita Kåven again towards the end of the week
    • Not done
  • contact Ája (Kåfjord)
    • Not done
  • create weekly cron job to mirror Odin URL and detect new/updated pages
    • Not done
  • Check the status & license of the corpus texts
    • Done
  • wait for R. Valkeapää, call him next week
• public tender:
  • assist with letter to the project board
    • Read through the letter, participated in discussions
• corpus access:
  • zip an xml-stripped version of our free texts for to Olavi
    • Done!
  • meeting 23.5 at 9.30: discuss and decide upon the exact access policy we want
    • Not done
  • set up user account/corpus access for Olavi
    • Not needed?
  • set upp the unix group structure for corpus users (also Saara, Trond)
    • To be done after the above-mentioned meeting
• install latest SEE
  • Done
• install Gobby for Thomas and Maaren
  • Not done
• fix bugs!

Maaren

• On sick leave

Saara

• Create a parallel corpora of the new testaments.
• add more texts to the graphical corpus interface
• grammatical searchability in the graphical corpus interface
  • these are not done
• Implement links to parallel files in corpus header.
• Implement parallel corpus upload in web upload script
  • not done
• Check the status & license of the corpus texts and change the default of the availability to "license" instead of "free"
  • done
• rerun the corpus conversion
  • done on daily basis
• Install Gobby
  • can I have the tarball discussed in the last meeting?
• update the corpus script(s) to only copy texts to the free/ dir which are explicitly marked as free
  • done, some testing required (and cleaning the free-directory)
• Add some language recognition flags to write into the xsl file
  • not yet ready
• rename corpus dirs, and create symlinks
  • done, there is an unused directory gt which should be removed.
• set upp the unix group structure for corpus users (also Børre, Trond)
  • not done
• fix bugs!

Sjur

• public tender:
  • write letter to the project board
    • done
• fst gymnastics to add hyphenation and word boundary marks to hyphenation transducer
  • not yet done
• name lexicon:
  • implement editing functions
  • finalise refactoring for multiple collections (regular search interface)
    • nothing this week
• change corpus-summary processing to generate smaller pages
  • nothing yet - needs to be discussed
• send bug report to Apple re filename matching and accented characters in Terminal
  • done
• meeting 23.5 at 9.30: discuss and decide upon the exact access policy we want
  • delayed
• fix bugs!

Thomas

• correct hyphenation of exceptions (sme)
  • still working with this
• work on compounding and derivation
  • planned continues work on work-shop with Trond and Maaren
• sme G3 issue
  • nothing done
• set up a linguistic workshop while Maaren is in Tromsø
  • done

Tomi

• new proper name lexicon
  • implement data synchronisation of proper nouns between risten.no and CVS
    • not done
  • XQuery refactoring and code development for our proper noun editor
    • not done
  • new version of xml2lexc (based on ccat), should handle complex names correct: construct entries like we have now from the different parts of a complex name entry
    • not done
• read aligner docu, install, provide feedback
  • not done
• install and test Gobby
  • not done
• Set up the mechanism for the hash-mark transducer package
  • not done
• add ccat option to analyse text while keeping the xml tags and structure
  • not done
• fix bugs!

Trond

• better smj NT text
• get fin, swe, nob and nno NT and OT in paratext format
  • Not done.
• install aligner, test it and give feedback
  • Discussed issue with Lars, will continue with Saara
• fst gymnastics to add hyphenation and word boundary marks to hyphenation transducer
  • Not done
• get/upgrade keys for Børre's room for Tomi and Thomas Ordered, still not received answer.
• Rethink the doubletagging procedure for names, consider grammatically motivated semtag conversion routines ("Helsinki" from Plc to Obj to Org)
  • No news.
• Check the status & license of the corpus texts
  • Done.
• Work on the graphical corpus tag list
  • Made progress.
• send Saara smj files for language recognition
  • Talked to saara
• create a short smj word list to help the trigram heuristics
• Put Saara and Tero in contact with each other
  • Not yet.
• ask Lars Nygård and Tero Avellan to install Gobby 0.3
• set up a linguistic workshop while Maaren is in Tromsø
  • done
• meeting 23.5 at 9.30: discuss and decide upon the exact access policy we want
  • delayed
• set upp the unix group structure for corpus users (also Børre, Saara)
  • delayed until the principles are settled
• fix bugs!.

TODO:

• documentation on how to apply for a user account for the corpus repo ( Børre)
  • we will administer the corpus user accounts ourselves
  • We first have to discuss and decide what we want before Børre can write documentation (see further down)
    • Not done. We didn't have the meeting.

Collecting

See a previous meeting memo for what's to be done.

TODO:

• Send out the rest of the letters (Børre)
  • Working on this. Summing up the most important writers from Davvi Girji.

New contracts:

• none last 2 weeks

Olavi Korhonen's Lule Sámi dictionary.

Phoned Korhonen. He was willing to sign the contracts, and wanted some kind of access to our corpus. He wants to collect words for his Northern Sámi dictionary.

TODO:

• set up user account/corpus access for Olavi (Børre)
  • we need the infrastructure for corpus access in place first
• done: sent a Word document containing all free texts (2400 pages), as well as a list of all words (baseform)

KIO Grafisk and the Iđut books

TODO:

• send letters to the authors (Børre)
  • talked to Kurt Tore Andersen, who has written a lot of smj school books. He is interested in cooperation with us.

Bible texts

We will get text from Finland, but still haven't received any. Swedish html has arrived, no paratext. Norsk bibelselskap has not sent corrected New Testament versions for sme, and not paratext for nno/nob.

TODO:

• convert smj NT to paratext (Børre)
• get fin, swe, nob and nno NT and OT in paratext format. (Trond)

Davvi Girji

Called her last week. She said Davvi Girji os would give us permission to use texts by their authors. She wasn't sure if we could get the texts directly from Davvi Girji, because of copyrights of pictures and other artwork in the books. Said they (Davvi Girji workers) were going to have a meeting right after our conversation by phone, and take some kind of desicion on this case. Haven't heard anything from Kåven, though.

TODO:

• call Brita Kåven again towards the end of the week (Børre)
• call the authors (Børre)

Min Áigi

The Min Áigi format should be dealt with: \@ingress etc should be dealt with for the .txt, but business as usal for the .doc files. Saara has done the xsl conversion routine for the typographic tags. It still needs some fine tuning, as there are some @ tags that were not included in our initial list.

TODO:

• send bug report to Apple (typing filenames in Terminal does not match, moving filenames across plattforms via tarballs creates problems - the names are not the same!) (Sjur)
  • not done yet
• will have to complete metainformation

Kåfjord

Promised to send us texts. Some texts have arrived, but nothing from Ája.

TODO:

• contact Ája (Børre)
  • not done

Sámi Instituhtta

Børre contacted Richard Valkeapää, the IT-consult at NSI. He put it on his todo list, as he would have to contact the person who has worked with the newspaper texts anyway. He said this would be done in the near future (within a month).

TODO:

• wait for R. Valkeapää, call him this week (Børre)

User accounts and access

We will probably have different kinds of users, some will only need access through the webinterface to the corpus, others might want and need access to the commandline, to utilize the corpus in an efficient way. This calls for an access policy for these users.

TODO:

• before anything else is done: Zip an xml-stripped version of our free texts and send them to Olavi. In that way, he gets something to work on, and we will get feedback (Børre)
  • done
• discuss and decide upon the exact access policy we want to give corpus users; meeting on Tuesday May 23, at 9.30 (Børre, Sjur, Trond)
  • not done, new date: Thursday 1.6.2006, at 9.30
• set upp the unix group structure to open for a new category of users: Linguists with access to the texts on the servers. Some of Børre, Saara, Trond.
• make a text-only corpus in the Oslo interface (dump the texts on omilia), thereby making them accessible to simple lexical lookup. The cost of doing this should be evaluated against what it takes waiting for the analysed corpus to be in place. (Saara, Trond)
  • Discussion on this issue after this meeting

Name change again?

TODO:

• rename corpus dirs, and create symlinks (Saara)
  • done, remove the gt dir? Yes.

Free and non-free texts

More info in a previous meeting memo.

TODO:

• Check the status of the texts, again. (Børre, Trond, Saara)
  • Complete (only new files need to be checked from now on)
• Rerun the conversion afterwards (Saara is the one with the magic spell)
  • done automatically every night
• update the script(s) to only copy texts to the free/ dir which are explicitly marked as free (Saara)
  • done, but buggy

More texts to the graphical corpus interface:

We need to get the infrastructure complete to be able to do this, then it should be a piece of cake.

TODO:

• We would like to have more than the NT in the graphical interface ( Saara)
  • We add the largest texts first.
• We would like to have grammatical searchability, not only POS. (Saara, Trond This presupposes a discussion with Oslo. (Trond and Saara to continue this discussion)
• For Lule Sámi: We would like to have a parallel corpus interface with NT (text only). This presupposes better quality texts (Trond, Børre)
  • Better Lule NT text still not made.
• The list of good candicates: The longest (admin) texts.
  • We need a new option in ccat for analysing text while still keeping the xml tags and structure (Tomi)
  • xml texts number <p>, preprocess finds <.> <?> <!> and ccat numbers them as <s>.
  • Then the aligner aligns...

Top-three priorities:

1. Finish the tag unification (korpustags.txt) (Trond)
2. change ccat to be able to create the right input for the corpus analysis (xml- tagged output) (Tomi). Work estimate: a few (2-3?) days - nothing this week, we will re-evaluate (and schedule) next Monday)
3. add text to the server (Lars)

Language recognition

TODO:

• refine language recognition (Saara)
  • in progress, continue discussion in 249
• create a short smj word list to help the trigram heuristics (Trond)
• send Saara smj files (Trond)
  • Call Árran again (Trond, then handle it over to Sámediggi, that is, to Børre)
• Add some flag to write into the xsl file (Saara)
  • unfinished

Some Lule Sami text is found on Infonuorras site.

Corpus summary

TODO:

• trim generated pages (now sme generates a table with 10 000 entries!)
  • suggestion: lump together files with content less than X paragraphs (X < 5?)

Proofed vs unproofed corpus files

The Min Áigi material contains partially parallel unproofed vs proofed documents. We need to find a decent way of handling this parallellism, and preferably a way to make use of the information contained in the diff between the two versions. At best, we should be able to automatically extract the corrections made, generating an XML document that contains correction markup as discussed in the newsthread "Corpus DTD: corrections" (from 17.03.2006). I don't know how much work this is, or whether it is at all possible, but let's discuss it.

For now there are so few such files that it hardly pays off, but Saara will make a list of these files to evaluate the potential benefit.

TODO:

• count the number of parallel files of type unproofed/proofread (Saara)

Aligner

Trond and Saara will continue this issue.

Paradigm generation

Goal: Reuse Greenlandic code for paradigm generation.

TODO:

• Put Saara and Tero in contact with each other (Trond)
  • still open
• The paradigm generator should also have an xml-out option (for use in e.g. electronic dictionaries) - on hold till we know more about the generator

Hyphenator

Thomas is finished with adding ^ tags to the smj noun file, and has continued working on the sme noun file.

Trond and Thomas have been working on the smj rule component, and have improved both the treatment of weak grade consonant clusters (preconsonantal geminates) and on some loan word patterns.

TODO:

• correct the treatment of hyphenation of word boundaries and exceptions (fst gymnastics) (Sjur, Trond)
  • Still not done.
• Update the sme and sma rule sets with the insights gained from smj updates.

General issues

Rethink the doubletagging procedure for names, consider grammatically motivated semtag conversion routines ("Helsinki" from Plc to Obj to Org, or the Lyndi England issue, Aftenposten Obj Org WoA Pub)

Possible rule:

If Plc then Obj

North Sámi

TODO:

• set up a linguistic workshop while Maaren is in Tromsø (and remember that she is on sick leave all the time) (Trond, Thomas, Maaren)
  • done, more needed:
• Maaren is in Tromsø this week, we should continue our linguistic discussions.

Lule Sámi

There are some open issues in the marginal area of the smj transducer:

• numerals, e.g. Our poor treatment of number words becomes more visible as we get real texts (guhttatuvsánanielljatjuodegålmmålåkvihtta is not recognised, etc.).
• names => waiting for the new name lexicon
• compounds? Shortening here as well, but not in written language (some lexicalised exceptions, as in sme) => nothing to discuss/implement
• loanwords? We should consider importing the ^LOAN words from sme and Lule-ify them. Otherwise, loan words should come from corpora.

TODO:

• 50 unknown words left+2 abbr. +moaddi etc (numerals) need more checks
• add proper numeral analysis/treatment
• add loanwords (e.g. latin -ere verbs)

TODO:

1. finish refactoring for multiple collections in the search interfarce ( Sjur)
   1. nothing done last week
2. develop the needed XQueries and interface (Sjur, Tomi)
   1. progressing, done some, haven't commited (adding new term, create-termc-entry.xq)
3. data synchronisation between risten.no and the cvs repo (Tomi)
   1. discussion started on eXist-list, we'll wait a couple of days to see what's coming out of it, and if nothing useful to us, we'll add our use case with questions
4. test and review when ready

Timeline:

• One-time conversion lexc2xml (sme to common)
  • done (must be redone at D-day, but the *script* is done)
• editing functions in risten.no - what about editing in emacs/see/other editors
  • in the works
• Automatic converson: xml2lexc (modulo language) (based on ccat) (ready to be executed) (Tomi)
  • not done

We will remove this speller section till we have something to report.

Finnut called, and here's their evaluation: if we think that the offers are incomplete or otherwise not fully acceptable, we can enter negotiations with the companies, effectively cancelling the current public tender. This can only be done as long as we don't change the public tender document (that is, the foundation for the public tender) - if it is changed, we have to announce the whole competition again, with the usual 53 days minimum deadline for applications.

Sjur will send an e-mail to the project board, outlining the different aspects of the two offers, and ask about their opinion on the following questions:

• accept the offers as is, or enter negotiations?
• if accept, which one?

TODO:

• write e-mail to the project board, ask for their opinion regarding the offers ( Sjur)
  • done, waiting for answers from the board

Summer vacation

Bug fixing

43 open Divvun/Disamb bugs (two down!), and 25 risten.no bugs

Guess: 1/3 of the bugs are fixed already (?)

Please help Saara with bug 279 (Perl locale). Not much help... Saara will contact Roy on this issue.

After the corpus issues have been somewhat settled, we should do a bug barnraising. ... and then a new one after the name lexicon is fixed.

Gobby

0.3 is working fine on Mac, Linux and Windows. Should be installed on all computers c.f. http://darcs.0x539.de/trac/obby/cgi-bin/trac.cgi/wiki/InstallationGuide (our preinstalled Xcode veriosn is 2.0, must be 2.1):

• Børre - ok by copying /opt/local/ from Trond
• Maaren - Børre to do it
• Saara - todo
• Sjur - ok
• Thomas - Børre to do it
• Tomi - not working
• Trond - ok

Easy way out when the standard Darwin Ports installation isn't working: just get a copy of /opt/local/ from Børre.

Trond should ask Lars Nygård and Tero Avellan to install Gobby as well; has asked Per Langgård.

SEE 2.5 extensions

• syntax coloring of meeting memos
• script to extract tasks?

TODO:

• give the jspwiki syntax colouring mode to all (Sjur)

06.06.2006 09: 30

Closed at 11: 20

Børre

• corpus collection:
  • send out contracts with accompanying letter
  • Gather public texts, preferrably also parallel ones
  • Continue converting text from input format to our xml
  • convert nob and nno bible texts to be used as part of a parallel corpus
  • review the paratext2xml converter
  • convert smj NT to paratext
  • Send out letters to the rest of the Iđut authors
  • call Brita Kåven again towards the end of the week
  • contact Ája (Kåfjord)
  • create weekly cron job to mirror Odin URL and detect new/updated pages
  • wait for R. Valkeapää, call him this week
• public tender:
• corpus access:
  • meeting 23.5. t 9.30: discuss and decide upon the exact access policy we want
  • set upp the unix group structure for corpus users (also Saara, Trond)
• install Gobby for Thomas and Maaren, send /opt/local/ tarball to Saara
• fix bugs!

Maaren

• On sick leave

Saara

• Create a parallel corpora of the new testaments.
• add more texts to the graphical corpus interface
• grammatical searchability in the graphical corpus interface
• Implement links to parallel files in corpus header.
• Implement parallel corpus upload in web upload script
• Install Gobby
• clean the free/ dir and test copying of only free texts to it
• Add some language recognition flags to write into the xsl file
• set upp the unix group structure for corpus users (also Børre, Trond)
• count and examine pairs of proofread and nonproofed MinAigi files
• evaluate addition of a text-only corpus in the Oslo interface (with Trond)
• fix bugs!

Sjur

• public tender:
  • waiting for answers from the board, then act
• fst gymnastics to add hyphenation and word boundary marks to hyphenation transducer
• name lexicon:
  • implement editing functions
  • finalise refactoring for multiple collections (regular search interface)
• change corpus-summary processing to generate smaller pages
• send bug report to Apple re filename matching and accented characters in Terminal
• meeting 23.5. t 9.30: discuss and decide upon the exact access policy we want
• fix bugs!

Thomas

• correct hyphenation of exceptions (sme)
• work on compounding and derivation
• sme G3 issue

Tomi

• new proper name lexicon
  • data synchronisation of proper nouns between risten.no and CVS
  • XQuery refactoring and code development for our proper noun editor
  • new version of xml2lexc (based on ccat), should handle complex names correct: construct entries like we have now from the different parts of a complex name entry
• read aligner docu, install, provide feedback
• install and test Gobby
• Set up the mechanism for the hash-mark transducer package
• add ccat option to analyse text while keeping the xml tags and structure
• fix bugs!

Trond

• better smj NT text
• get fin, swe, nob and nno NT and OT in paratext format
• install aligner, test it and give feedback
• fst gymnastics to add hyphenation and word boundary marks to hyphenation transducer
• get/upgrade keys for Børre's room for Tomi and Thomas
• Rethink the doubletagging procedure for names, consider grammatically motivated semtag conversion routines ("Helsinki" from Plc to Obj to Org)
• Work on the graphical corpus tag list
• send Saara smj files for language recognition
• create a short smj word list to help the trigram heuristics
• Call Árran again, and then transfer smj corpus discussions to Børre
• Put Saara and Tero in contact with each other
• ask Lars Nygård and Tero Avellan to install Gobby 0.3
• continue linguistic discussions while Maaren is in Tromsø
• meeting 23.5. t 9.30: discuss and decide upon the exact access policy we want
• set upp the unix group structure for corpus users (also Børre, Saara)
• evaluate addition of a text-only corpus in the Oslo interface (with Saara)
• fix bugs!.