Meeting_2006-06-06

• Date: 06.06.2006
• Time: 09.30 Norw. time
• Place: Where we are
• Tools: SubEthaEdit, iChat

1. Opening, agenda review
2. Reviewing the task list from two weeks ago
3. Documentation - divvun.no
4. Corpus gathering
5. Corpus infrastructure
6. Infrastructure
7. Linguistics
8. name lexicon infrastructure
9. Spellers
10. Other issues
11. Summary, task lists
12. Closing

Opened at 09: 37.

Present: Børre, Saara, Sjur, Thomas, Trond, Tomi

Absent: Maaren, Saara leaving 10.30.

Agenda accepted as is, some additions under Other.

Børre

• corpus collection:
  • send out contracts with accompanying letter
    • Harald Gaski and Lill-Tove Fredriksen
  • Gather public texts, preferrably also parallel ones
    • Done
  • Continue converting text from input format to our xml
    • Updating meta data in MinAigi, using Saaras script
  • convert nob and nno bible texts to be used as part of a parallel corpus
    • Not done
  • review the paratext2xml converter
    • Not done
  • convert smj NT to paratext
    • Not done
  • Send out letters to the rest of the Iđut authors
    • Not done
  • call Brita Kåven again towards the end of the week
    • Not done
  • contact Ája (Kåfjord)
    • Not done
  • create weekly cron job to mirror Odin URL and detect new/updated pages
    • Done
  • wait for R. Valkeapää, call him this week
    • He needed some renaming script, said I would send him some of ours.
• public tender:
• corpus access:
  • meeting 23.5. t 9.30: discuss and decide upon the exact access policy we want
    • Not done
  • set upp the unix group structure for corpus users (also Saara, Trond)
    • Not done
• install Gobby for Thomas and Maaren, send /opt/local/ tarball to Saara
  • Done
• fix bugs!

Maaren

• On sick leave

Saara

• Create a parallel corpora of the new testaments.
• add more texts to the graphical corpus interface
• grammatical searchability in the graphical corpus interface
• Implement links to parallel files in corpus header.
  • sent a note to the newsgroup.
• Implement parallel corpus upload in web upload script
  • not done
• Install Gobby
  • not done
• clean the free/ dir and test copying of only free texts to it
  • done
• Add some language recognition flags to write into the xsl file
  • done
• set up the unix group structure for corpus users (also Børre, Trond)
• count and examine pairs of proofread and nonproofed MinAigi files
  • done
• evaluate addition of a text-only corpus in the Oslo interface (with Trond)
  • done
• improve add-hyph-tags.pl to handle XML-structures better.
  • done
• Updated decode.pm to handle msword titles.
  • almost ready
• convert nob and nno bible texts to be used as part of a parallel corpus
  • We don't have parallel text markup in corpus.dtd.
• fix bugs!

Sjur

• public tender:
  • waiting for answers from the board, then act
    • received answers from all, as well as a clarifying e-mail from Finnut; sent a reply and comment based on the Finnut letter, suggesting that we continue to ask for clarification before we pick one of the offers as received; also checked the monetary situation for the whole project to guide the tender process.
• fst gymnastics to add hyphenation and word boundary marks to hyphenation transducer
• name lexicon:
  • implement editing functions
  • finalise refactoring for multiple collections (regular search interface)
    • progress - all look-up functions are now multi-collection aware
• change corpus-summary processing to generate smaller pages
  • not done
• send bug report to Apple re filename matching and accented characters in Terminal
  • done
• meeting 23.5. t 9.30: discuss and decide upon the exact access policy we want
• give copies of the SEE JSPWiki mode to others
  • checked in to CVS, it can be installed from there with a double-click
• fix bugs!

Thomas

• correct hyphenation of exceptions (sme)
  • done
• work on compounding and derivation
  • nothing done this week
• sme G3 issue
  • nothing done this week

Tomi

• new proper name lexicon
  • data synchronisation of proper nouns between risten.no and CVS
  • XQuery refactoring and code development for our proper noun editor
  • new version of xml2lexc (based on ccat), should handle complex names correct: construct entries like we have now from the different parts of a complex name entry
• read aligner docu, install, provide feedback
• install and test Gobby
• Set up the mechanism for the hash-mark transducer package
• add ccat option to analyse text while keeping the xml tags and structure
• fix bugs!

Trond

• better smj NT text
  • Not done
• get fin, swe, nob and nno NT and OT in paratext format
  • Not done
• install aligner, test it and give feedback
  • Not done
• fst gymnastics to add hyphenation and word boundary marks to hyphenation transducer
  • Not done
• get/upgrade keys for Børre's room for Tomi and Thomas
  • No response from key office, the key person has been away
• Rethink the doubletagging procedure for names, consider grammatically motivated semtag conversion routines ("Helsinki" from Plc to Obj to Org)
  • The rethinking is done, implementation is needed.
• Work on the graphical corpus tag list
  • Done some, but awaiting Tomi's script to get this issue going.
• send Saara smj files for language recognition
  • No new smj files.
• create a short smj word list to help the trigram heuristics
  • This was done earlier.
• Call Árran again, and then transfer smj corpus discussions to Børre
  • Worked on this issue
• Put Saara and Tero in contact with each other
  • Awaiting the dust to be settled after his new employment.
• ask Lars Nygård and Tero Avellan to install Gobby 0.3
  • Issue open.
• continue linguistic discussions while Maaren is in Tromsø
  • Had one session.
• meeting 23.5. t 9.30: discuss and decide upon the exact access policy we want
  • Not done
• set upp the unix group structure for corpus users (also Børre, Saara)
  • Not done
• evaluate addition of a text-only corpus in the Oslo interface (with Saara)
  • We still go for the "everything first" strategy
• fix bugs!.
  • Worked on bugs.

TODO:

• documentation on how to apply for a user account for the corpus repo ( Børre)
  • we will administer the corpus user accounts ourselves
  • We first have to discuss and decide what we want before Børre can write documentation (see further down)

Collecting

See a previous meeting memo for what's to be done.

TODO:

• Send out the rest of the letters (Børre)

New contracts:

• none last 2 weeks

Olavi Korhonen's Lule Sámi dictionary.

TODO:

• set up user account/corpus access for Olavi (Børre)
  • we need the infrastructure for corpus access in place first

KIO Grafisk and the Iđut books

TODO:

• send letters to the authors (Børre)
  • talked to Kurt Tore Andersen, will get a letter from us (Børre)

Bible texts

We will get text from Finland, but still haven't received any. Swedish html has arrived, no paratext. Norsk bibelselskap has not sent corrected New Testament versions for sme, and not paratext for nno/nob.

TODO:

• convert smj NT to paratext (Børre)
• get fin, swe, nob and nno NT and OT in paratext format. (Trond)

Davvi Girji

Talked to Harald Gaski, he will send the contracts to the writers' organisation, to ensure the contract is ok.

TODO:

• call Brita Kåven again towards the end of the week (Børre)
• call the authors (Børre)
  • Harald Gaski

Min Áigi

TODO:

• send bug report to Apple (typing filenames in Terminal does not match, moving filenames across plattforms via tarballs creates problems - the names are not the same!) (Sjur)
  • done
• complete metainformation (Børre)

Kåfjord

Promised to send us texts. Some texts have arrived, but nothing from Ája.

TODO:

• contact Ája (Børre)
  • Not done

Sámi Instituhtta

TODO:

• wait for R. Valkeapää, call him this week (Børre)

Čálliid Lágádus

Børre talked to them, they are positive, and will give us print-ready pdf's. http://www.calliidlagadus.org/

Árran

The negotiations are underway, they discuss it in a meeting today, and we have an appointment for later today or tomorrow (both their internal texts, and discussions on Báhko). All further negotiations will go through Bård Eriksen / Báhko.

TODO:

• call Bård Eriksen (Børre)

User accounts and access

TODO:

• discuss and decide upon the exact access policy we want to give corpus users; meeting on Tuesday May 23, at 9.30 (Børre, Sjur, Trond)
  • still not done, new date: Friday 9.6.2006, at 9.30
• set upp the unix group structure to open for a new category of users: Linguists with access to the texts on the servers. Some of Børre, Saara, Trond.
• make a text-only corpus in the Oslo interface (dump the texts on omilia), thereby making them accessible to simple lexical lookup. The cost of doing this should be evaluated against what it takes waiting for the analysed corpus to be in place. (Saara, Trond)
  • Discussion on this issue after the previous meeting
    • Conclusion: skip the text-only version, and go for the analyzed one right away

More texts to the graphical corpus interface:

We need to get the infrastructure complete to be able to do this, then it should be a piece of cake.

Main obstacle for further progress: a tool to analyse text while keeping the xml structure. Here's some very simple pseudo-code:

1. parse xml
2. send each <p> node to lookup etc
3. analyse it
4. add </s><s> after each <.>, <!>, <?> mark
5. wrap up the analysis output in xml
6. put it in a <p>node again

The first implementation in Perl (Saara), later possibly a C implementation ( Tomi) for speed reasons.

TODO:

1. make an analyser that retains the xml structure (Saara)
2. Finish the tag unification (korpustags.txt) (Trond)
   1. progressing, but open questions remain (awaiting text to see how things crash)
3. add text to the server (Lars)

Aligner

Trond and Saara will continue this issue.

We need markup of parallelism in the corpus DTD, at least an indication of which documents belong together. Discussion to continue in the newsgroup (Saara has started it - please respond!).

Language recognition

TODO:

• create a short smj word list to help the trigram heuristics (Trond)
  • done (but the smj.lm file suffer from the narrow corpus base. The smj.wm file will also improve with more texts, but not that much)
• Add some flag to write into the xsl file (Saara)
  • done

Free and non-free texts

TODO:

• test the script(s) copying texts to the free/ dir which are explicitly marked as free (Saara)
  • done

Corpus summary

TODO:

• trim generated corpus summary pages (Sjur)
  • suggestion: lump together files with content less than X paragraphs (X < 5?)
  • nothing done

Proofed vs unproofed corpus files

TODO:

• count the number of parallel files of type unproofed/proofread (Saara)
  • done, 35 such file pairs
  • delayed for the time being, move to Bugzilla (Sjur)

Paradigm generation

Goal: Reuse Greenlandic code for paradigm generation.

TODO:

• Put Saara and Tero in contact with each other (Trond)
  • still open, awaiting both Tero and the more crucial corpus tasks
• The paradigm generator should also have an xml-out option (for use in e.g. electronic dictionaries) - on hold till we know more about the generator
  • move to Bugzilla (Sjur)

Hyphenator

Thomas is finished with adding ^ tags to the sme noun file.

Trond and Thomas have been working on the smj rule component, and have improved both the treatment of weak grade consonant clusters (preconsonantal geminates) and on some loan word patterns.

TODO:

• correct the treatment of hyphenation of word boundaries and exceptions (fst gymnastics) (Sjur, Trond)
  • Still not done - postponed till next week (w 24)
• Update the sme and sma rule sets with the insights gained from smj updates.
• add exception marks to the sme lexicons (Thomas)
  • done

Automatic Bugzilla reminder for untouched bugs

We need to get a summary by e-mail for all bugs not touched in more than 5(?) weeks. Do we want it? Yes. It is possible? Børre will look around, if nothing found, he'll ask Thor-Øivind.

TODO:

• set up Bugzilla to send automatic reminders for bugs not touched in a given timeframe; ask Thor-Øivind for help if needed (Børre)

North Sámi

Topic: Actio+compound - how productive? How much does it destroy speller performeance by being close to real spelling errors with a high editing distance to the correct form?

Spelling error from typos.txt:
4       vuolggahansádji             vuolggasadji

vuolggahan
vuolggahan      vuolgga+N+Sg+Nom+Foc
vuolggahan      vuolggahit+V+TV+PrfPrc
vuolggahan      vuolggahit+V+TV+Ind+Prs+Sg1
vuolggahan      vuolggahit+V+TV+Actio+Acc
vuolggahan      vuolggahit+V+TV+Actio+Gen
vuolggahan      vuolggahit+V+TV+Actio+Nom

vuolggahansadji
vuolggahansadji vuolgga+N+SgNomCmp#ho+N+SgNomCmp#atnu+N+SgNomCmp#sadji+N+Sg+Nom
vuolggahansadji vuolgga+N+SgNomCmp#ho+N+SgNomCmp#atnu+N+SgNomCmp#sadji+N+Sg+Nom
vuolggahansadji vuolgga+N+SgNomCmp#ho+N+SgNomCmp#atnu+N+SgNomCmp#sadji+N+Sg+Nom
vuolggahansadji vuolggahit+V+TV+Actio#sadji+N+Sg+Nom
vuolggahansadji vuolggahit+V+TV+Actio#sadji+N+Sg+Nom
vuolggahansadji vuolggahit+V+TV+Actio#sadji+N+Sg+Nom

TODO:

• Maaren is in Tromsø this week, we should continue our linguistic discussions
  • not done, and Maaren is now back in Guovdageaidnu (?)

Lule Sámi

There are some open issues in the marginal area of the smj transducer:

• numerals, e.g. Our poor treatment of number words becomes more visible as we get real texts (guhttatuvsánanielljatjuodegålmmålåkvihtta is not recognised, etc.).
• names => waiting for the new name lexicon
• compounds? Shortening here as well, but not in written language (some lexicalised exceptions, as in sme) => nothing to discuss/implement
• loanwords? We should consider importing the ^LOAN words from sme and Lule-ify them. Otherwise, loan words should come from corpora.

TODO:

• 50 unknown words left+2 abbr. +moaddi etc (numerals) need more checks
• add proper numeral analysis/treatment
• add loanwords (e.g. latin -ere verbs)

TODO:

1. finish refactoring for multiple collections in the search interfarce ( Sjur)
   1. progressing, still major work to be done
2. develop the needed XQueries and interface (Sjur, Tomi)
   1. progressing, done some, commited (adding new term, create-termc-entry.xq)
3. data synchronisation between risten.no and the cvs repo (Tomi)
   1. discussion started on eXist-list, nothing useful came up. We need to reformulate the question from our perspective, and bring it up again ( Sjur)

Timeline:

• One-time conversion lexc2xml (sme to common)
  • done (must be redone at D-day, but the *script* is done)
• editing functions in risten.no - what about editing in emacs/see/other editors
  • in the works
• Automatic converson: xml2lexc (modulo language) (based on ccat) (ready to be executed) (Tomi)
  • not done

TODO:

• act, based on the answers (Sjur)
  • done: we are asking for more clarification, and then pick the best
• write clarification letters to PL and LS (Sjur with help from Børre, Trond)
  • LS - done
  • PL - has started, not finished

Summer vacation

Bug fixing

43 open Divvun/Disamb bugs (two down!), and 25 risten.no bugs

Guess: 1/3 of the bugs are fixed already (?)

Please help Saara with bug 279 (Perl locale). Not much help... Saara will contact Roy on this issue.

After the corpus issues have been somewhat settled, we should do a bug barnraising. ... and then a new one after the name lexicon is fixed.

Forrest:Unicode & JSPWiki

Sjur has identified a way of making forrest run read correctly the jspwiki files (UTF-8 encoded):

forrest run -Dforrest.jvmargs="-Dfile.encoding=utf-8"

It is added to our Forrest documentation.

RAM on our G5

We got some defect RAM. Our business contact has received a whole party of defect RAM, they are investigating this. After that is resolved we will receive "good" RAM.

Gobby

0.3 is working fine on Mac, Linux and Windows. Should be installed on all computers c.f. http://darcs.0x539.de/trac/obby/cgi-bin/trac.cgi/wiki/InstallationGuide (our preinstalled Xcode veriosn is 2.0, must be 2.1):

• Børre - ok
• Maaren - ok
• Saara - todo
• Sjur - ok
• Thomas - ok
• Tomi - ok
• Trond - ok

Trond has asked Lars Nygård and Tero Avellan to install Gobby as well. Per Langgård is already using it.

SEE 2.5 extensions

TODO:

• give the jspwiki syntax colouring mode to all (Sjur)
  • JSPWiki.mode checked in to cvs in gt/src/ - just double-click the mode!
• write a perl script to extract all TODO items and sort them according to responsible person; wrap the script in a AppleScript available from the menu
• other modes on the whish list:
  • twol mode
  • xfst mode
  • lexc mode
  • vislcg mode

12.06.2006 09: 30

Closed at 11: 08

Børre

• corpus collection:
  • send out contracts with accompanying letter
  • Gather public texts, preferrably also parallel ones
  • Continue converting text from input format to our xml
  • convert nob and nno bible texts to be used as part of a parallel corpus
  • review the paratext2xml converter
  • convert smj NT to paratext
  • Send out letters to the rest of the Iđut authors
  • send contract to Kurt Tore Andersen
  • call Brita Kåven again
  • call Harald Gaski
  • contact Ája (Kåfjord)
  • Send renaming scripts to R. Valkeapää
  • complete Min Áigi metadata
  • call Bård Eriksen
• corpus access:
  • meeting 9.6. t 9.30: discuss and decide upon the exact access policy we want
  • set upp the unix group structure for corpus users (also Saara, Trond)
• set up Bugzilla automatic reminders for open issues; ask Thor-Øivind if needed
• fix bugs!

Maaren

• On sick leave

Saara

• Create a parallel corpora of the new testaments
• add more texts to the graphical corpus interface
• grammatical searchability in the graphical corpus interface
• Implement links to parallel files in corpus header
• Implement parallel corpus upload in web upload script
• Install Gobby
• set upp the unix group structure for corpus users (also Børre, Trond)
• Test the aligners once again
• make an analyser that retains the xml structure for Oslo
• discuss parallel text markup in newsgroup
• fix bugs!

Sjur

• public tender:
  • write e-mail to PL and ask for more clarification
• fst gymnastics to add hyphenation and word boundary marks to hyphenation transducer
• name lexicon:
  • implement editing functions
  • finalise refactoring for multiple collections (regular search interface)
• change corpus-summary processing to generate smaller pages
• meeting 23.5. t 9.30: discuss and decide upon the exact access policy we want
• move to Bugzilla:
  • proofed/unproofed parallel text issue
  • xml output of paradigm generator
• fix bugs!

Thomas

• hyphenation-rule-set
• work on compounding and derivation
• lule sámi incoming words
• sme G3 issue

Tomi

• new proper name lexicon
  • data synchronisation of proper nouns between risten.no and CVS
  • XQuery refactoring and code development for our proper noun editor
  • new version of xml2lexc (based on ccat), should handle complex names correct: construct entries like we have now from the different parts of a complex name entry
• read aligner docu, install, provide feedback
• Set up the mechanism for the hash-mark transducer package
• add ccat option to analyse text while keeping the xml tags and structure
• fix bugs!

Trond

• better smj NT text
• get fin, swe, nob and nno NT and OT in paratext format
• install aligner, test it and give feedback
• fst gymnastics to add hyphenation and word boundary marks to hyphenation transducer
• Put Saara and Tero in contact with each other
• meeting 9.6. t 9.30: discuss and decide upon the exact access policy we want
• set upp the unix group structure for corpus users (also Børre, Saara)
• fix bugs!.