Meeting_2006-05-22

• Date: 22.05.2006
• Time: 09.30 Norw. time
• Place: Where we are
• Tools: SubEthaEdit, iChat

1. Opening, agenda review
2. Reviewing the task list from two weeks ago
3. Documentation - divvun.no
4. Corpus gathering
5. Corpus infrastructure
6. Infrastructure
7. Linguistics
8. name lexicon infrastructure
9. Spellers
10. Other issues
11. Summary, task lists
12. Closing

Opened at 09: 56.

Present: Børre, Sjur, Thomas, Trond, Tomi

Absent: Maaren, Saara

Main secretary: Thomas (with help from others)

Agenda accepted as is.

Børre

• corpus work:
  • send out contracts with accompanying letter
    • Sent to Elle Márjá Vars
  • Gather public texts, preferrably also parallel ones
    • Gathered, some added to the corpus
  • Continue converting text from input format to our xml
    • Done
  • convert nob and nno bible texts to be used as part of a parallel corpus
    • Not done
  • review the paratext2xml converter
    • Not done
  • convert smj NT to paratext
    • Not done
  • Send out letters to the rest of the Iđut authors
  • call Brita Kåven
    • Done
  • contact Kåfjord
    • Will have to contact Ája
  • create weekly cron job to mirror Odin URL and detect new/updated pages
    • Not done
  • Check the status & license of the corpus texts
    • Done some
  • contact Korhonen & Kuhmunen
    • Done
• public tender:
  • continue public tender offer evaluation
    • meeting with Sjur, Tomi and Trond on friday afternoon
  • meeting on Thursday 18.5. at 10.00 AM with Sjur, Tomi, Trond
    • see above
  • telephone meeting with Finnut
    • Not done
• install latest SEE
  • Not done
• install Gobby using Darwin Ports (also for Thomas and Maaren)
  • Done for me
• move to Bugzilla:
  • write docu for how to apply for a corpus user account
    • Not done, but talked to Roy Dragseth about this (more below).
• fix bugs!

Maaren

• On sick leave

Saara

• been on sick leave
• Create a parallel corpora of the new testaments.
• add more texts to the graphical corpus interface
• grammatical searchability in the graphical corpus interface
• Implement links to parallel files in corpus header.
  • not done
• Implement parallel corpus upload in web upload script
  • not done
• Check the status & license of the corpus texts and change the default of the availability to "license" instead of "free"
  • not done
• Rerun corpus conversion
  • done on daily basis
• Install Gobby
  • got problems during installation
• make xsl conversion routine for the typographic tags
  • done
• update the corpus script(s) to only copy texts to the free/ dir which are explicitly marked as free
  • not done
• Add some language recognition flags to write into the xsl file
  • not done
• rename corpus dirs, and create symlinks
  • not done
• fix bugs!

Sjur

• public tender:
  • read & evaluate received offers
    • done
  • meeting on Thursday 18.5. at 10.00 AM with Børre, Tomi, Trond
    • we had a meeting later (I had technical problems with my Mac at the time)
  • telephone meeting Friday with Finnut
    • short call, will have the meeting on Monday 22 instead
• fst gymnastics to add hyphenation and word boundary marks to hyphenation transducer
  • nothing
• name lexicon:
  • implement editing functions
    • updated my local cvs sandbox to use the new server (victorio), and checked in my recent work
  • finalise refactoring for multiple collections (regular search interface)
    • done some, but not finished
• update corpus-summary processing to adhere to the new structure
  • done, but the generated page/file size for sme is way too large due to the large number of Min Áigi files. The corpus summary needs to be broken down in smaller parts, probably on source and year (will give 2-3000 documents/page)
• send bug report to Apple re filename matching and accented characters in Terminal
  • not done
• fix bugs!

Thomas

• correct hyphenation of exceptions
  • finished lule sámi, now working with north sámi
• correct hyphenation of smj -st-
  • done
• work on compounding and derivation
  • worked a little
• smj G3 issue
  • finished
• sme G3 issue
  • not done
• set up a linguistic workshop while Maaren is in Tromsø
  • not done

Tomi

• move to Bugzilla:
  • aspell UTF-8 suffix bug
    • done
  • Document aspell infrastructure: finish doc/proof/spelling/X-spell/aspell.xml (it's almost done, but there are a couple of loose ends)
    • done
• new proper name lexicon
  • implement data synchronisation of proper nouns between risten.no and CVS
    • not done
  • XQuery refactoring and code development for our proper noun editor
    • some done
  • new version of xml2lexc (based on ccat), should handle complex names correct: construct entries like we have now from the different parts of a complex name entry
    • not done
• read aligner docu, install, provide feedback
  • not done
• install and test Gobby
  • done, but not successfully
• Set up the mechanism for the hash-mark transducer package
  • not done
• meeting on Thursday 18.5. at 10.00 AM with Børre, Sjur, Trond
  • done
• add ccat option to analyse text while keeping the xml tags and structure
  • not done
• fix bugs!

Trond

• better smj NT text
  • Not done
• get fin, swe, nob and nno NT and OT in paratext format
  • Not done
• install aligner, test it and give feedback
  • Not done
• fst gymnastics to add hyphenation and word boundary marks to hyphenation transducer
  • Not done
• get/upgrade keys for Børre's room for Tomi and Thomas
  • Ordered upgrade.
• Rethink the doubletagging procedure for names, consider grammatically motivated semtag conversion routines ("Helsinki" from Plc to Obj to Org)
  • Done some thinking
• Check the status & license of the corpus texts
  • Not done
• Work on the graphical corpus tag list
  • Done some.
• send Saara smj files for language recognition
  • Asked for text, discussing with Árran
• Put Saara and Tero in contact with each other
  • Not done
• meeting on Thursday 18.5. at 10.00 AM with Børre, Sjur, Tomi
  • Done
• ask Lars Nygård, Per Langgård and Tero Avellan to install Gobby 0.3
  • Asked Per, not the others.
• set up a linguistic workshop while Maaren is in Tromsø
  • Not done
• fix bugs!.

TODO:

• documentation on how to apply for a user account for the corpus repo ( Børre)
  • we will administer the corpus user accounts ourselves
  • We first have to discuss and decide what we want before Børre can write documentation (see further down)

Collecting

See a previous meeting memo for what's to be done.

TODO:

• Send out the rest of the letters (Børre)

New contracts:

• none last week

Olavi Korhonen's Lule Sámi dictionary.

Phoned Korhonen. He was willing to sign the contracts, and wanted some kind of access to our corpus. He wants to collect words for his Northern Sámi dictionary.

TODO:

• set up user account/corpus access for Olavi (Børre)

KIO Grafisk and the Iđut books

TODO:

• send letters to the authors (Børre)

Bible texts

We will get text from Finland, but still haven't received any. Swedish html has arrived, no paratext. Norsk bibelselskap has not sent corrected New Testament versions for sme, and not paratext for nno/nob.

TODO:

• convert smj NT to paratext (Børre)
• get fin, swe, nob and nno NT and OT in paratext format. (Trond)

Davvi Girji

Called her last week. She said Davvi Girji os would give us permission to use texts by their authors. She wasn't sure if we could get the texts directly from Davvi Girji, because of copyrights of pictures and other artwork in the books. Said they (Davvi Girji workers) were going to have a meeting right after our conversation by phone, and take some kind of desicion on this case. Haven't heard anything from Kåven, though.

TODO:

• call Brita Kåven again towards the end of the week (Børre)

Min Áigi

The Min Áigi format should be dealt with: \@ingress etc should be dealt with for the .txt, but business as usal for the .doc files. Saara has done the xsl conversion routine for the typographic tags. It still needs some fine tuning, as there are some @ tags that were not included in our initial list.

TODO:

• send bug report to Apple (typing filenames in Terminal does not match, moving filenames across plattforms via tarballs creates problems - the names are not the same!) (Sjur)
  • not done yet
• make xsl conversion routine for the typographic tags (Saara)
  • Done, some adjustment needed.

Kåfjord

Promised to send us texts. Some texts have arrived, but nothing from Ája.

TODO:

• contact Ája (Børre)

Sámi Instituhtta

Børre contacted Richard Valkeapää, the IT-consult at NSI. He put it on his todo list, as he would have to contact the person who has worked with the newspaper texts anyway. He said this would be done in the near future (within a month).

TODO:

• wait for R. Valkeapää, call him next week (Børre)

User accounts and access

Talked to Roy Dragseth last week about this. Turns out that we should do the administration ourselves. We will have to discuss on what level we would like to give access to users of our corpus.

This is from our contracts:

Contract 1 

3.7 Mottakar kan gje personleg bruksrett til tekstsamlinga til personar som har
skrive under på bruksrettskontrakten i Vedlegg 2. Mottakar skal ikkje gje
bruksrett til tekstsamlinga til personar som ein har grunn til å tru vil bryte
vilkåra i kontrakten. Mottakar forpliktar seg til å informere avgjevar med ein
gong han/ho får kjennskap til mogleg brot på desse vilkåra.

(Vedlegg 2 = Contract 3) 

Contract 3

4.1 Brukaren har berre rett til å bruke tekstane til forsking eller slike
kommersielle språkteknologiske eller andre liknande formål, som ikkje bryt med
Lov om opphavsrett til åndsverk. Brukaren kan bruke tekstane for å gjere seg
nytte av dei språktrekka (t.d. statistisk informasjon, grammatiske reglar og
semantiske skildringar) han/ho har funne gjennom forsking, og plukke ut kortare
sitat frå tekstane.

5.3 Brukaren får ikkje ta større delar av løpande tekst i tekstsamlinga enn
korte sitat bort frå den tenaren som tekstsamlinga er installert på. Det er lov
til å lagre temporære kopiar på sjølve tenaren på det vilkåret at brukaren tek
omsyn til datatryggleiken. Denne avgrensinga gjeld ikkje offentlege dokument
(t.d. NOU, stortingsmeldingar o.l.). Det går klårt fram av kvart dokument kva
for lisens som er knytta til det.

TODO:

• before anything else is done: Zip an xml-stripped version of our free texts and send them to Olavi. In that way, he gets something to work on, and we will get feedback (Børre)
• discuss and decide upon the exact access policy we want to give corpus users; meeting on Tuesday May 23, at 9.30 (Børre, Sjur, Trond)
• set upp the unix group structure to open for a new category of users: Linguists with access to the texts on the servers. Some of Børre, Saara, Trond.
• make a text-only corpus in the Oslo interface (dump the texts on omilia), thereby making them accessible to simple lexical lookup. The cost of doing this should be evaluated against what it takes waiting for the analysed corpus to be in place. (who?)

Name change again?

TODO:

• rename corpus dirs, and create symlinks (Saara)

Free and non-free texts

More info in a previous meeting memo.

TODO:

• Check the status of the texts, again. (Børre, Trond, Saara)
  • some is done, not complete yet
• Rerun the conversion afterwards (Saara is the one with the magic spell)
  • in progress
• update the script(s) to only copy texts to the free/ dir which are explicitly marked as free (Saara)
  • not done
• update the processing of the corpus summary files (Sjur)
  • done

More texts to the graphical corpus interface:

TODO:

• We would like to have more than the NT in the graphical interface ( Saara)
  • We add the largest texts first.
• We would like to have grammatical searchability, not only POS. (Saara, Trond This presupposes a discussion with Oslo. (Trond and Saara to continue this discussion)
• For Lule Sámi: We would like to have a parallel corpus interface with NT (text only). This presupposes better quality texts (Trond, Børre)
  • Better Lule NT text still not made.
• The list of good candicates: The longest (admin) texts.
  • We need a new option in ccat for analysing text while still keeping the xml tags and structure (Tomi)
  • xml texts number <p>, preprocess finds <.> <?> <!> and ccat numbers them as <s>.
  • Then the aligner aligns...

Top-three priorities:

1. discuss more with Lars on tag unification, and unify them (Trond)
2. change ccat to be able to create the right input for the corpus analysis (xml- tagged output) (Tomi). Work estimate: a few (2-3?) days - nothing this week, we will re-evaluate (and schedule) next Monday)
3. add text to the server (Lars)

Language recognition

TODO:

• refine language recognition (Saara)
  • in progress, continue discussion in 249
• create a short smj word list to help the trigram heuristics (Trond)
• send Saara smj files (Trond)
  • Trond has tried to get files, see task report above
• Add some flag to write into the xsl file (Saara):
  • method: do not run lg recognition
  • method: Choose between these 2: nob, sme, etc.

Paradigm generation

Goal: Reuse Greenlandic code for paradigm generation.

TODO:

• Put Saara and Tero in contact with each other (Trond)
  • still open
• The paradigm generator should also have an xml-out option (for use in e.g. electronic dictionaries) - on hold till we know more about the generator

Aligner

Trond and Saara will continue this issue.

Hyphenator

Thomas is finished with adding ^ tags to the smj noun file, and has continued working on the sme noun file.

Trond and Tomi have been working on the smj rule component, and have improved both the treatment of weak grade consonant clusters (preconsonantal geminates) and on some loan word patterns.

TODO:

• correct the treatment of hyphenation of word boundaries and exceptions (fst gymnastics) (Sjur, Trond)
  • Still not done.
• Update the sme and sma rule sets with the insights gained from smj updates.

General - hyphenation

See discussion, open questions and decission in a previous meeting memo.

TODO:

• Set up the mechanism for the hash transducer package - fst gymnastics, see above.
• add exceptions marks to the smj lexicon (boks^távva)
  • done

North Sámi

TODO:

• set up a linguistic workshop while Maaren is in Tromsø (and remember that she is on sick leave all the time) (Trond, Thomas)

Lule Sámi

There are some open issues in the marginal area of the smj transducer:

• numerals, e.g. Our poor treatment of number words becomes more visible as we get real texts (guhttatuvsánanielljatjuodegålmmålåkvihtta is not recognised, etc.).
• names => waiting for the new name lexicon
• compounds? Shortening here as well, but not in written language (some lexicalised exceptions, as in sme) => nothing to discuss/implement
• loanwords? We should consider importing the ^LOAN words from sme and Lule-ify them. Otherwise, loan words should come from corpora.

TODO:

• 50 unknown words left+2 abbr. +moaddi etc (numerals) need more checks
• add proper numeral analysis/treatment
• add loanwords (e.g. latin -ere verbs)

TODO:

1. finish refactoring for multiple collections in the search interfarce ( Sjur)
   1. improving, not finished
2. develop the needed XQueries and interface (Sjur, Tomi)
   1. progressing, done some, haven't commited (adding new term, create-termc-entry.xq)
3. data synchronisation between risten.no and the cvs repo (Tomi)
   1. discussion started on eXist-list, we'll wait a couple of days to see what's coming out of it, and if nothing useful to us, we'll add our use case with questions
4. test and review when ready
5. Rethink the doubletagging procedure for names, consider grammatically motivated semtag conversion routines ("Helsinki" from Plc to Obj to Org, or the Lyndi England issue) (Trond)

Nothing until the new proper noun lexicon is in place. We don't have enough people to do both. Here's our most important targets for spellers in the near future:

• aspell
• hunspell

Finnut called, and here's their evaluation: if we think that the offers are incomplete or otherwise not fully acceptable, we can enter negotiations with the companies, effectively cancelling the current public tender. This can only be done as long as we don't change the public tender document (that is, the foundation for the public tender) - if it is changed, we have to announce the whole competition again, with the usual 53 days minimum deadline for applications.

Sjur will send an e-mail to the project board, outlining the different aspects of the two offers, and ask about their opinion on the following questions:

• accept the offers as is, or enter negotiations?
• if accept, which one?

TODO:

• meeting on Thursday 18.5. at 10.00 AM (Børre, Sjur, Tomi Trond)
  • done (on Friday)
• telephone meeting with Finnut (Børre, Sjur)
  • short call on Friday, he called today (during the meeting, see above)
• write e-mail to the project board, ask for their opinion regarding the offers ( Sjur)

Summer vacation

Bug fixing

45 open Divvun/Disamb bugs, and 25 risten.no bugs

Guess: 1/3 of the bugs are fixed already (?)

Please help Saara with bug 279 (Perl locale). Not much help... Saara will contact Roy on this issue.

After the corpus issues have been somewhat settled, we should do a bug barnraising. ... and then a new one after the name lexicon is fixed.

Gobby

0.3 is working fine on Mac, Linux and Windows. Should be installed on all computers c.f. http://darcs.0x539.de/trac/obby/cgi-bin/trac.cgi/wiki/InstallationGuide (our preinstalled Xcode veriosn is 2.0, must be 2.1):

• Børre - ok by copying /opt/local/ from Trond
• Maaren - Børre to do it
• Saara - todo
• Sjur - ok
• Thomas - Børre to do it
• Tomi - todo
• Trond - ok

Easy way out when the standard Darwin Ports installation isn't working: just get a copy of /opt/local/ from Børre.

Trond should ask Lars Nygård and Tero Avellan to install Gobby as well; has asked Per Langgård.

SEE autosave AppleScript

Copy the following into a ScriptEditor window:

tell application "SubEthaEdit"
	repeat until false is true
		save documents
		delay 60
	end repeat
end tell

and click "run". All your SubEthaEdit documents will be automatically saved every minute (the interval can be changed by specifying another value (in seconds) for delay)

Børre

• corpus collection:
  • send out contracts with accompanying letter
  • Gather public texts, preferrably also parallel ones
  • Continue converting text from input format to our xml
  • convert nob and nno bible texts to be used as part of a parallel corpus
  • review the paratext2xml converter
  • convert smj NT to paratext
  • Send out letters to the rest of the Iđut authors
  • call Brita Kåven again towards the end of the week
  • contact Ája (Kåfjord)
  • create weekly cron job to mirror Odin URL and detect new/updated pages
  • Check the status & license of the corpus texts
  • wait for R. Valkeapää, call him next week
• public tender:
  • assist with letter to the project board
• corpus access:
  • zip an xml-stripped version of our free texts for to Olavi
  • meeting 23.5 t 9.30: discuss and decide upon the exact access policy we want
  • set up user account/corpus access for Olavi
  • set upp the unix group structure for corpus users (also Saara, Trond)
• install latest SEE
• install Gobby for Thomas and Maaren
• fix bugs!

Maaren

• On sick leave

Saara

• Create a parallel corpora of the new testaments.
• add more texts to the graphical corpus interface
• grammatical searchability in the graphical corpus interface
• Implement links to parallel files in corpus header.
• Implement parallel corpus upload in web upload script
• Check the status & license of the corpus texts and change the default of the availability to "license" instead of "free"
• rerun the corpus conversion
• Install Gobby
• update the corpus script(s) to only copy texts to the free/ dir which are explicitly marked as free
• Add some language recognition flags to write into the xsl file
• rename corpus dirs, and create symlinks
• set upp the unix group structure for corpus users (also Børre, Trond)
• fix bugs!

Sjur

• public tender:
  • write letter to the project board
• fst gymnastics to add hyphenation and word boundary marks to hyphenation transducer
• name lexicon:
  • implement editing functions
  • finalise refactoring for multiple collections (regular search interface)
• change corpus-summary processing to generate smaller pages
• send bug report to Apple re filename matching and accented characters in Terminal
• meeting 23.5 t 9.30: discuss and decide upon the exact access policy we want
• fix bugs!

Thomas

• correct hyphenation of exceptions (sme)
• work on compounding and derivation
• sme G3 issue
• set up a linguistic workshop while Maaren is in Tromsø

Tomi

• new proper name lexicon
  • implement data synchronisation of proper nouns between risten.no and CVS
  • XQuery refactoring and code development for our proper noun editor
  • new version of xml2lexc (based on ccat), should handle complex names correct: construct entries like we have now from the different parts of a complex name entry
• read aligner docu, install, provide feedback
• install and test Gobby
• Set up the mechanism for the hash-mark transducer package
• add ccat option to analyse text while keeping the xml tags and structure
• fix bugs!

Trond

• better smj NT text
• get fin, swe, nob and nno NT and OT in paratext format
• install aligner, test it and give feedback
• fst gymnastics to add hyphenation and word boundary marks to hyphenation transducer
• get/upgrade keys for Børre's room for Tomi and Thomas
• Rethink the doubletagging procedure for names, consider grammatically motivated semtag conversion routines ("Helsinki" from Plc to Obj to Org)
• Check the status & license of the corpus texts
• Work on the graphical corpus tag list
• send Saara smj files for language recognition
• create a short smj word list to help the trigram heuristics
• Put Saara and Tero in contact with each other
• ask Lars Nygård and Tero Avellan to install Gobby 0.3
• set up a linguistic workshop while Maaren is in Tromsø
• meeting 23.5 t 9.30: discuss and decide upon the exact access policy we want
• set upp the unix group structure for corpus users (also Børre, Saara)
• fix bugs!.

29.05.2006 09: 30

Closed at 11: 35