Meeting_2009-11-16
Contents:
- Meeting setup
- Agenda
- Opening, agenda review, participants
- Updated task status since last meeting
- Oahpa!
- Corpus gathering
- Promoting Divvun
- Future plans, directions and ideas
- Infrastructure
- Linguistics
- Name lexicon/risten.no infrastructure
- Proofing tools
- Other
- Next meeting, closing
- Appendix - task lists for the next week
Meeting setup
- Date: 16.11.2009
- Time: 09.30 Norw. time
- Place: Internet
- Tools: SubEthaEdit, iChat
Agenda
Cf. one of the following, depending on context:
- the upper bar of the SEE window (provided you use the JSPWiki syntax mode)
- the TOC in Forrest-rendered output, like HTML and PDF
Opening, agenda review, participants
Opened at 10: 08.
Present:
Absent: none
Agenda accepted as is.
Updated task status since last meeting
Børre
- install the HFST tools
- set up Maja's old computer with Boot Camp or some virtualization software
- order scanners (one book and one flatbed with sheet feeder)
- implement language switch for static divvun site
- upgrade the XServe to Snow Leopard Server
- improve XSL script to transform leaflet Forrest XDocs to an OOo Draw document
-
gt/Makefile remake
- get translations of thank-you letter
- fix bugs!
Ciprian
- MT - Autshumato:
- test Autshumato Alpha version on Windows using Launcher
- make Autshumato run on Mac
- todo
- todo
- test Autshumato Alpha version on Windows using Launcher
- eXist:
- make eXist running with the SLT-DB
- todo
- todo
- make eXist running with the SLT-DB
- infrastructure
- make a schema/dtd description of the lexC-file (experiment with
- transform sme-lexC files into XML format
- todo
- todo
- make a schema/dtd description of the lexC-file (experiment with
- GT web:
- coloring output of the disambiguation
- add a tree visualizer for the dependency trees
- todo
- todo
- coloring output of the disambiguation
- Bodø Oahpa:
- generalize for semantic classes and pos
- implement a login mechanism
- implement a upload data mechanism
- setup the Bodø version on victorio
- todo
- todo
- generalize for semantic classes and pos
- Sandbox Oahpa:
- finish the db installation for sb_oahpa
- implement and test the Finnish lemmata (and if it works transfer it to the official OAHPA implementation)
- todo
- todo
- finish the db installation for sb_oahpa
- Running Oahpa:
- add an Oahpa clock and date excercise (cf. Numra)
- Generate fin/xml/{nouns|verbs|adjectives}.xml, and implement the new Leksa
- todo
- todo
- add an Oahpa clock and date excercise (cf. Numra)
- KomEngFin:
- split by pos and transform the actual format into gt_dict format
- compile only one version of dictionary and use css for user preferences
- generate the first StarDict version of komeng and komfin dictionary
- test the automatic sorting by Komi alphabet in xsl (as discussed
- todo
- todo
- split by pos and transform the actual format into gt_dict format
- SmeNob:
- the StarDict on Windows: try the HTML-plugin
- try to reduce the dict-size on mac: experiment with xPointer, etc.
- incorporate the passives into the last version of the sme: nob
- todo
- todo
- the StarDict on Windows: try the HTML-plugin
- SmaNobSwe:
- extend the smanobswe dictionary
- check all bug in the sma: nob-swe dictionary
- fix the bug with the string length in the popup window (sma: nob)
- todo
- todo
- extend the smanobswe dictionary
- SmjNob:
- start working at a Lule-Sami based on Bruce Moren's data
- done
- done
- start working at a Lule-Sami based on Bruce Moren's data
- SjdRus:
- continue the work at the Kildin-Russian dictionary, next internal deadline is
- ongoing
- ongoing
- continue the work at the Kildin-Russian dictionary, next internal deadline is
- Corpus:
- check the processing of new corpus documents (error logging during
- todo
- todo
- check the processing of new corpus documents (error logging during
- Lexicon workshop
- finish the preparation of the teaching materials
- prepare the talk for the workshop
- ongoing
- finish the preparation of the teaching materials
Maja
- SGM letter meeting Tuesday 9: 30 with Sjur, Thomas
- work meeting on loan words and proper nouns
- Prepare text´s about normativity issue to SGL/SGM
- done
- done
- more work on sma adjectives
- done
- done
- contact Turi Aleksandersen about book signature follow-up from Joseph
- Will do
- Will do
- contact authors of sma aticles in 9-10 eds. of the yearbook
- Not done yet
Sjur
- SGM letter meeting Tuesday 9: 30 with Maja, Thomas
- Sorry, that meeting slipped off. Retry this week.
- Sorry, that meeting slipped off. Retry this week.
- difftest for fst and PL speller
- fix the sma installer build issues
- @Barents: plan a meeting/seminar with potential cooperation partners
- @Barents: plan for The Real Thing
- @TTS: start public tender process
- make Leif Åge send out CD's to distribution points
- start Nordplus Sprog project
- send eXist log files to Ciprian
- Write a formal letter to Davvi girji about electronic dictionaries
- make XSL script to transform leaflet Forrest XDocs to an OOo Drawer document
- continue to write a proofing API & implementation specification
- Voikko is coming up as a candidate for the first version of our HFST-based
- Voikko is coming up as a candidate for the first version of our HFST-based
- name db/risten.no
- follow-up on some Polderland-related bugs: 621, 630, 652
- support and maintenance contract for sme and smj, MS+Adobe tools
- Sámi languages as part of Norsk språkbank
- find and contact the correct person in SD, to get the manuscript for all Sámi
-
fix bugs!
- other things:
- still more HFST debugging and bug reporting
Thomas
- SGM letter meeting Tuesday 9: 30 with Maja, Sjur
- not done
- not done
- work meeting on loan words and proper nouns
- not done
- not done
- prepare text´s about normativity issue to SGL/SGM
- not much done this week
- not much done this week
-
sma adjectives
- not much done this week
- not much done this week
- Digitalize south saami books
- worked
- worked
-
fix bugs!
- some work done
Tomi
- install the HFST tools
- done
- done
- document how compounding is controlled in the PLX conversion
- not done
- not done
- fix double hyphen bugs
- not done
- not done
- fix PL smj hyphenator bug
- not done
- not done
- fix PL and Hunspell conversion bugs
- done some
- done some
-
fix bugs!
- other
- compiled & tested new spellers
Trond
- work meeting on loan words and proper nouns
- Mikogo for Lene and Linda
- clock and date for Numra
- Kildin Sámi Keyboard for Windows
- fix bugs!.
Oahpa!
Meeting memos can be found at
TODO
- Register oahpa.no (Trond)
- Generate fin/xml/{nouns|verbs|adjectives}.xml, and implement the new Leksa
- clock and date for Numra (Trond, Ciprian).
Corpus gathering
TODO:
- Which books are digitized?
- These books should be OCR'ed/digitized: Anna Jacobsens books
- Its not digitized yet, should do it this week.
- use the corpus gathering document to keep track of each book
- These books should be OCR'ed/digitized: Anna Jacobsens books
- continue gathering sma corpus texts (Maja)
- get sma articles in Š-bláđđi
- the Gun Utsi book is almost there - one contract missing (Maja)
- get sma articles in Š-bláđđi
- write formal letter to Davvi Girji (Sjur)
- send a copy of the signed contracts back to the authors, translators and
- find and contact the correct person in SD, to get the manuscript for all Sámi
- get the sma yearbooks from Saemien sïjhte ( Maja)
Promoting Divvun
TODO:
- make leaflet to inform about the project (Børre)
- add InDesign text (Sjur)
- make XSL script to transform Forrest XDocs to an OOo Drawer document
- add InDesign text (Sjur)
- distribute CD version through the library bus, the language centres and common
- make him send out CD's accordingly (Sjur)
- make him send out CD's accordingly (Sjur)
- update online download log statistics page (Børre)
Future plans, directions and ideas
See a separate document in plan/strat/5year.jspwiki.
Northern areas project
TODO:
- Attend a beginners' course in Russian (priority: the alphabet!) near you..
- plan a meeting/seminar with potential cooperation partners (Trond, Sjur)
- plan for The Real Thing (Trond, Sjur)
Infrastructure
To accomodate future enhancements in different directions (in rough order of
- test bench for all parts of our language technology efforts
- test bench enhanced, but not yet complet
- test bench enhanced, but not yet complet
- set up the Leopard Server features for collaborative support:
- permanent chat rooms
- stored (and indexed) chat transcripts of the chat rooms
- iCal server / group calendars
- wiki
- permanent chat rooms
- wiki? on G5 (is part of Leopard Server) or other web-based documentation
- improve Forrest stability and i18n support ( the divvun crashes)
-
Sjur has been working on better i18n and pdf rendering
-
Børre has some ideas for getting back to serving static html files
-
Sjur has been working on better i18n and pdf rendering
- reorganise the documentation:
- differ between target groups
- get better grouping
- decide what to write in forrest and what in wiki
- update/add missing parts
- differ between target groups
- migrate lexc lexicons to XML, splitting the task
- Name lexica (the Name project)
- Dictionaries (already in XML, task is to integrate them)
- At least migrate the lexc open POSes (Komi as a pilot case)
- Name lexica (the Name project)
- change the look of the documentation web
- use HFST as alternative to XFST
- this is now a real option (HFST2 is released), reqiures Makefile enhancements
- this is now a real option (HFST2 is released), reqiures Makefile enhancements
- corpus content moved to Max Planck repositories? Norsk språkbank?
- update infrastructure to allow content-restricted spellers for special target
TODO:
- install the HFST tools (Børre, Tomi)
- usage: make hfst TARGET=xxx, or use hwfst if you want a weighted one
- fao: make hfst
- usage: make hfst TARGET=xxx, or use hwfst if you want a weighted one
- upgrade the XServe to Snow Leopard Server (Børre)
- ask Lene, Linda to install Mikogo (Trond)
- restructure and clean the script catalogue, using subdirs to categorise the
- see posting in the news group
- see posting in the news group
- infrastructure remake: ( Børre, Ciprian, Saara, Sjur, Tomi, Trond)
- make a test-all target that runs all tests we have (Ciprian, Sjur, Trond)
- delayed until we have restructured the make/build process
- delayed until we have restructured the make/build process
- define and document testing routines (Ciprian, Sjur, Trond)
- delayed until we have restructured the make/build process
Linguistics
North Sámi
(nothing new, see proofing bugs below)
Lule Sámi
(nothing new, see proofing bugs below)
South Sámi
TODO:
- Prepare text´s about normativity issue to SGL/SGM (Maja, Thomas)
- meeting Tuesday 9: 30 (Maja, Sjur, Thomas)
- meeting Tuesday 9: 30 (Maja, Sjur, Thomas)
- adjectives (Maja with Thomas, Trond, Sjur)
- two competing naming conventions of continuation lexicons
- One naming goes ATTRSUFF-PREDSUFF-STEMTYPE
- One follows the sme convention of naming key adjectives
- There are duplicate lexica
- One naming goes ATTRSUFF-PREDSUFF-STEMTYPE
- The comparative issue open here and there
- The ATTRSUFF-PREDSUFF-STEMTYPE lexica now go to EVENCOMP and
- The ATTRSUFF-PREDSUFF-STEMTYPE lexica now go to EVENCOMP and
- two competing naming conventions of continuation lexicons
- work meeting on loan words and proper nouns, this week - tomorrow?
Name lexicon/risten.no infrastructure
Tomi has played with couchdb as a replacement
Instead of building our own webforms and back-end update scripts, use XForms
From the meeting with the terminology and IT teams last week:
- no major rework on the present search interface now
- no work on the editing section; instead:
- add existing lists of sanctioned terminology as separate term entities
- add a dictionary if we can make one with sufficient quality
This means the following tasks:
- find already approved lists, in paper or electronic form (term team)
- convert paper lists to electronic lists (term team)
- convert lists to standard XML (Sjur, Tomi)
- add prepared lists to risten.no (Sjur, Tomi)
TODO:
- send eXist log files to Ciprian (Sjur)
- fix i18n bug in risten.no/G5 (so they will work without the proper locale
- fix bugs in lexc2xml; add comments to the log element (Saara)
- finish first version of the editing (Sjur)
- test editing of the xml files. If ok, then: ( Sjur, Thomas, Trond)
- make terms-smX.xml <=== automatically from propernoun-sme-lex.xml (add nob
- convert propernoun-($lang)-lex.txt to a derived file from common xml files
- implement data synchronisation between risten.no and
- start to use the xml file as source file
- clean terms-sme.xml such that all names have the correct tag for their use
- merge placenames which are errouneously in different entries: e.g. Helsinki,
- publish the name lexicon on risten.no (Sjur)
- add missing parallel names for placenames (linguists)
- add informative links between first names like Niillas and Nils
Dictionaries
TODO:
- set up risten.no on eXist/XServe (as a beta version site) (Sjur)
- set up required infra for smenob on risten.no/XServe (Sjur)
- Continue the dictionary infrastructure discussion (Ciprian, Sjur, Trond)
- end user documentation (how to download and install) (Ciprian, Trond)
- Contact Davvi Girji about cooperation on electronic dictionaries
- developing the mobile phone version of smenob:
- http: //pfed.info/wksite/
- http: //pfed.info/wksite/
- Komi
- take out the doublets to a separate file (Ciprian)
- merge the doublets (Jaska, Trond)
- start looking at the komfin and komeng MacDicts (Ciprian)
- Completing the automaton to some state (Trond, Jaska, Paula)
- take out the doublets to a separate file (Ciprian)
Proofing tools
South Sámi second alpha
Maja:
- looks good: )
- some words are written correct, but the speller rejects them
Trond:
- a frequent bug was nubpie because of a wrong tag
- speller - analyser diff
Thomas:
- the phonetic rules seem to work now
Tomi:
- baseform test is checking whether the baseforms are recognised by the speller
TODO:
- Go through the phonrules file (Thomas)
- Pick a text for proofreading (Thomas)
- Proofread a text with tre eror$error notation (Maja)
- difftest for fst and PL speller (Sjur)
- Compile a new version by Friday (Tomi)
-
cd $GTHOME/prooftools/; sudo make mac-dmg
-
cd $GTHOME/prooftools/; sudo make mac-dmg
- Look for external beta testers?
- David and Jovsset will get the speller, for other people
- David and Jovsset will get the speller, for other people
- fix the installer build issues (Sjur)
- correct easter egg to say "South Sámi", not "North Sámi" (Sjur)
- fix sma installer bugs (Sjur)
HFST-based proofing tools
Voikko will be our near future proofing solution, using
TODO:
- continue to write an API specification, and an implementation specification
Testing
Spelling Error Markup
TODO:
- test new and nested error markup (Sjur)
- nesting still needs to be tested, depends on new ccat feature
Speller testing
TODO:
- test the error type selection feature in ccat (Sjur)
Testing open-source Norwegian spellers
Sjur has invited the open-source group to test their spell-checker using our
We should go to their developer meetings, and present our work and how to work
Speller bugs
List of bugs returned from Polderland:
- 621
- 630
- 652
- 656
- 676
Tag reordering for abbreviations have caused a lot of problems:
smj: hr. hr. hr+ABBR+Acc cand.philol. cand.philol. cand.philol+ABBR+N+Acc Per Per Per+N+Prop+Mal+Sg+Attr sme: hr. hr. hr+N+ABBR+Acc Per Per Per+N+Prop+Mal+Sg+Attr
Open issues based on test results:
sme
- 399 - missing numerals (plural forms) - still OPEN
- 425 - X not recognised; single letters were left out - still OPEN
- 435 - roman numbers - inflection of single letter numbers
- we should pregenerate all numbers once and for all, and store them in a
- we should pregenerate all numbers once and for all, and store them in a
- 461 - REGRESSION: missing suggestion (sáhkki)
- 508 - REGRESSION: accepts smj entries (most likely abbreviation missing)
- 520 - REGRESSION: r9 and š9 not defined (abbr. missing)
- 595 - prefix+name without hyphen (ovdaLot instead of ovda-Lot) - still
- 603 - suomabealdi accepted - still OPEN
- 606 - compound-tags LEXICON VUOHTA - still OPEN
- 613 - short gen. as second compound part - still OPEN
- 619 - numerals and pronouns to NAMÁK and SASJ fails - vihttasoarttat
- 629 - a taking part in compounding without hyphen - still OPEN
- only open case has word A-finálaid compounded
- only open case has word A-finálaid compounded
- 647 - numerals+NOUN - still OPEN, open case has uppercase letters
- 648 - unmotivated suggestions with numeral+noun - still OPEN
- 661 - REGRESSION: abbr. not recognized
- 709 - sámedikkeválga accepted - OPEN
- 728 - vowel shortening GenCmp+Left-tagged - still OPEN
- 779 - caseforms of pronoun okatahat - still OPEN
- 785 - does not recognize alphabet-abbr+noun - OPEN
- 802 - NEW: multiword propernouns
- 803 - NEW: FINJU- words accepted single-handed
- 804 - NEW: guovttilogát, njealjilogát
- 805 - NEW: Nouns+acronyms
smj
- 435 - roman number - single letter numbers now recognised
- we should pre-generate all numbers once and for all, and store them in a
- please note that inflection of single letter numerals is fine in
- we should pre-generate all numbers once and for all, and store them in a
- 482 - polardutkamin not recognized - FIXED
- 496 - REGRESSION: unrecognised clitics
- 556 - non-existent word accepted - FIXED
- 594 - lågenanguoktáj not recognized - still OPEN
- 595 - REGRESSION: prefix+name as split comp without hyphen
- 596 - C-giellan is not accepted - still OPEN
- 600 - Gen+hyph compound - FIXED
- 627 - prefix + hyhpen does not get accepted - FIXED
- 647 - numerals+NOUN - still OPEN, open case has uppercase letters
- 648 - unmotivated suggestions with numeral+noun - still OPEN
- 650 - REGRESSION: noun prefix+name compound without hyphen
- 652 - UPPERCASE-typos only get acronym-suggestions - still OPEN
- 692 - numeral-variants - all but one fixed (gáktsalågenantjuotakta), but
- 744 - REGRESSION: numerals + clitic
- 803 - NEW: VINJU- words accepted single-handed
- 805 - NEW: Nouns+acronyms
TODO:
- document how compounding is controlled in the PLX conversion (Tomi)
Hyphenator bugs
Open issues based on test results :
sme
No known issues!
smj
- 670 - Hard hyphen replaced with soft hyphen: 10-biejvvásattja (the word is
sma
Command to test the hyphenator:
preprocess dev/corp/pressemelding.txt | lookup bin/hyph-sma.fst | cut -f2 | \ lookup bin/hyphrules-sma.fst | grep -v '^$' | cut -f2 | uniq | see
TODO:
- fix PL hyphenator errors (Tomi)
- almost done - one smj bug left
Installer changes
TODO:
- test InDesign installer (Sjur)
User documentation
TODO:
- InDesign documentation (Sjur)
- Norwegian translation received from Davvi Girji
1.2 release
Content:
- several smj bug fixes
- lexicalisations
- InDesign Mac & Win
- new OOo beta
- improved installers, at least for Mac, preferably also for Windows
Other
Autumn planning
- speller test project -> Sjur, Børre, Thomas, X
- speech synthesis -> Sjur, Trond, BA as a starter
- dictionaries: smanob, komfineng, fkvnob -> Ciprian, Sjur, Trond
- risten.no -> Ciprian, Sjur, Tomi
- Barents project follow-up meeting
- sme and smj proofing tools, next version -> Thomas, Tomi, Sjur
- HFST-based proofing tools -> Sjur
Text to speech
TODO:
- refine syntax / dependency rules (Biret Ánne)
- start public tender process (Sjur)
- find appropriate voices (Biret Ánne)
MT and CAT
TODO:
- make A-ITE compile and run on Windows (Ciprian)
Open source
The repository is properly closed/open now, and the availability of the source
TODO:
- announce the availability of our repository on relevant linguistic lists
Windows testing machine
It has been used as a backup Mac computer last week.
TODO:
- set up Maja's old computer with Boot Camp or some virtualization software
Next meeting, closing
The next meeting is 23.11.2009, 9.30 Norwegian time.
The meeting was closed at 10: 49.
Appendix - task lists for the next week
Boerre
- install the HFST tools
- set up Maja's old computer with Boot Camp or some virtualization software
- order scanners (one book and one flatbed with sheet feeder)
- implement language switch for static divvun site
- upgrade the XServe to Snow Leopard Server
- improve XSL script to transform leaflet Forrest XDocs to an OOo Draw document
-
gt/Makefile remake
- get translations of thank-you letter
- fix bugs!
Ciprian
- MT - Autshumato:
- test Autshumato Alpha version on Windows using Launcher
- make Autshumato run on Mac
- test Autshumato Alpha version on Windows using Launcher
- eXist:
- make eXist running with the SLT-DB
- make eXist running with the SLT-DB
- infrastructure
- make a schema/dtd description of the lexC-file (experiment with
- transform sme-lexC files into XML format
- make a schema/dtd description of the lexC-file (experiment with
- GT web:
- coloring output of the disambiguation
- add a tree visualizer for the dependency trees
- coloring output of the disambiguation
- Bodø Oahpa:
- generalize for semantic classes and pos
- implement a login mechanism
- implement a upload data mechanism
- setup the Bodø version on victorio
- generalize for semantic classes and pos
- Sandbox Oahpa:
- finish the db installation for sb_oahpa
- implement and test the Finnish lemmata (and if it works transfer it to the official OAHPA implementation)
- finish the db installation for sb_oahpa
- Running Oahpa:
- add an Oahpa clock and date excercise (cf. Numra)
- Generate fin/xml/{nouns|verbs|adjectives}.xml, and implement the new Leksa
- add an Oahpa clock and date excercise (cf. Numra)
- KomEngFin:
- split by pos and transform the actual format into gt_dict format
- compile only one version of dictionary and use css for user preferences
- generate the first StarDict version of komeng and komfin dictionary
- test the automatic sorting by Komi alphabet in xsl (as discussed
- split by pos and transform the actual format into gt_dict format
- SmeNob:
- the StarDict on Windows: try the HTML-plugin
- try to reduce the dict-size on mac: experiment with xPointer, etc.
- incorporate the passives into the last version of the sme: nob
- the StarDict on Windows: try the HTML-plugin
- SmaNobSwe:
- extend the smanobswe dictionary
- check all bug in the sma: nob-swe dictionary
- fix the bug with the string length in the popup window (sma: nob)
- extend the smanobswe dictionary
- SjdRus:
- continue the work at the Kildin-Russian dictionary, next internal deadline is
- continue the work at the Kildin-Russian dictionary, next internal deadline is
- Corpus:
- check the processing of new corpus documents (error logging during
- check the processing of new corpus documents (error logging during
- Lexicon workshop
- finish the preparation of the teaching materials
- prepare the talk for the workshop
- finish the preparation of the teaching materials
Maja
- SGM letter meeting Tuesday 9: 30 with Sjur, Thomas
- work meeting on loan words and proper nouns
- Prepare text´s about normativity issue to SGL/SGM
- more work on sma adjectives
- contact Turi Aleksandersen about book signature follow-up from Joseph
- contact authors of sma aticles in 9-10 eds. of the yearbook
Sjur
- SGM letter meeting Tuesday 9: 30 with Maja, Thomas
- difftest for fst and PL speller
- fix the sma installer build issues
- @Barents: plan a meeting/seminar with potential cooperation partners
- @Barents: plan for The Real Thing
- @TTS: start public tender process
- make Leif Åge send out CD's to distribution points
- start Nordplus Sprog project
- send eXist log files to Ciprian
- Write a formal letter to Davvi girji about electronic dictionaries
- make XSL script to transform leaflet Forrest XDocs to an OOo Drawer document
- continue to write a proofing API & implementation specification
- name db/risten.no
- follow-up on some Polderland-related bugs: 621, 630, 652
- support and maintenance contract for sme and smj, MS+Adobe tools
- Sámi languages as part of Norsk språkbank
- find and contact the correct person in SD, to get the manuscript for all Sámi
- fix bugs!
Thomas
- work meeting on loan words and proper nouns
- prepare text´s about normativity issue to SGL/SGM
-
sma adjectives
- Digitalize south saami books
- fix bugs!
Tomi
- install the HFST tools
- document how compounding is controlled in the PLX conversion
- fix double hyphen bugs
- fix PL smj hyphenator bug
- fix PL and Hunspell conversion bugs
- fix bugs!
Trond
- work meeting on loan words and proper nouns
- Mikogo for Lene and Linda
- clock and date for Numra
- Kildin Sámi Keyboard for Windows
- fix bugs!.