Meeting_2007-03-26
Contents:
- Meeting setup
- Agenda
- 1. Opening, agenda review, participants
- 2. Updated task status since last meeting
- 3. Documentation
- 4. Corpus gathering
- 5. Corpus infrastructure
- 6. Infrastructure
- 7. Linguistics
- 8. Name lexicon infrastructure
- 9. Spellers
- 10. Other
- 11. Next meeting, closing
- Appendix - task lists for the next week
Meeting setup
- Date: 26.03.2007
- Time: 09.00 Norw. time
- Place: Internet
- Tools: SubEthaEdit, iChat
Agenda
- Opening, agenda review
- Reviewing the task list from last week
- Documentation - divvun.no
- Corpus gathering
- Corpus infrastructure
- Infrastructure
- Linguistics
- name lexicon infrastructure
- Spellers
- Other issues
- Summary, task lists
- Closing
1. Opening, agenda review, participants
Opened at 09: 48.
Present: Børre, Maaren, Saara, Sjur, Steinar, Thomas, Tomi, Trond
Absent: none
Agenda accepted as is.
2. Updated task status since last meeting
Børre
- update and fix our documentation and infrastructure as Steinar finds
- nothing last week
- nothing last week
- fix sme texts in corpus this month
- not done
- not done
- find missing nob parallel texts in corpus
- not done
- not done
- add prefixes to the PLX conversion
- two of them
- two of them
- add middle nouns to the PLX conversion
- done
- done
- improve number PLX conversion
- not done
- not done
- add sma texts to the corpus repository
- not done
- not done
- Improve automatic alignment process
- not done
- not done
- Add potential speller test texts
- not done
- not done
- collect a list of PR recipients, forward to Berit Karen Paulsen
- not done
- not done
- add version info to the generated speller lexicons
- not done
- not done
- run all known spelling errors in the corpus through the speller
- not done
- not done
- test the typos.txt list, and check that all entries are properly corrected
- not done
- not done
- consider how to do a regression self-test
- not done
- not done
- fix bugs!
Maaren
- lexicalise actio compounds
- Nickel s.310-315 (actio combounds)
- Nickel s.310-315 (actio combounds)
- Manually mark speller test documents for typos
- working with lundui.fi documents
Saara
- continue aligning the rest of the parallel files
- prepare more files for manual alignment
- mark-up the added speller test texts, using our existing xml format
- done for the files that exist in prooftest
- done for the files that exist in prooftest
- improve cgi-bin scripts
- add new features to the paradigm generator
- not done
- not done
- add gt/cwb/paradigm.smj.txt file into gt/script/server_anl.pl
- done
- done
- add new features to the paradigm generator
- add new XSL/XML headers for proofing test docs
- not done
- not done
-
fix bugs!
- other
- script for speller test data reformatting
- first version ready
- first version ready
- write conference paper
- done
- script for speller test data reformatting
Sjur
- hire linguist
- job interview done
- job interview done
- finish press release for the beta
- not done
- not done
- collect a list of PR recipients
- not done
- not done
- add version info to the generated speller lexicons
- discussed with Polderland
- discussed with Polderland
- run all known spelling errors in the corpus through the speller
- we're not ready yet for that - depends on new feature in ccat
- we're not ready yet for that - depends on new feature in ccat
- consider how to do a regression self-test
- no progress
- no progress
- document the AppleScript testing tool
- not finished
- not finished
- write tools for statistical analysis of test results
- started conversion to xml, handed the task over to Saara
- started conversion to xml, handed the task over to Saara
- make improved smj speller (incl. derivations and compounds)
-
Tomi worked on this one
-
Tomi worked on this one
-
fix bugs!
- other:
- compiled new speller
- other speller-related work
- read up on OOo speller alternatives
- compiled new speller
Steinar
- Beta testing: Align manually (shorter texts)
- Manually mark speller test texts for typos (making them into gold standards),
- marked and tested some texts
- marked and tested some texts
- Complete the semantic sets in sme-dis.rle
- no work this week
- no work this week
- missing lists
- no work this week
- no work this week
- Look at the actio compound issue when adding from missing lists
- not done
- not done
- Align corpus manually
- fix bugs!
Thomas
- work with compounding
- worked and still working
- worked and still working
- Lack of lowering before hyphen: Twol rewrite.
- not done
- not done
- translate beta release docs to sme and smj
- not done
- not done
- Add potential speller test texts
- not done
- not done
-
fix bugs!
- not any this week
Tomi
- make improved smj speller (incl. derivations and compounds)
- add Makefile target for PLX conversion of lexc files
- make PLX conversion test sample; add conversion testing to the make file
- improve number PLX conversion
- update ccat to handle error/correction markup
- add version info to the generated speller lexicons
- fix bugs!
Trond
- Test the beta versions
- Done, added wds.
- Done, added wds.
- Work on the parallel corpus issues
- Nothing this week
- Discuss with Anders
- Nothing this week
- Nothing this week
- Work on the aligner with (Børre)
- Nothing this week
- Nothing this week
- fix sme texts in corpus this month
- find missing nob parallel texts in corpus, go through Saara's list
- Nothing this week
- Postpone these tasks to after the beta:
- update the smj proper noun lexicon, and refine the morphological
- Go through the Num bugs
- update the smj proper noun lexicon, and refine the morphological
- Improve automatic alignment process
- Align corpus manually
- Add potential speller test texts
- collect a list of PR recipients
- fix bugs!.
3. Documentation
The open documentation issues fall into these three categories:
- Beta documentation for testers
- Documentation for the online corpora
- General documentation improvement after Steinar's test (for open-source
TODO:
- write form to request corpus user account (Børre, Sjur, Trond)
- delayed till after the beta release
- delayed till after the beta release
- document how to apply for access to closed corpus, and details on the corpus
- delayed till after the beta release
- delayed till after the beta release
- correct and improve it based on feedback from Steinar ( Børre)
- low priority
- low priority
- beta documentation (see separate beta section below)
4. Corpus gathering
Trond added one new text from Harald Gaski
TODO:
-
sme texts: no new additions, fix corpus errors during this month
- missing nob parallel texts should be added if such holes are found
- Go through the list of missing or errouneous nob texts, based upon
- add sma texts to the corpus repository (Børre)
- add missing words from new corpus text (Maaren)
Go to victorio, cd gt/sme ccat -l sme zcorp/bound/sme/facta/Girjjalasv_doahpagat.doc.xml | preprocess --abbr=bin/abbr.txt --corr=src/typos.txt | lo | grep '\?' | cut -f1 | sort | uniq -c | sort -nr | less
5. Corpus infrastructure
Alignment
TODO
- go through other directories (nob dicrectories, sd directories), fix
- Improve the automatic process:
- Improve the anchor list and realign (Trond, Børre)
- Only adding words does not improve alignment, you have to consider the format
- The documents have still some formatting issues which cause trouble in
- Test and improve settings in the aligner
- Improve the anchor list and realign (Trond, Børre)
- Align manually (Trond, Steinar) (especially shorter terminological texts)
6. Infrastructure
TODO:
- update and fix our documentation and infrastructure as Steinar finds
- started, working on it
7. Linguistics
North Sámi
Maaren is working on actio compounds, using the info found in Nickel.
LEXICON BOAHTALADDAN ! not compounding LEXICON BOAHTIN ! compounding, like: boahtin-
TODO:
- lexicalise actio compounds. Example: vuolggasadji vs. vuolginsadji
- fix stuorra-oslolaš lower case o ( Sjur, Thomas, Trond)
- postponed till after the public beta
Lule Sámi
TODO:
- refine smj proper noun lexica, cf. the propernoun-smj-lex.txt
8. Name lexicon infrastructure
Decisions made in Tromsø can be found in this meeting memo.
TODO:
- fix bugs in lexc2xml; add comments to the log element (Saara)
- finish first version of the editing (Sjur)
- test editing of the xml files. If ok, then: ( Sjur, Thomas, Trond)
- make terms-smX.xml <=== automatically from propernoun-sme-lex.xml (add nob as
- convert propernoun-($lang)-lex.txt to a derived file from common xml files
- implement data synchronisation between risten.no and
- start to use the xml file as source file
- clean terms-sme.xml such that all names have the correct tag for their use
- merge placenames which are errouneously in different entries: e.g. Helsinki,
- publish the name lexicon on risten.no (Sjur)
- add missing parallel names for placenames (linguists)
- add informative links between first names like Niillas and Nils
9. Spellers
OOo speller(s)
TODO after the MS Office Beta is delivered:
- add Aspell/Hunspell data generation to the lexc2xspell (Tomi - after the
- study Hunspell, perhaps also Soikko (Børre, Sjur, Tomi)
- Aspell is not suitable
- Soikko is not a viable alternative either (they are looking for replacing
- HunSpell will be our choice for now
- Aspell is not suitable
Testing
Selecting test texts
In principle, we need the same text types as the ones we already aim at in our
- Min Áigi net issue
- Blogs
- Our own linguistic texts
- New department texts from the net
Storing test texts
TODO:
- Manually mark test texts for typos (making them into gold standards)
-
erorr§errors
- when correcting to multiple strings: erorr.Og§(error. Og)
- update correction markup xml conversion to handle the second case
- update correction markup xml conversion to handle the second case
-
erorr§errors
- Add the marked test texts to the corpus/prooftest/orig/ catalogue
- done
- done
- change ccat to handle error/correction markup (Tomi)
- extract the document text with original errors (input to standard speller
- extract the document text with the available corrections (the correct docu),
- extract all and only the spelling errors with their corrections, in a tab
- extract both the original text and their corrections, in a tab
- extract the document text with original errors (input to standard speller
- Set up ways of adding meta-information (source info, used in testing or not,
- add the wanted xml elements to the XSL header (Saara) (source info is
- outworned (ie not suitable for speller testing any more, only for regression
- lexicalised (all unknown, correctly spelled words added to lexicon)
- outworned (ie not suitable for speller testing any more, only for regression
- add the wanted xml elements to the XSL header (Saara) (source info is
- Conduct tests on new beta versions on the basis of the unspoiled gold standard
- alternatively: make test scripts that will run the tests automatically,
- include the ones already tested in the testing/ catalogue
- test 0.3 on the same texts
- test each version before beta release
Testing tools
TODO:
- document the AppleScript testing tool (Sjur)
- not finished
- not finished
- write tools for statistical analysis of test results (Sjur)
- started, Saara continued, will continue with Forrest integration
Regression tests
TODO:
- add extraction of all known spelling errors in the corpus (not the
- test the typos.txt list, and check that all entries are properly corrected
- consider how to do a regression self-test, ie, how to test the full
Localisation
We need to translate the info added to our front page (and a separate page)
TODO:
- translate beta release docs to sme ( Thomas)
- translate beta release docs to smj ( Thomas)
Lexicon conversion to the PLX format
TODO:
- add gt/cwb/paradigm.smj.txt file into gt/script/server_anl.pl
- done
- done
- add prefixes to the PLX (Børre)
- middle nouns (Børre)
- add Makefile target for PLX conversion of lexc files (Tomi):
- done
- adjectives
- nouns
- propernouns
- verbs derived into other POSes
- verbs - must be done on gtsvn.uit.no
- produced by the paradigm server on victorio? or regenerate every night, and
- produced by the paradigm server on victorio? or regenerate every night, and
- done
- make conversion test sample; add conversion testing to the make file
- improve number conversion (Børre, Tomi)
- compile verb lists on victorio, make them available on our public server
- speed issue with smj verb derivations - and this is only derivations to
Public Beta release
Due to the problems with generating the PLX files discussed above, we need to
Linguistic issues still open:
- prefixes (eahpe, ii-) (Børre)
- middle nouns (LEXICON: lexc: Rmiddle, plx: L) (Børre)
DONE:
- delivered PLX data of sme and smj including compounding
- translated Windows installer to sme and smj
- installed PLX compiler in G5 at /usr/local/bin/mklex* (one version for
- added resources needed for compiling PLX lexicons to our cvs repo
- tested the beta drop from Polderland - good we did, it is absolutely
- questions for Polderland:
- version info in the speller
- done
- done
- remaking/updating the installer packages with linguistic updates
- we have received complete installer packages
- we have received complete installer packages
- version info in the speller
- add compilation of MS Office spellers part of the Makefile
- install Windows and MS Office; test tools on Windows
TODO:
- improved smj speller (incl. derivations and compounds) (Sjur, Tomi)
- finish press release (Sjur)
- add info to front page (incl. download links) (Børre)
- write separate page with detailed info (incl. download links) (Børre)
- translate press release, web pages (Børre, Thomas, whoever)
- collect a list of PR recipients, forward to Berit Karen Paulsen
Version identification of speller lexicons
TODO:
- add version info to the generated speller lexicons (Børre, Sjur, Tomi)
10. Other
Project meeting IRL
The planned gathering will have to be on 16.-20.4., in Tromsø, most likely. All
Corpus contracts
TODO:
- publish corpus contracts and project infra as open-source on NoDaLi-sta
- delayed until the public beta is out the door
Updates:
- MacOS: 10.4.9
- SubEthaEdit: 2.6.2
Bug fixing
48 open Divvun/Disamb bugs, and 23 risten.no bugs
Easter
- Børre: away
- Maaren:
- Saara: working
- Sjur: working
- Thomas: away 2/4 - 4/4
- Tomi: At work, in Karigasniemi
- Trond:
11. Next meeting, closing
The next meeting is 2.4.2007, 09: 30 Norwegian time.
The meeting was closed at 10: 49.
Appendix - task lists for the next week
Boerre
- update and fix our documentation and infrastructure as Steinar finds
- find missing nob parallel texts in corpus
- add prefixes to the PLX conversion
- improve number PLX conversion
- add sma texts to the corpus repository
- improve automatic alignment process
- add potential speller test texts
- collect a list of PR recipients, forward to Berit Karen Paulsen
- add version info to the generated speller lexicons
- run all known spelling errors in the corpus through the speller
- test the typos.txt list, and check that all entries are properly corrected
- consider how to do a regression self-test
- add extraction of all known spelling errors in the regular corpus (not the
- fix bugs!
Maaren
- lexicalise actio compounds
- Manually mark speller test documents for typos
Saara
- prepare more files for manual alignment
- mark-up the added speller test texts, using our existing xml format
- improve cgi-bin scripts
- add new features to the paradigm generator
- add new features to the paradigm generator
- add new XSL/XML headers for proofing test docs
- continue with speller test data
- compilation of verb lists
- speed in smj conversion
- fix bugs!
Sjur
- hire linguist
- finish press release for the beta
- collect a list of PR recipients
- add version info to the generated speller lexicons
- run all known spelling errors in the corpus through the speller
- consider how to do a regression self-test
- document the AppleScript testing tool
- write tools for statistical analysis of test results
- make improved smj speller (incl. derivations and compounds)
- fix bugs!
Steinar
- Beta testing: Align manually (shorter texts)
- Manually mark speller test texts for typos (making them into gold standards),
- Complete the semantic sets in sme-dis.rle
- missing lists
- Look at the actio compound issue when adding from missing lists
- Align corpus manually
- fix bugs!
Thomas
- work with compounding
- Lack of lowering before hyphen: Twol rewrite.
- translate beta release docs to sme and smj
- Add potential speller test texts
- fix bugs!
Tomi
- make improved smj speller (incl. derivations and compounds)
- make PLX conversion test sample; add conversion testing to the make file
- improve number PLX conversion
- update ccat to handle error/correction markup
- add version info to the generated speller lexicons
- fix bugs!
Trond
- Test the beta versions
- Work on the parallel corpus issues
- Discuss with Anders
- Work on the aligner with (Børre)
- fix sme texts in corpus this month
- find missing nob parallel texts in corpus, go through Saara's list
- Discuss with Anders
- Postpone these tasks to after the beta:
- update the smj proper noun lexicon, and refine the morphological
- Go through the Num bugs
- update the smj proper noun lexicon, and refine the morphological
- Improve automatic alignment process
- Align corpus manually
- Add potential speller test texts
- collect a list of PR recipients
- fix bugs!.