Meeting_2007-05-07

Meeting setup

  • Date: 07.05.2007
  • Time: 09.30 Norw. time
  • Place: Internet
  • Tools: SubEthaEdit, iChat

Agenda

  1. Opening, agenda review
  2. Reviewing the task list from last week
  3. Documentation - divvun.no
  4. Corpus gathering
  5. Corpus infrastructure
  6. Infrastructure
  7. Linguistics
  8. name lexicon infrastructure
  9. Spellers
  10. Other issues
  11. Summary, task lists
  12. Closing

1. Opening, agenda review, participants

Opened at 00: 48.

Present: Børre, Maaren, Per-Eric, Sjur, Steinar, Thomas, Tomi, Trond

Absent: Saara

Agenda accepted as is.

2. Updated task status since last meeting

Børre

  • add sma texts to the corpus repository
    • not done
  • collect a list of PR recipients, forward to Berit Karen Paulsen
    • not done
  • run all known spelling errors in the prooftest corpus through the speller
    • not done
  • add extraction of all known spelling errors in the regular corpus (not the prooftest corpus), and check that they are properly corrected
    • not done
  • update installer packages with latest speller lexicon
    • still no new speller lexicon
  • add numbers, compound restrictions to both spellers if time permits
    • not done
  • update and fix our documentation and infrastructure as Steinar finds problem areas - low priority
    • not done
  • find missing nob parallel texts in corpus
    • not done
  • study the Hunspell formalism in detail together with Sjur and Tomi
    • nothing new
  • add info to front page (incl. download links)
    • not done
  • write separate page with detailed info (incl. download links) (Børre)
    • a separate page for the beta speller, with installation instructions, etc.
      • not done
  • translate press release, web pages (Børre, Thomas, whoever)
    • not done
  • fix internet setup for Per-Eric's satelite modem
  • ask MacOffice for larger disks for the G5
    • not done
  • ask for larger disks for victorio
    • done, Roy Dragseth will look at it
  • ask for newer server OS on victorio, many of the installed tools are quite old
    • under consideration,
  • fix bugs!
    • not done

Maaren

  • lexicalise actio compounds
    • done little
  • Manually mark speller test documents for typos
    • working

Per-Eric

  • expand the smj typos list
  • add missing smj words

Saara

  • prepare more files for manual alignment
    • all files available are now aligned
  • improve cgi-bin scripts
    • add new features to the paradigm generator
      • not done, will be done this week with documentation
  • add new XSL/XML headers for proofing test docs
  • compilation of verb lists
  • read the manual for graphical corpus interface and try to add files with Lars.
    • read the manual, will add files after some fixes
  • fix bugs!
  • other
    • submitted the law article

Sjur

  • finish press release for the beta
    • not done
  • collect a list of PR recipients
    • not done
  • run all known spelling errors in the corpus through the speller
    • not done
  • document the AppleScript testing tool
    • not done
  • integrate regression self tests with the make file
    • not done
  • make improved smj speller (incl. derivations and compounds)
    • took most of my time, but it still won't compile - I feel like banned by the speller gods, and by my G5 account...
  • improve speller test bench
    • not done
  • update installer packages with latest speller lexicon
    • not done
  • integrate the ccat speller testing options in the make file
    • not done
  • fix internet setup for Per-Eric's satelite modem
    • bugged the Swedish company making the modem, received some very initial documentation, not very helpful so far. I have asked them to send more.
  • ask for Linux version of the Polderland command-line speller (for victorio)
    • no Polderland meeting last week (our contact person was on vacation)
  • ask for mklex for Linux (victorio) from Polderland (for victorio)
    • no Polderland meeting last week (our contact person was on vacation)
  • look over the Bugzilla status mails
    • asked Børre to do this
  • fix bugs!
  • other:
    • read through and commented Saara's law article

Steinar

  • Beta testing: Align manually (shorter texts)
  • Manually mark speller test texts for typos (making them into gold standards), add the texts to the orig/catalogue
    • added marked news-files, working with facta-files
  • include the files already publically tested into the prooftest cataloge
    • not done yet
  • Complete the semantic sets in sme-dis.rle
    • no work lately
  • missing lists
    • no work lately
  • Look at the actio compound issue when adding from missing lists
    • not done
  • Align corpus manually
  • fix bugs!

Thomas

  • work with compounding
    • worked and still working
  • Lack of lowering before hyphen: Twol rewrite.
    • not done
  • translate beta release docs to sme and smj
    • not done
  • Add potential speller test texts
    • not done
  • fix bugs!
    • assisted

Tomi

  • make improved smj speller (incl. derivations and compounds)
    • not done
  • add numbers, compound restrictions to both spellers if time permits
  • make PLX conversion test sample; add conversion testing to the make file
    • not done
  • improve number PLX conversion
    • not done
  • improve prefix and middle-noun PLX conversion
    • not done
  • integrate the ccat speller testing options in the Makefile
    • not done
  • fix bugs!
    • fixed some
  • other:
    • improved plx conversion
      • added hyphen conversion to fst
      • added hard hyphens to fst

Trond

  • Work on the parallel corpus issues
    • Not this week.
  • Postpone these tasks to after the beta:
    • update the smj proper noun lexicon, and refine the morphological analysis, cf. the propernoun-smj-lex.txt
    • Go through the Num bugs
  • collect a list of PR recipients
    • Not done.
  • check with Lars when more aligned texts are ready
    • Still open.
  • ask for larger disks for victorio
    • Done, new disc offer forthcoming.
  • ask for newer server OS on victorio, many of the installed tools are quite old
    • Done, forthcoming.
  • fix bugs!.

3. Documentation

Nothing new.

The open documentation issues fall into these three categories:

  • Beta documentation for testers
  • Documentation for the online corpora
  • General documentation improvement after Steinar's test (for open-source release)

TODO:

  • write form to request corpus user account (Børre, Sjur, Trond)
    • delayed till after the beta release
  • document how to apply for access to closed corpus, and details on the corpus and its use in general (Børre, Sjur, Trond)
    • delayed till after the beta release
  • correct and improve it based on feedback from Steinar ( Børre)
    • low priority
  • beta documentation (see separate beta section below)

4. Corpus gathering

Nothing new.

TODO:

  • sme texts: no new additions, fix corpus errors during this month ( Børre, Trond, Saara)
  • missing nob parallel texts should be added if such holes are found ( Børre, Trond)
  • Go through the list of missing or errouneous nob texts, based upon Saara's perfect list (Børre, Trond)
  • add sma texts to the corpus repository (Børre)

5. Corpus infrastructure

Alignment

All parallell texts are now aligned. Lars has made some work in Oslo, but no visible changes to the pages yet.

TODO

  • check with Lars when more texts are ready (Trond)

6. Infrastructure

The http://giellatekno.uit.no main page has been restructured, and Saara's nice paradigm generator has been given a more visible place. The North Sámi generator is in place, but the Lule Sámi one still has not been implemented. On the downside, the online wordform generators do not work, and all the links now accidently go to the English rather than the Sámi pages..

TODO:

  • update and fix our documentation and infrastructure as Steinar finds problem areas (Børre)
  • fix internet setup for Per-Eric's satelite modem (Sjur, Børre)
    • this influences iChat, SEE sharing, and ARD connetions
  • Add a Lule Sámi paradigm generators, and link the p-smj.en.html page to it. (Saara)
  • Translate the paradigm pages to Sámi, fix all the se links (Børre, Trond)

7. Linguistics

North Sámi

Actio compounds not being allowed to compound creates problems for constructions like: boahtaladdan- ja vuolggasadji. We should probably look into whether we need to open up for actio compounds again, and the disambiguator people should then find another solution to their problem.

TODO:

  • lexicalise actio compounds. Example: vuolggasadji vs. vuolginsadji ( Maaren)
    • almost all done
  • fix stuorra-oslolaš lower case o ( Sjur, Thomas, Trond)
    • postponed till after the public beta

Lule Sámi

Buorrek issue: -k clitic (or abessive?) Clitic issue seems phonological in lulesámi: tjuvdek but not *tjuvdevk, *tjuvdijk, *tjuvdijnk tjuvdijn ge instead of tjuvdijnk?

What is a clitic? Member of LEXICON K.

  1. totally uncritical when it comes to hosts (accept all hosts) <===
  2. does not (usually) interfere with the phonology of the host

If you may add -k to all wordforms pointing to the lexicon K today, then -k should stay in K. If no, it must go.

North sami: we use K instead of #, since basically all words (and word forms) may get -go, -ba, -ge, etc.

lijge, lage, muorrage - buorrek, *lak

Say, for the sake of the argument that

  • -k added to nominal forms
  • other K members to any wordform
  • ==> calls for split of K lexicon. 
LEXICON K
 ENDLEX ;
! +Clt:clitic ENDLEX ;
 +Clt+ge:#ge ENDLEX ;
 +Clt+gen:#gen ENDLEX ;
 +Clt+ga:#ga ENDLEX ;
 +Clt+k:#k ENDLEX ;

TODO:

  • refine smj proper noun lexica, cf. the propernoun-smj-lex.txt ( Thomas, Trond)
  • research the –k ending, whether it is a clitic or a regular inflection, and where it applies (Per-Eric, Thomas)

8. Name lexicon infrastructure

Decisions made in Tromsø can be found in this meeting memo.

TODO:

  1. fix bugs in lexc2xml; add comments to the log element (Saara)
  2. finish first version of the editing (Sjur)
  3. test editing of the xml files. If ok, then: ( Sjur, Thomas, Trond)
  4. make terms-smX.xml <=== automatically from propernoun-sme-lex.xml (add nob as well) (the morphological section should be kept intact, in e.g. propernoun-sme-morph.txt) (Sjur, Saara)
  5. convert propernoun-($lang)-lex.txt to a derived file from common xml files ( Sjur, Tomi, Saara)
  6. implement data synchronisation between risten.no and the cvs repo, and possibly other servers (ie the G5 as an alternative server to the public risten.no - it might be faster and better suited than the official one; also local installations could be treated the same way)
  7. start to use the xml file as source file
  8. clean terms-sme.xml such that all names have the correct tag for their use (e.g. @type=secondary) (Thomas, Maaren, linguists)
  9. merge placenames which are errouneously in different entries: e.g. Helsinki, Helsingfors, Helsset (linguists)
  10. publish the name lexicon on risten.no (Sjur)
  11. add missing parallel names for placenames (linguists)
  12. add informative links between first names like Niillas and Nils ( linguists)

9. Spellers

OOo speller(s)

TODO after the MS Office Beta is delivered:

  • add Hunspell data generation to the lexc2xspell (Tomi - after the PLX data generation is finished)
  • study the Hunspell formalism in detail (Børre, Sjur, Tomi)

Testing

Spelling Error Markup

TODO:

  • Manually mark test texts for typos (making them into gold standards) ( Steinar)
    • continued
  • Set up ways of adding meta-information (source info, used in testing or not, added to lexicon or not)
  • Conduct tests on new beta versions on the basis of the unspoiled gold standard documents (whoever has time), and fill in data from testing (the testers: who?)
  • include the files already publically tested into the prooftest cataloge ( Steinar, Trond)
  • test each version before beta release

Testing tools

TODO:

  • document the AppleScript testing tool (Sjur)
  • improve speller test bench (Sjur)
    • integrate the ccat speller testing options in the Makefile (Sjur, Tomi)
  • ask for Linux version of the Polderland command-line speller (for victorio) ( Sjur)

Regression tests

TODO:

  • add extraction of all known spelling errors in the corpus (not the prooftest corpus), and check that they are properly corrected ( Børre, Sjur)
    • ccat now ready, it should be integrated in the Makefile (Sjur, Tomi)
  • test the typos.txt list, and check that all entries are properly corrected ( Børre, Sjur)
  • consider how to do a regression self-test, ie, how to test the full wordlist (Børre, Sjur)
    • extract all the base forms in the lexicon, and run them through the speller
    • extract all SUB-marked entries, and run them through the lexicon
      • integrate these in the make file (Sjur)

Localisation

We need to translate the info added to our front page (and a separate page) regarding the beta release. Also the press release needs to be translated.

TODO:

  • translate beta release docs to sme ( Thomas)
  • translate beta release docs to smj ( Thomas)

Lexicon conversion to the PLX format

Børre got the conversion working all the way to the final speller, for sme. A new speller lexicon is soon available for download. smj is next.

TODO:

  • ask MacOffice for larger disks for the G5 (Børre), and report about possible RAM-related problems, and the warranty on the RAM.
    • not yet
  • ask for larger disks for victorio (Børre, Trond)
    • discussed, in principle yes, reluctant to the schedule. But in the end, he would be willing to update the disks, but not the OS until June. He will send us an e-mail with costs for new disks.
  • ask for newer server OS on victorio, many of the installed tools are quite old (it is using Cent OS - a new version was released not long ago) ( Børre, Trond)
    • see previous point.
  • ask for mklex for Linux (victorio) from Polderland (Sjur)
    • not yet (no meeting with them last week)

Compounding restrictions

How to include compounding restriction comment tags in the transducers: 

giv0ri:giv'ri ALBMI ; !+SgNomCmp +SgGenCmp +PlGenCmp
=> (using a perl script or similar)
+SgNomCmp+SgGenCmp+PlGenCmpgiv0ri:giv'ri ALBMI ; !

TODO:

  1. improve prefix conversion to PLX (Tomi)
  2. improve middle noun conversion to PLX (Tomi)
  3. improve noun + adjective PLX conversion: ( Tomi)
    1. compounding stems - how do we generate them? Using the java client? +SgNomCmp+Cmpnd = sáme–, should give the correct compounding stem, shouldn't it? We want to optionally go from: sáme- NLI to sáme NL: - NLI (->) NL, which means we should be able to extract correct compounding stems using xfst methods only.
    2. compounding tags - we need to obey them when making the transducers. Suggestion - see above.
  4. make conversion test sample; add conversion testing to the make file ( Tomi)
    1. to regression test / QA the PLX conversion.
  5. improve number conversion (Børre, Tomi)
  6. ask for larger disk for the web server (Trond, Børre)

Public Beta release

TODO:

  • working and updated smj speller (Sjur, Tomi)
  • add numbers, compound restrictions to both spellers if time permits ( Børre, Tomi)
  • finish press release (Sjur)
  • add info to front page (incl. download links) (Børre)
    • download page made, only needs to add the speller beta when it is ready.
  • write separate page with detailed info (incl. download links) (Børre)
    • a separate page for the beta speller, with installation instructions, etc.
  • translate press release, web pages (Børre, Thomas, whoever)
  • collect a list of PR recipients, forward to Berit Karen Paulsen ( Børre, Sjur, Trond)
  • test final beta speller installers on Windows and Mac (Børre)
  • update installer packages with latest speller lexicon (Børre, Sjur)
    • we need to test the procedure, and make sure it works

10. Other

Corpus contracts

TODO:

  • publish corpus contracts and project infra as open-source on NoDaLi-sta ( Sjur)
    • delayed until the public beta is out the door

Bug fixing

35 open Divvun/Disamb bugs, and 23 risten.no bugs

TODO:

  • look over the Bugzilla status mails (Børre)
    • not yet

The next gathering

The Sámediggiráđđi meeting is moved one week. Thus, instead of around 13. June, it will be around 20. That's midsummer here in Finland: (

11. Next meeting, closing

The next meeting is 7.5.2007, 09: 30 Norwegian time.

The meeting was closed at 11: 30.

Appendix - task lists for the next week

Boerre

  • add sma texts to the corpus repository
  • collect a list of PR recipients, forward to Berit Karen Paulsen
  • run all known spelling errors in the prooftest corpus through the speller
  • add extraction of all known spelling errors in the regular corpus (not the prooftest corpus), and check that they are properly corrected
  • update installer packages with latest speller lexicon
  • add numbers, compound restrictions to both spellers if time permits
  • update and fix our documentation and infrastructure as Steinar finds problem areas - low priority
  • find missing nob parallel texts in corpus
  • study the Hunspell formalism in detail together with Sjur and Tomi
  • test the typos.txt list, and check that all entries are properly corrected
  • add info to front page (incl. download links)
  • write separate page with detailed info (incl. download links) (Børre)
    • a separate page for the beta speller, with installation instructions, etc.
  • translate press release, web pages (Børre, Thomas, whoever)
  • update installer packages with latest speller lexicon (Børre, Sjur)
  • fix internet setup for Per-Eric's satelite modem
  • ask MacOffice for larger disks for the G5
  • follow up the server OS on victorio
  • fix bugs!

Maaren

  • lexicalise actio compounds
  • Manually mark speller test documents for typos

Per-Eric

  • expand the smj typos list
  • add missing smj words
  • research the –k ending, whether it is a clitic or a regular inflection

Saara

  • improve cgi-bin scripts
    • add new features to the paradigm generator
    • paradigm generator for Lule Sámi
  • add new XSL/XML headers for proofing test docs
  • compilation of verb lists
  • read the manual for graphical corpus interface and try to add files with Lars.
  • fix bugs!

Sjur

  • finish press release for the beta
  • collect a list of PR recipients
  • run all known spelling errors in the corpus through the speller
  • document the AppleScript testing tool
  • integrate regression self tests with the make file
  • make improved smj speller
  • improve speller test bench
  • update installer packages with latest speller lexicon
  • integrate the ccat speller testing options in the make file
  • fix internet setup for Per-Eric's satelite modem
  • ask for Linux version of the Polderland command-line speller (for victorio)
  • ask for mklex for Linux (victorio) from Polderland (for victorio)
  • look over the Bugzilla status mails
  • fix bugs!

Steinar

  • Beta testing: Align manually (shorter texts)
  • Manually mark speller test texts for typos (making them into gold standards), add the texts to the orig/catalogue
  • include the files already publically tested into the prooftest cataloge
  • Complete the semantic sets in sme-dis.rle
  • missing lists
  • Look at the actio compound issue when adding from missing lists
  • fix bugs!

Thomas

  • work with compounding
  • Lack of lowering before hyphen: Twol rewrite.
  • translate beta release docs to sme and smj
  • Add potential speller test texts
  • research the –k ending, whether it is a clitic or a regular inflection
  • fix bugs!

Tomi

  • make improved smj speller (incl. derivations and compounds)
  • add numbers, compound restrictions to both spellers if time permits
  • make PLX conversion test sample; add conversion testing to the make file
  • improve number PLX conversion
  • improve prefix and middle-noun PLX conversion
  • integrate the ccat speller testing options in the Makefile
  • fix bugs!

Trond

  • Work on the parallel corpus issues
  • Add Steinars corr texts to the relevant catalogue
  • Postpone these tasks to after the beta:
    • update the smj proper noun lexicon, and refine the morphological analysis, cf. the propernoun-smj-lex.txt
    • Go through the Num bugs
  • collect a list of PR recipients
  • check with Lars when more aligned texts are ready
  • Follow up the server OS on victorio
  • fix bugs!.