Meeting_2007-08-13

Meeting setup

  • Date: 13.8.2007
  • Time: 09.30 Norw. time
  • Place: Internet
  • Tools: SubEthaEdit, iChat/Skype

Agenda

  1. Opening, agenda review
  2. Reviewing the task list from last week
  3. Documentation - divvun.no
  4. Corpus gathering
  5. Corpus infrastructure
  6. Infrastructure
  7. Linguistics
  8. name lexicon infrastructure
  9. Spellers
  10. Other issues
  11. Summary, task lists
  12. Closing

1. Opening, agenda review, participants

Opened at 10: 00.

Present: Børre, Maaren, Per-Eric, Sjur, Thomas, Tomi

Absent: Saara, Trond

Agenda accepted as is.

2. Updated task status since last meeting

Børre

  • add sma texts to the corpus repository
    • not done
  • run all known spelling errors in the prooftest corpus through the speller
    • not done
  • add extraction of all known spelling errors in the regular corpus (not the prooftest corpus), and check that they are properly corrected
    • not done
  • update and fix our documentation and infrastructure
    • not done
  • follow-up contact with Davvi Girji
    • Sent e-mail, no answer so far
  • fix bugs!
    • Fixed #454
  • Other
    • Trying to port sme from Xerox tools to SFST
    • Trying to fix the G5, which has caused us some trouble lately.

Maaren

  • lexicalise actio compounds
    • not done

Per-Eric

  • expand the smj typos list
    • work and still working
  • add missing smj words
    • work and still working
  • fix errors in smj/src/typos.txt
    • working with it
  • lexicalise words from the Olavi missing list
    • work and still working
  • test new speller for actios of 3-syllable verbs, adverbs of 3-s adjs and æ/ä
    • done some tests
  • add compounding tags to adjectives
    • done
  • ask SD/Sweden about smj place names (Per-Eric)
    • not done

Saara

  • add new XSL/XML headers for proofing test docs
  • Try to add files with Lars to the corpus interface.
  • fix bugs!

Sjur

  • run all known spelling errors in the corpus through the speller
    • not done
  • document the AppleScript testing tool
    • not done
  • integrate regression self tests with the make file
    • not done
  • improve speller test bench
    • working on it
  • integrate the ccat speller testing options in the make file
    • working on it
  • fix internet setup for Per-Eric's satelite modem
    • not done
  • check with Sámi publishing houses whether support for CS2 is still needed
    • one publisher still open, but I expect it won't be a problem; will check this week.
  • publish corpus contracts and project infra as open-source on NoDaLi-sta
    • not done
  • ask Trond whether Ilona can add the sme place names from Finland
    • not done
  • fix bugs!

Thomas

  • work with compounding
    • still working
  • fix stuorra-oslolaš lower case o
    • not done
  • test new speller for actios of 3-sybbable verbs, adverbs of 3-s adjs and æ/ä
    • actios and adverbs are okey, æ/ä does not work, do not now why
  • fix bugs!
    • not done

Tomi

  • make PLX conversion test sample; add conversion testing to the make file
    • not done
  • open up compounding for all actios
    • done
  • add Hunspell data generation/conversion
    • not done
  • add G5 bug to Bugzilla
    • added
  • make baseform extraction tool
    • done
  • fix bugs!

Trond

  • Work on the web corpus issues
  • update the smj proper noun lexicon, and refine the morphological analysis, cf. the propernoun-smj-lex.txt
  • fix bugs!.

3. Documentation

TODO:

  • write form to request corpus user account (Børre, Sjur, Trond)
  • document how to apply for access to closed corpus, and details on the corpus and its use in general (Børre, Sjur, Trond)
  • correct and improve documentation (infrastructure) (Børre)

4. Corpus gathering

Per-Eric will meet Sigga Tuolja-Sandström on his way home from Östersund.

Torkel Rasmussen will give texts to us, he will be in Tromsø in week 39-40, and transfer texts from his computer to ours then.

TODO:

  • Go through the list of missing or errouneous nob texts, based upon Saara's perfect list (Trond)
  • add sma texts to the corpus repository (Børre)

5. Corpus infrastructure

Nothing this week either.

6. Infrastructure

The G5 is down/off the net after a system update, Børre is working on it.

TODO:

  • update and fix our documentation and infrastructure (Børre)
  • fix internet setup for Per-Eric's satelite modem (Sjur)

7. Linguistics

North Sámi

The twol problems in sme were "solved" by revering the changes that caused them. Thus, it works now, but with the flaws that started the whole process.

Thomas has finished compounding markup - very good!

The sme names from Finland is still not added to the lexicon (Børre has them). This could be done by Ilona. Sjur will ask Trond.

TODO:

  • lexicalise actio compounds. Example: vuolggasadji vs. vuolginsadji ( Maaren)
    • possibly turn on free compounding as part of the PLX conversions (ie free compounding in the spellers, but not in the analyzers/transducers)
      • Tomi added this
  • fix stuorra-oslolaš lower case o ( Sjur, Thomas, Trond)
  • open up compounding for all actios in the PLX format (Tomi)
    • done
  • ask Trond whether Ilona can add the sme place names from Finland ( Sjur)

Lule Sámi

The æ-ä alternation issue has turned into an interesting direction. With the latest speller (29.6.), it behaves like the following: 

dæbbaga -  ok
däbbaga -  däbboga
           dæbbaga
           dæbbaga--
           dibága
           dibága--
           dubága
           dubága--
           tubága
vællahit - ok
vällahit - vællahit
           vellahit
gæhtjáj  - ok
gähtjáj  - ok

That is, ä works in some cases, but not in other. æ seems to work everywhere it should.

Nils-Olof Sortelius at Sámediggi in Sweden ( smj place names) is in hospital - Per-Eric will talk to Kåre Tjikkom instead.

TODO:

  • refine smj proper noun lexica, cf. the propernoun-smj-lex.txt ( Thomas, Trond)
  • ä/æ in speller (Thomas, Sjur)
    • does only work partially, see data above.
  • lexicalise words from the Olavi missing list, but check against the pdf original where in doubt (Per-Eric)
    • still some to do
  • actios of 3-syllable verbs must be checked in the next speller ( Thomas, Per-Eric)
    • they are ok
  • adverbs of 3-syllable adjectives must be tested in the next speller ( Thomas, Per-Eric)
    • they are ok
  • add compounding tags to (some weeks of work):
    • nouns (Thomas)
    • adjs (Per-Eric)
  • ask SD/Sweden about smj place names (Per-Eric)
  • call Samuel Gælok about texts (Per-Eric)
  • call Kåre Tjikkom about smj place names and correct-doc (Per-Eric)

8. Name lexicon infrastructure

Decisions made in Tromsø can be found in this meeting memo.

TODO:

  1. fix bugs in lexc2xml; add comments to the log element (Saara)
  2. finish first version of the editing (Sjur)
  3. test editing of the xml files. If ok, then: ( Sjur, Thomas, Trond)
  4. make terms-smX.xml <=== automatically from propernoun-sme-lex.xml (add nob as well) (the morphological section should be kept intact, in e.g. propernoun-sme-morph.txt) (Sjur, Saara)
  5. convert propernoun-($lang)-lex.txt to a derived file from common xml files ( Sjur, Tomi, Saara)
  6. implement data synchronisation between risten.no and the cvs repo, and possibly other servers (ie the G5 as an alternative server to the public risten.no - it might be faster and better suited than the official one; also local installations could be treated the same way)
  7. start to use the xml file as source file
  8. clean terms-sme.xml such that all names have the correct tag for their use (e.g. @type=secondary) (Thomas, Maaren, linguists)
  9. merge placenames which are errouneously in different entries: e.g. Helsinki, Helsingfors, Helsset (linguists)
  10. publish the name lexicon on risten.no (Sjur)
  11. add missing parallel names for placenames (linguists)
  12. add informative links between first names like Niillas and Nils ( linguists)

9. Spellers

OOo spellers

Tomi is working on the lexicon conversion to the Hunspell format. Some issues to solve. Software problems on the G5 - will be added to Bugzilla.

TODO:

  • add Hunspell data conversion (Tomi)
  • add G5 bug to Bugzilla (Tomi)
    • done, solved - add note to the bug report about the actual problem (environment error on G5 account) (Tomi)

Testing

Spelling Error Markup

Steinar finished his job last week, and copied it all over to victorio. There is still some clean-up to do, mainly because the markup is added to the xml files, and not the originals. The present conversion process doesn't handle error markup in this format.

TODO:

  • Set up ways of adding meta-information (source info, used in testing or not, added to lexicon or not) (Saara)
  • move the error markup in the xml files to (a copy of) the original ( Børre)

Testing tools

TODO:

  • document the AppleScript testing tool (Sjur)
  • improve speller test bench (Sjur)
    • integrate the ccat speller testing options in the Makefile (Sjur)

Regression tests

Tomi utilized oe of the java classes in the PLX conversion tool to create a separate tool/command to extract the baseforms from the lexc lexicons.

TODO:

  • add extraction of all known spelling errors in the corpus (not the prooftest corpus), and check that they are properly corrected ( Børre, Sjur)
  • how to do a regression self-test:
    • extract all the base forms in the lexicon, and run them through the speller
    • extract all SUB-marked entries, and run them through the speller
      • integrate these in the make file (Sjur)
    • make baseform extraction tool (Tomi)
      • done

Lexicon conversion to the PLX format

It seems that we don't need to get new licenses from either Polderland or Xerox - Børre's nice ssh solution is working very well for smj. There was a problem with sme that needs to be investigated.

TODO:

  • ask for mklex for Linux (victorio) from Polderland (Sjur)
    • delayed until the present ssh solution is tested more
  • ask Xerox for a commercial lisense for the xfst tools on the G5 (Sjur)
    • delayed until the present ssh solution is tested more

Publicity follow-up

Per-Eric is being interviewed next week by Anna Sunna (SR Sámi Radio), and will be demonstrating installation and usage of the proofing tools.

It was cancelled by SR.

New public beta

Delayed till the majority of the present bugs are fixed.

10. Other

Corpus contracts

TODO:

  • publish corpus contracts and project infra as open-source on NoDaLi-sta ( Sjur)

Bug fixing

When fixing bugs, record the version number containing the fix in the Bugzilla bug report, such that for each bug, we know exactly when it should have been fixed, in what file(s) and what version.

56 open Divvun/Disamb bugs (21 of these 56 are speller bugs, 35 are general bugs), and 23 risten.no bugs

TODO:

  • look over the Bugzilla status mails (Børre)

Board meeting

Sjur is going to Oslo on Thursday Aug. 16 to meet the board.

Project meeting

It would be good to have another project meeting sometime in September, to work on the hardest remaining issues. We'll decide the date next week.

11. Next meeting, closing

The next meeting is 20.8.2007, 09: 30 Norwegian time.

The meeting was closed at 10: 57.

Appendix - task lists for the next week

Boerre

  • add sma texts to the corpus repository
  • run all known spelling errors in the prooftest corpus through the speller
  • add extraction of all known spelling errors in the regular corpus (not the prooftest corpus), and check that they are properly corrected
  • update and fix our documentation and infrastructure
  • follow-up contact with Davvi Girji
  • fix bound corpus access for external users (documentation, application)
  • move error markup in xml files to (a copy of) the originals
  • fix bugs!

Maaren

  • lexicalise actio compounds

Per-Eric

  • expand the smj typos list
  • add missing smj words
  • fix errors in smj/src/typos.txt
  • lexicalise words from the Olavi missing list
  • test new speller for actios of 3-sybbable verbs, adverbs of 3-s adjs and æ/ä
  • add compounding tags to adjectives
  • ask SD/Sweden about smj place names
  • call Kåre Tjikkom about smj place names and correct-doc
  • call Samuel Gælok about texts

Saara

  • add new XSL/XML headers for proofing test docs
  • Try to add files with Lars to the corpus interface.
  • fix bugs!

Sjur

  • run all known spelling errors in the corpus through the speller
  • document the AppleScript testing tool
  • integrate regression self tests with the make file
  • improve speller test bench
  • integrate the ccat speller testing options in the make file
  • fix internet setup for Per-Eric's satelite modem
  • check with Sámi publishing houses whether support for CS2 is still needed
  • publish corpus contracts and project infra as open-source on NoDaLi-sta
  • fix bound corpus access for external users (documentation, application)
  • fix stuorra-oslolaš lower case o
  • ä/æ in smj speller
  • restart work on the XML name editor/risten.no integration
  • plan the rest of the project period
  • fix bugs!

Thomas

  • work with compounding
  • fix stuorra-oslolaš lower case o
  • test new speller for æ/ä
  • fix bugs!

Tomi

  • make PLX conversion test sample; add conversion testing to the make file
  • add Hunspell data generation/conversion
  • fix bugs!

Trond

  • Work on the web corpus issues
  • update the smj proper noun lexicon, and refine the morphological analysis, cf. the propernoun-smj-lex.txt
  • fix bound corpus access for external users (documentation, application)
  • fix stuorra-oslolaš lower case o
  • fix bugs!.