Meeting_2007-11-05

Meeting setup

  • Date: 5.11.2007
  • Time: 09.30 Norw. time
  • Place: Internet
  • Tools: SubEthaEdit, iChat/Skype

Agenda

  1. Opening, agenda review
  2. Reviewing the task list from last week
  3. Documentation - divvun.no
  4. Corpus gathering
  5. Corpus infrastructure
  6. Infrastructure
  7. Linguistics
  8. name lexicon infrastructure
  9. Spellers
  10. Other issues
  11. Summary, task lists
  12. Closing

1. Opening, agenda review, participants

Opened at 10: 00.

Present: Børre, Ilona, Per-Eric, Risten, Sjur, Thomas, Tomi, Trond

Absent: none

Agenda accepted as is.

2. Updated task status since last meeting

Børre

  • move Steinar's error markup in the xml files to (a copy of) the original
    • Not done
  • adding support for the sami languages in OpenOffice.org
    • Made some locales
  • fix Unicode bug in Hunspell conversion java code
    • Done some more research, but it still persists
  • fix bug 550
    • not done
  • fix bugs!
    • not done
  • other:
    • Have prepared the G5 to take over the role as webserver instead of giellatekno.uit.no.
    • Installed Leopard on my machine
    • Added derivations to hunspell generation with a lot of help from Tomi. The problem with the approach we used is that sme.aff went from 2.4Mb to 34 Mb, so we'll have to come up with a better way of doing it.

Ilona

  • lexicalise smj missing words
    • Working on it.
  • add smj proper nouns
    • No list for it
  • other smj tasks
    • None

Maaren

  • lexicalise actio compounds

Per-Eric

  • lexicalise words from the Olavi missing list
    • worked and still working
  • fix bugs!
    • nothing done

Risten

  • finish the design/text for the CD and the cover
    • still working
  • set up CD-printing printer
    • Leif Åge has ordered a new printer

Saara

  • add new XSL/XML headers for proofing test docs
  • Set up ways of adding meta-information for proofing correct corpus docs (source info, used in testing or not, added to lexicon or not)
  • add nested error markup to xml conversion

Sjur

  • work on the XML name editor/risten.no integration
    • nope: (
  • set up risten.no on the G5 again
    • added eXist (ver. 1.1.1-newcore, actually a very old release, but stable)
  • test new and nested error markup
  • get command line hyphenator for automated testing of the hyph-lexicons
    • should arrive today
  • add hyphenation testing
    • waiting for the hyphenator
  • add hunspell testing
    • working on it
  • fix bug 550
    • nothing
  • fix bugs!
    • added, closed, commented
  • other tasks:
    • installed Leopard, made installation notes
    • tested hunspell manually in OOo
    • automated hunspell testing, added request for a new output format more suitable for usability testing (suggestion quality)

Thomas

  • sme->smj lexicon conversion to build bilingual lexicon resources
    • some done
  • add smj proper nouns
    • some done
  • check for bad hyphenation
    • not this week
  • look at test cases still not behaving properly
    • some done
  • fix bugs!
    • some done

Tomi

  • Hunspell lexicon conversion
    • worked
  • sme->smj lexicon conversion to build bilingual lexicon resources
    • not done
  • fix Unicode bug in Hunspell conversion java code
    • it is fixed ?
  • fix bugs!
    • fixed speller bugs

Trond

  • sme->smj lexicon conversion to build bilingual lexicon resources
    • Been working hard on this one, progress.
  • fix bugs!.
    • Not in Bugzilla last week : -(

3. Documentation

Nothing new.

TODO:

4. Corpus infrastructure

Nothing.

5. Infrastructure

Børre has been working on moving giellatekno.uit.no and www.divvun.no to the G5. The plan is to finish the work tomorrow.

Sjur has installed eXist on the G5 (but no risten.no yet).

6. Linguistics

North Sámi

Proper nouns and hyphens: when the name derives, like in -laš, we should get a hyphenation point in front of the derivation, but we don't. Example: 

Oslo > os^lo*laš
* = missing hyphenation point

sme $ lookup -flags mbTT -utf8 bin/hyph-sme.fst
0%>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>100%
oslolaš
oslolaš Os^lolaš

sme$lookup -flags mbTT -utf8 bin/hyphrules-sme.fst 
0%>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 100%
oslolaš
oslolaš os^lo^laš

One possible solutioln is to introduce a marker for derivational suffixes (and, by extension to other languages, also prefixes):

  • > - suffix
  • < - prefix

What about inflectional suffixes? > is used in sma and nob. This speaks against using it for derivation. 

root*1*der*2*infl
    %>     %>
Der1 Der2 Der3
C+   VC*  CVC

l u  tags, stem
l l  diacr  here
= 
g u  diacr  here
g l  wordform (w or w/o hyphen)

>: 0 - no hyphenation

If visisble hyphenation: >: - IFF _ C V >: 0 IFF _ C # 

HYPH = read regex ( @\"$(TARGET)/bin/hyphrules-$(TARGET).fst\"  .o. \
					@\"$(TARGET)/bin/hyph-i$(TARGET).save\"     .o. \
					@\"$(TARGET)/bin/$(TARGET)-norm.fst\" ) ; \n

SPLRNONREC = read regex [ [@\"$(TARGET)/bin/derivation-filter.fst\"     ] .o. \
						  [@\"$(TARGET)/bin/spellernonrec-$(TARGET).save\"  ] .o. \
						  [@\"common/bin/downcase.fst\"                 ] .o. \
						  [@\"common/bin/remove-hyphen.fst\"            ] .o. \
						  [@\"$(TARGET)/bin/hyphrules-$(TARGET).fst\".i ]     \
						] ; \n

$(TARGET)/bin/spellernonrec-$(TARGET).fst: \
							common/bin/tag-not-save.fst \
							common/bin/downcase.fst \
							$(TARGET)/bin/spellernonrec-$(TARGET).save \
							$(TARGET)/bin/derivation-filter.fst \
							common/bin/remove-hyphen.fst \
							$(TARGET)/bin/hyphrules-$(TARGET).fst

os^lolaš
os^lolaš        Oslo+N+Prop+Plc+Der1+Der/laš+A+Sg+Nom
os^lolaš        Oslo+N+Prop+Plc+Der1+Der/laš+A+SgGenCmp+Cmpnd
os^lolaš        Oslo+N+Prop+Plc+Der1+Der/laš+A+SgNomCmp+Cmpnd
os^lolaš        Oslo+N+Prop+Plc+Der1+Der/laš+A+Attr
os^lolaš        Oslo+N+Prop+Plc+Der1+Der/laš+A+Attr+Cmpnd

os^lo^laš
os^lo^laš       os^lo^laš       +?

In smj, also inflections fall outside our hyphenation rules: 

Basudissaj
Basudissaj      Ba^su^dissaj
Bájddárin
Bájddárin       Bájd^dárin
Bájddárin       Bájd^dárin
Heandarahkii
Heandarahkii    Hean^da^rahkii

TODO:

  • fix hyphenation of derivations (Thomas, Tomi, Sjur, Trond)
    • meeting today after lunch, 12: 30

Lule Sámi

TODO:

  • lexicalise words from the Olavi missing list, but check against the pdf original where in doubt (Per-Eric)
  • sme->smj lexicon conversion to build bilingual lexicon resources, and increase smj coverage (Trond, Thomas, Svenne)
  • sme->smj name conversion (Trond, Thomas)
  • add proper nouns (Thomas, Ilona)
  • look at missing baseforms (Thomas)

7. Name lexicon infrastructure

This sub-project needs to get up and running soon. Mainly Sjur's task.

Decisions made in Tromsø can be found in this meeting memo.

TODO:

  1. set up Tomcat and risten.no on the G5 again (Sjur, Børre)
    1. install risten.no
      1. eXist is now installed
  2. fix bugs in lexc2xml; add comments to the log element (Saara)
  3. finish first version of the editing (Sjur)
  4. test editing of the xml files. If ok, then: ( Sjur, Thomas, Trond)
  5. make terms-smX.xml <=== automatically from propernoun-sme-lex.xml (add nob as well) (the morphological section should be kept intact, in e.g. propernoun-sme-morph.txt) (Sjur, Saara)
  6. convert propernoun-($lang)-lex.txt to a derived file from common xml files ( Sjur, Tomi, Saara)
  7. implement data synchronisation between risten.no and the cvs repo, and possibly other servers (ie the G5 as an alternative server to the public risten.no - it might be faster and better suited than the official one; also local installations could be treated the same way)
  8. start to use the xml file as source file
  9. clean terms-sme.xml such that all names have the correct tag for their use (e.g. @type=secondary) (Thomas, Maaren, linguists)
  10. merge placenames which are errouneously in different entries: e.g. Helsinki, Helsingfors, Helsset (linguists)
  11. publish the name lexicon on risten.no (Sjur)
  12. add missing parallel names for placenames (linguists)
  13. add informative links between first names like Niillas and Nils ( linguists)

8. Proofing tools

Hunspell

TODO:

  1. Follow-up support for the sami languages in OpenOffice.org (Børre, Sjur)
  2. Hunspell lexicon conversion (Tomi, Børre)
  3. add hunspell testing to the make file (Sjur)
  4. debug and fix remaining conversion issues (Børre, Tomi)

Testing

Spelling Error Markup

TODO:

  • Set up ways of adding meta-information (source info, used in testing or not, added to lexicon or not) (Saara)
  • move Steinar's error markup in the xml files to (a copy of) the original ( Børre, Kimme)
  • add nested error markup to xml conversion (Saara)
  • test new and nested error markup (Sjur)

Automated testing

TODO:

  • add hyphenation testing (Sjur)
  • add hunspell testing (Saara, Sjur)
  • add paradigm testing (Børre, Sjur)

Lexicon conversion to the PLX format

Open issues based on test results:

smj

svierigaládde
svierigaládde   svieriga#ládde+N+Sg+Nom

polardutkamin
polardutkamin   polar#dutkat+V+TV+Actio+Der3+Der/m+N+Ess
polardutkamin   polar#dutkat+V+TV+Actio+Der3+Der/m+N+Sg+Ine

svieriga-
svieriga-       svieriga+Cmpnd

polar-
polar-  polar+Cmpnd

482 - still problematic, 552 - still open, Svierigadárogielan - still rejected

sme

397 - not as good as it should be, 408 - fixed, 419, 425 - roman number, 431, 452 - miel is a prefix, 461 - ovda accepted, almost 50 % (17) gets correct suggestion, 489, 508 - fixed, 522, 524.

TODO:

  • look at test cases still not behaving properly (Thomas, Tomi)

InDesign tools

TODO:

  • add hyphenation testing (Sjur)

Hyphenators

We should look into the possibility of generating pattern-based hyphenation for OOo. It shouldn't be too hard, or require too much work, but needs investigation.

TODO:

  • get command line hyphenator (Sjur)

Release version

The CD cover etc. will be worked on by John-Marcus Kuhmunen, and will follow the SD design rules. He is now waiting for the text to be put on the CD cover and other places.

There is a bug when installing from CD over an old installation.

TODO:

  • try to burn a CD at SD (Risten, Leif-Åge)
  • write text to go on the CD cover (Risten)
  • set up CD-printing printer (Risten)
  • fix Windows CD installation bug (Sjur, Børre)

Actual release

December 11-13, one of these days.

9. Other

Corpus contracts

Delayed till after final release.

TODO:

  • publish corpus contracts and project infra as open-source on NoDaLi-sta ( Sjur)

Faroese

Speller for fao using our infrastructure and the knowledge we have.

TODO:

  • set up a telephone meeting with them and Sjur ( Trond)

Bug fixing

When fixing bugs, record the version number containing the fix in the Bugzilla bug report, such that for each bug, we know exactly when it should have been fixed, in what file(s) and what version.

69 open Divvun/Disamb bugs (35 of these 56 are speller-related bugs, 34 are other bugs), and 23 risten.no bugs

SD yearly personell seminar

6.-7. December. Sjur will discuss it with Julia, but our view is that we don't have time to go this year. The release are just a few days later.

Software updates

  • SubEthaEdit
  • Leopard, 10.5
  • Skype 2.6.x

10. Next meeting, closing

The next meeting is 12.11.2007, 09: 30 Norwegian time.

The meeting was closed at 11: 37.

Appendix - task lists for the next week

Boerre

  • move Steinar's error markup in the xml files to (a copy of) the original
  • follow-up support for the sami languages in OpenOffice.org
  • fix Unicode bug in Hunspell conversion java code
  • fix bug 550
  • install Leopard on the G5 and the Xserve
  • move giellatekno.uit.no (including Bugzilla) and www.divvun.no to the G5.
  • add paradigm testing
  • fix Windows CD installation bug
  • fix bugs!

Ilona

  • lexicalise smj missing words
  • add smj proper nouns
  • other smj tasks, ask Thomas

Maaren

  • lexicalise actio compounds

Per-Eric

  • lexicalise words from the Olavi missing list
  • fix bugs!

Risten

  • finish the design/text for the CD and the cover
  • set up CD-printing printer
  • try to burn a CD at SD

Saara

  • add new XSL/XML headers for proofing test docs
  • Set up ways of adding meta-information for proofing correct corpus docs (source info, used in testing or not, added to lexicon or not)
  • add nested error markup to xml conversion
  • add hunspell testing

Sjur

  • work on the XML name editor/risten.no integration
  • set up risten.no on the G5 again
  • test new and nested error markup
  • get command line hyphenator for automated testing of the hyph-lexicons
  • add hyphenation testing
  • add hunspell testing
  • add paradigm testing
  • fix bug 550
  • fix hyphenation of derivations, inflections
  • follow-up support for the sami languages in OpenOffice.org
  • fix Windows CD installation bug
  • fix bugs!

Thomas

  • sme->smj lexicon conversion to build bilingual lexicon resources
  • add smj proper nouns
  • check for bad hyphenation
  • look at test cases still not behaving properly
  • fix hyphenation of derivations, inflections
  • sme->smj name conversion
  • fix bugs!

Tomi

  • Hunspell lexicon conversion
  • sme->smj lexicon conversion to build bilingual lexicon resources
  • fix Unicode bug in Hunspell conversion java code
  • fix hyphenation of derivations, inflections
  • fix bugs!

Trond

  • sme->smj lexicon conversion to build bilingual lexicon resources
  • fix hyphenation of derivations, inflections
  • sme->smj name conversion
  • telephone meeting with Sjur and the faroese group re faroese speller
  • fix bugs!.