Meeting_2006-10-09

Meeting setup

  • Date: 09.10.2006
  • Time: 09.30 Norw. time
  • Place: Where we are
  • Tools: SubEthaEdit, iChat

Agenda

  1. Opening, agenda review
  2. Reviewing the task list from two weeks ago
  3. Documentation - divvun.no
  4. Corpus gathering
  5. Corpus infrastructure
  6. Infrastructure
  7. Linguistics
  8. name lexicon infrastructure
  9. Spellers
  10. Other issues
  11. Summary, task lists
  12. Closing

1. Opening, agenda review, participants

Opened at 09: 51.

Present: Børre, Saara, Sjur, Thomas, Tomi, Trond

Absent: Maaren

Agenda accepted as is.

2. Updated task status since last meeting

Børre

  • corpus collection:
    • contact Ája (Kåfjord)
    • contact Richard Valkepää at NSI about older Min Áigi and Áššu files
      • Done
    • remind Bård Eriksen about the book catalogue
      • Done. Got it
  • Move norwegian documents in Min Áigi from sme to nob
    • Not done
  • corpus access:
    • possibly deploy the user account form as an HTML form
    • Write both user and admin documentation (Børre, review: Sjur, Thomas)
      • Not done
  • set up Bugzilla automatic reminders for open issues
    • Not done
  • finish Forrest i18n work (pdf)
    • Not done
  • Get more sma, smj texts to improve language recognition
    • Nothing this week
  • set up Tomcat for use with eXist and the propnouns db on the G5
    • Not done
  • add the new Words section to the site
    • Not done
  • fix bugs!
    • Not done
  • Other:
    • Gathered texts from the sami parliament computers
    • Investigated on how to use the Sami parliament Marratech server. We have to be connected to the sami parliament internal net ...

Maaren

  • download and install latest Marratech

Saara

  • add more texts to the graphical corpus interface
  • remove headers and footers from pdf documents
    • not finished.
  • Implement server of the analysis tools.
    • not finished.
  • generate parallel corpus files manually (with Trond)
    • in progress
  • Improve text_cat
    • not yet finished.
  • export corpus tools to location available to all (with cron), cf news disc.
    • not done
  • Write a script for cleaning sme-hyph-output.
    • done
  • Implement M4-rules for sme-dis.rle
    • waiting for some desicions.
  • improve smj LM based on the new, larger corpus
    • not done
  • document recent changes made in corpus infrastructure
    • done
  • fix bugs!

Sjur

  • finish the hyphenator clean-up script
    • Saara did this for me - thanks!
  • name lexicon:
    • implement editing functions
      • some done in the summer (Tomi), the rest waiting for the next item to finish
    • finalise refactoring for multiple collections - search interface
    • implement improvements decided upon in Tromsø
      • delayed until the previous one is in place
    • XQuery refactoring and code development for our proper noun editor
      • delayed until the previous one is in place
  • review user and admin documentation for corpus access
    • not done
  • write user account form, probably ask for copy of existing ones from the IT centre (with Trond)
    • not done
  • start hiring process of linguist and programmer
    • I have prioritised the risten.no/propernoun work, as well as what is needed for Polderland
  • finish i18n work of Forrest
    • nothing done. What we have works in a webapp setting, but not with the CLI. This is a largish task, needs consideration.
  • consider the problems of lexicalised derivations schewing analyses of derivation patterns
    • not done
  • install eXist and our local copy of risten.no and propnouns on the G5
    • not done
  • speller follow-up from the Tromsø meeting
    • done, discussed with Tomi
  • get instructions on how to use Marratech, and test it
    • still nothing
  • fix bugs!
    • nothing last week

Thomas

  • work with Polderland phonetic rules
    • finished with sme, soon finished with smj
  • refine smj proper noun lexica, cf. the propernoun-smj-lex.txt
    • finished with cont. lexica
  • find and study all derived verbs in our corpus (depends on Trond)
    • not yet
  • suggest which derivations could be generated (depends on Trond)
    • not yet

Tomi

  • start to plan the implementation of the speller data conversion/generation
    • started - written a first Java app for the purpose
  • fix bugs!

Trond

  • make shell script wrappers for the most common commands for user friendlyness
    • Done (pseudocode)
  • write user account form, probably ask for copy of existing ones from the IT centre (with Sjur)
    • Not done
  • write documentation for our bound users, with pointers to the ordinary documentation
    • Not done
  • refine smj proper noun lexica, cf. the propernoun-smj-lex.txt
    • Started work with Thomas, not much done.
  • Get more sma texts to improve language recognition
    • Tried to contact relevant people, still no result.
  • study paragraphs with mixed content
    • Not done.
  • fix bugs!.

3. Documentation

TODO:

  • finish i18n work (Børre and Sjur)
    • done when running in 'forrest run' mode - it works perfect now
    • does NOT work when running 'forrest site', instead it generates a heap of errors
      • solution/workaround: only use it with 'forrest run', webapp on our server
    • i18n does not work in PDF ("Table of Content" won't translate)
      • we need to ask on the Forrest list
  • Write both user and admin documentation (Børre, review: Sjur, Thomas)
    • User documentation probably in several languages. This covers how to apply for an account, on what grounds one can apply, and pointers to documentation telling how to use the corpus.
    • Admin documentation, telling how we set the permissions to the corpus files, and whatever other processes and tasks needed to set up a corpus account.
      • move these issues into Bugzilla (Sjur)
  • add the new Words section to the site (Børre)
    • done

4. Corpus gathering

NSI: the Mac has crashed during the summer, they need a new motherboard for it, but they don't have money to buy it! This is a major setback for us - the major part of Min Áigi and Áššu can't be delivered. Unless... Børre is going to Kautokeino on a course in the near future, and could probably get the hard disk mounted on his computer. Discuss and plan this with the NSI guy.

Bård Eriksen has sent the book list. It contains only about 10 titles.

Lars Kintel has contacted us, and offered all his translations to smj free of charge. We still need to contact the authours, to get their approval as well.

TODO:

  • contact NSI (Børre)
    • done
  • bug Bård Eriksen about the book list (Børre)
    • done
  • continue to contact authors (Børre)

5. Corpus infrastructure

General

TODO:

  • consider an article on our corpus framework (all)
    • put this into Bugzilla (Trond)

User accounts and access

The whole issue should be moved into Bugzilla as several tasks.

TODO:

  • add the issue with subissues to Bugzilla (Trond)

More texts to the graphical corpus interface:

We are waiting for parallel texts, and aligning parallel text. SD Parliament meeting memos are excellent for parallell testing, as they are accurately translated in both directions (but some of the Parliament plenary meeting memos do not have a nob counterpart, or we haven't got one yet).

Børre has found a lot of old Parliament meeting protocols, and will probably be able to add missing translations/parallell documents.

TODO:

  • align texts, analyse, and send to Lars (Trond, Saara)
  • add text to the server (Lars)

Aligner

Discuss NT parallel corpus

originals: Word, paratext, txt our format: xml

It boils down to two issues:

  1. Which original to choose
  2. Whether to align according to verses or according to our own preprocessor (verses) (send both options to the parsers and see which one we like)

Goal: Have the Bible texts in the same format as .

TODO:

  • gather parallel texts (Trond)
  • try out NT alignment strategies (Saara)

Language recognition

TODO:

  • Get more text of the poorly covered languages: sma, smj ( Trond, Børre)
    • sma: get the Bible texts (Trond)
  • improve the smj LM (Saara)
  • what about paragraphs with mixed content? Build a corpus of such paragraphs ( Trond)

6. Infrastructure

Xerox tools wrapped as servers

No restrictions on IP numbers or domains from which the server is accessed.

Modules to be includes / types of services:

  • morphological analyzer
  • morph. generator (norm/descr? If !SUB could be added as a regular tag, then we can have both)
  • hyphenator
  • disambiguator
  • speller (in the future) - this is not Xerox tools: -)

Supported input formats:

  • pure text
  • XML according to a fixed DTD - if XML in, always XML out

Output formats:

  • text (UTF-8)
  • XML (as option) - we need to decide upon the XML delivered (see below)

Access clients:

  • Java client (Tomi)
  • command-line client (Saara)
  • http access through CGI-script (Saara)

XML output format: 

Analyser and disambiguator:
<w form="Tromssan">
<reading lemma="Tromsa" analysis="N+Sg+Loc+.."/>
..
</w>

Generator and hyphenator should follow the same pattern. Saara will make something: -)

TODO:

  • improve and finish the present prototype (Saara)

Hyphenator

Tros^te^rud TROSTERUD

Čále dahje kopiere sániid lásii, ja deatti "Analysere". Čilgehus giellaoahpalaš

čá^le da^hje kopiere sá^niid lá^sii , ja deat^ti a^na^ly^se^re .

Čilgehus čil^ge^heh^kos ≠ čil^ge^heh^kos ≠ čil^ge^hus = čil^ge^hus = Čil^ge^hus <==

TODO:

  • improve the clean-up script to remove more-complex word forms (Saara)
    • done
  • Update the sma hyphenator rule set with the insights gained from smj updates ( Trond during weekends)
  • linguistic issues in the hyphenator: ( Sjur using M4)
    • shortening in compounds (grossly overgenerates in the hyphenator output)
  • consider case conversion problems (Saara)
  • check #- when comparing input string with hyphenated string - the hyphen could be part of the input string, keep it if it is, discard the hyphenated string if not (Saara)

Automatic Bugzilla reminder for untouched bugs

Some perl-libraries needed by Bugzilla weren't in the path, causing it to not work. Adding them should fix the issue.

TODO:

  • fix the remaining issues (Børre)

M4

TODO:

  • make speller make targets that utilise M4 to produce normative and hyphenation transducers - postponed a while (Tomi or Sjur)
  • disamb variants for derivations (see next) (Saara)
    • include make as part of a shell script, which will run the M4 processor on the disamb rules (Saara)

7. Linguistics

Names and multilinguality

We need a more principled approach to this.

Background: the name lexicon is getting attention from the SD name/terminology sections, and they would like to use our name lexicon also for public searching.

Observations:

1) Multilinguality is always optional.

2) We can observe that "foreign" names in texts follows a domination pattern: majority language forms can be found in minority language texts as real names ("Kautokeino produkter"), whereas minority language names almost always occur in majority language texts as citations. And citations should not be considered a natural part of the text.

3) When looking at our name classification, multilinguality varies according to: 

Ani - weak/none? (pet, myth anim.  names)
Fem - weak (informative)
Mal - weak (informative)
Obj - strong
Org - strong
Plc - strong for the national and country names, weak (informative) for foreign
       names
Sur - none
Tit - strong (titles)

Suggestion:

We need to reconsider the all names in all languages policy. That policy is valid only for Fem, Mal, and Sur (and Ani and Tit?). For Obj, Org, Plc the rule should be that if they have multilingual names, each name should only be used in it's own language. Then we need a modification saying that majority language names can be included in minority language lexicons if attested in our corpus. Also, the majority language varies according to country (obviously), which means that in a speller context, we might consider tailoring spellers for each country, leaving out noise relating to majority language names from another country.

A further issue is whether we should reconsider our cohort policy. Today, Sur and Plc are different readings. An alternative would be to have them as secondary tags, not in conflict with each other: 

"<Trosterud>"
        "Trosterud" N Prop Sur Sg Nom <<< @HNOUN
        "Trosterud" N Prop Plc Sg Nom <<< @HNOUN
"<Trosterud>"
        "Trosterud" N Prop Sg Nom <Sur> <Plc> <<< @HNOUN
"<Trosterud>"
        "Trosterud" N Prop Sg Nom &Sur &Plc <<< @HNOUN

Derivation and spellers like Aspell

  • enclose the CG rule that preferres lexicalised forms over derivations in M4 ( Saara)
  • find and study all derived verbs in our corpus (Thomas)
  • suggest which derivations could be generated (Thomas)
  • lexicalise the rest (Thomas)

North Sámi

Nothing this week?

Lule Sámi

TODO:

  • refine smj proper noun lexica, cf. the propernoun-smj-lex.txt ( Thomas, Trond)
    • done some more, much more left
  • hire new linguist (Sjur)
    • nothing real, but one new candidate has surfaced

Komi

It appears that the conversion form LexC to XML went wrong wrt to upper: lower pairs, consider the following entry from the Attic (noun file): 

вос:воск N ;

The corresponding XML is: 

<entry>
  <lemma>воск</lemma>
  <stem/>
  <contlex>Noun1</contlex>
  <POS>N</POS>
  <article>
    <ENG></ENG>
    <FIN></FIN>
    <CF></CF>
  </article>
</entry>

It should have been: 

  <lemma>вос</lemma>
  <stem>воск</stem>

There seems to be an issue. Trond will look into it.

On Victorio: 

/usr/local/cvs/repository/kt/kom/src/Attic/
~$ll /usr/local/cvs/repository/kt/kom/script/kom-utf.xml,v

TODO:

  • check the dictionary and lexc source files

8. Name lexicon infrastructure

Decided in Tromsø:

  • add logging facilities to the interface
  • add option to download local copies of the lexicon files directly from the db
  • batch editing (change all entries in the found set), should later be enhanced to allow selection of exceptions (the found set minus deselected items)
  • tag for excluding/including a name from certain applications
  • future epxansion: choose what info to display in the single language browser
  • display existing language entries when adding a new language to a record
  • add editor to change single, existing entries

Details can be found in the meeting memo.

TODO:

  • finish refactoring for multiple collections in the search interfarce ( Sjur)
  • develop the needed XQueries and UI (Sjur, Tomi)
  • turn Tomcat on on the G5; send admin username and password to Sjur ( Børre)
  • add eXist and the proper noun interface to the G5 (Sjur)

Postponed:

  • data synchronisation between risten.no and the cvs repo
  • new version of xml2lexc (based on ccat), should handle complex names correct: construct entries like we have now from the different parts of a complex name entry

9. Spellers

Speller data generation

TODO:

  • start to plan the implementation of the speller data conversion/generation ( Tomi)
    • done, progressing fine.

10. Other

Meeting with the map authorities

Sjur is going to a meeting in Oslo on Wednesday, meeting with the Ministeries and the Statens kartverk.

Issues:

  • open source vs money

Bug fixing

64 open Divvun/Disamb bugs, and 25 risten.no bugs

Guess: 1/3 of the bugs are fixed already (?)

Meetings and Marratech

Børre talked to Leif Åge about the Marratech server. All users need to be inside the SD firewall. Thus, Marratech is not an option.

Quoting Marratech:

Marratech on proxies and firewalls

What about using proxies for Maaren? iChat has a setup page for it in Preferences->Accounts. Børre will investigate

TODO:

  • investigate use of proxies for Maaren ( Børre)

Task lists as iCal entries

TODO:

  • update Maaren's installation to r430284 (Børre)

Words section

You all need to check out CVS/words, and link to the relevant place, cf how gt/doc/ is linked.

11. Next meeting, closing

Next meeting 16.10.2006 at 9: 30.

Saara will work offline next week, and won't be present in the meeting.

Closed at 12: 26.

Appendix -task lists for the next week

Boerre

  • corpus collection:
    • contact Ája (Kåfjord)
    • discuss access to older Min Áigi and Áššu files with Richard Valkeapää
  • Move norwegian documents in Min Áigi from sme to nob
  • set up Bugzilla automatic reminders for open issues
  • finish Forrest i18n work (pdf)
  • set up Tomcat for use with eXist and the propnouns db on the G5
  • investigate use of proxies for Maaren, for her to be able to use iChat AV
  • fix bugs!

Maaren

  • download and install latest Marratech

Saara

  • add more texts to the graphical corpus interface
  • finalize server of the Xerox tools.
  • generate parallel corpus files manually (with Trond)
  • Improve text_cat
  • export corpus tools to location available to all (with cron), cf news disc.
  • Improve hyph-filter.pl
  • Implement M4-rules for sme-dis.rle
  • improve smj LM based on the new, larger corpus
  • help Trond with some shell commands
  • fix bugs!

Sjur

  • name lexicon:
    • refactor SD-terms editor code
    • implement missing propnouns editing functions
    • implement improvements decided upon in Tromsø
  • move corpus user doc issue to Bugzilla
  • start hiring process of linguist and programmer
  • finish i18n work of Forrest
  • install eXist and our local copy of risten.no and propnouns on the G5
  • add normative M4 macros to the twol code; utilize it in the make file
  • fix bugs!

Thomas

  • work with Polderland phonetic rules for smj
  • refine smj proper noun lexica, cf. the propernoun-smj-lex.txt
  • find and study all derived verbs in our corpus (depends on Saara)
  • suggest which derivations could be generated (depends on Saara)

Tomi

  • continue implementation of the speller lexicon conversion
  • fix bugs!

Trond

  • Discuss shell script wrapper teaksta.sh with Saara
  • write user account form, probably ask for copy of existing ones from the IT centre (with Sjur)
  • write documentation for our bound users, with pointers to the ordinary documentation
  • refine smj proper noun lexica, cf. the propernoun-smj-lex.txt
  • Get more sma texts to improve language recognition
  • study paragraphs with mixed content
  • add corpus artice framework to Bugzilla
  • add corpus user accounts and access issues to Bugzilla
  • fix bugs!.