Meeting_2006-06-12

Meeting setup

  • Date: 06.06.2006
  • Time: 09.30 Norw. time
  • Place: Where we are
  • Tools: SubEthaEdit, iChat

Agenda

  1. Opening, agenda review
  2. Reviewing the task list from two weeks ago
  3. Documentation - divvun.no
  4. Corpus gathering
  5. Corpus infrastructure
  6. Infrastructure
  7. Linguistics
  8. name lexicon infrastructure
  9. Spellers
  10. Other issues
  11. Summary, task lists
  12. Closing

1. Opening, agenda review, participants

Opened at 09: 59.

Present: Sjur, Thomas, Trond, Tomi

Absent: Børre, Maaren, Saara

Agenda accepted as is.

2. Updated task status since last meeting

Børre

  • corpus collection:
    • send out contracts with accompanying letter
      • Maaren has signed a contract, which is sent to Davvi Girji
    • Gather public texts, preferrably also parallel ones
      • Not done
    • Continue converting text from input format to our xml
    • convert nob and nno bible texts to be used as part of a parallel corpus
      • Not done
    • review the paratext2xml converter
      • Not done
    • convert smj NT to paratext
      • Not possible before we have another paratext NT.
    • Send out letters to the rest of the Iđut authors
      • Not done
    • send contract to Kurt Tore Andersen
      • Not done
    • call Brita Kåven again
      • Haven't got in touch with her
    • call Harald Gaski
      • Met with him. He has to send contracts to his writers union, to check if the contracts are ok with them.
    • contact Ája (Kåfjord)
      • Not done
    • Send renaming scripts to R. Valkeapää
      • Working on them
    • complete Min Áigi metadata
      • Halfway done
    • call Bård Eriksen
      • Not done
  • corpus access:
    • meeting 9.6. t 9.30: discuss and decide upon the exact access policy we want
      • Delayed once more
    • set upp the unix group structure for corpus users (also Saara, Trond)
      • Waiting for outcome of above meeting
  • set up Bugzilla automatic reminders for open issues; ask Thor-Øivind if needed
    • Not done
  • fix bugs!

Maaren

  • On sick leave

Saara

  • Create a parallel corpora of the new testaments
  • add more texts to the graphical corpus interface
  • grammatical searchability in the graphical corpus interface
  • Implement links to parallel files in corpus header
    • done
  • Implement parallel corpus upload in web upload script
  • Install Gobby
  • set upp the unix group structure for corpus users (also Børre, Trond)
  • Test the aligners once again
  • make an analyser that retains the xml structure for Oslo
    • in progress (with Tomi). The exact structure of the output still open
  • discuss parallel text markup in newsgroup
    • done
  • fix bugs!

Sjur

  • public tender:
    • write e-mail to PL and ask for more clarification
      • done
  • fst gymnastics to add hyphenation and word boundary marks to hyphenation transducer
    • not done
  • name lexicon:
    • implement editing functions
    • finalise refactoring for multiple collections (regular search interface)
      • some work on researching new ways of doing all-collection searches, and at the same time allow for different xml structures
  • change corpus-summary processing to generate smaller pages
  • meeting 23.5. t 9.30: discuss and decide upon the exact access policy we want
    • postponed
  • move to Bugzilla:
    • proofed/unproofed parallel text issue
      • done
    • xml output of paradigm generator
      • done
  • fix bugs!

Thomas

  • hyphenation-rule-set
    • a few bugs, otherwise finished
  • work on compounding and derivation
    • finished with derivation
  • lule sámi incoming words
    • added a few
  • sme G3 issue
    • nothing this week

Tomi

  • new proper name lexicon
    • data synchronisation of proper nouns between risten.no and CVS
      • not done
    • XQuery refactoring and code development for our proper noun editor
      • not done
    • new version of xml2lexc (based on ccat), should handle complex names correct: construct entries like we have now from the different parts of a complex name entry
      • not done
  • read aligner docu, install, provide feedback
    • not done
  • Set up the mechanism for the hash-mark transducer package
    • not done
  • add ccat option to analyse text while keeping the xml tags and structure
    • done some
  • fix bugs!

Trond

  • better smj NT text
    • Not done
  • get fin, swe, nob and nno NT and OT in paratext format
    • Discussed with fin and swe societies, the Swedes are not quite satisfied with their paratext version, and prefer us to do the Word version. Børre and I should take a new round on this.
  • install aligner, test it and give feedback
    • Not done
  • fst gymnastics to add hyphenation and word boundary marks to hyphenation transducer
    • Not done
  • Put Saara and Tero in contact with each other
    • Per will send the PHP code. Details about Tero will be given at the meeting.
  • meeting 9.6. t 9.30: discuss and decide upon the exact access policy we want
    • Postponed.
  • set upp the unix group structure for corpus users (also Børre, Saara)
    • Postponed.
  • fix bugs!.
    • Worked on some bugs.

3. Documentation

TODO:

  • documentation on how to apply for a user account for the corpus repo ( Børre)
    • we will administer the corpus user accounts ourselves
    • We first have to discuss and decide what we want before Børre can write documentation (see further down)

4. Corpus gathering

Collecting

See a previous meeting memo for what's to be done.

TODO:

  • Send out the rest of the letters (Børre)

New contracts:

  • none last 2 weeks

Olavi Korhonen's Lule Sámi dictionary.

TODO:

  • set up user account/corpus access for Olavi (Børre)
    • we need the infrastructure for corpus access in place first

KIO Grafisk and the Iđut books

TODO:

  • send letter to Kurt Tore Andersen (Børre)
  • send letters to the other authors (Børre)

Bible texts

The Swedish Bible Society is reluctant to give us the paratext version, as it is linguistically inferiour. Instead they suggested a MS Word version of the text.

Børre and Trond suggest that we give the MS Word version a second try, at least to be able to get hard facts about the problems with that format when discussing with the swedes.

We have asked for any language version in paratext format from the Norwegian Bible Society, to use as a reference when making the smj paratext version.

We are still waiting for the Norwegian versions, but Finnish and Swedish are within reach. Finnish is available in a newsgroup (text-only) format, and we have received MS Word of the Swedish version. Børre has had some problems converting the Word file, though.

TODO:

  • get nob and nno NT and OT in paratext format. (Trond)
  • convert smj NT to paratext (Børre)
  • convert fin, swe to paratext or directly to our XML (Børre)

Davvi Girji

Talked to Harald Gaski, he will send the contracts to the writers' organisation, to ensure the contract is ok.

TODO:

  • call Brita Kåven again towards the end of the week (Børre)
  • call the authors (Børre)
    • Harald Gaski
      • done

Min Áigi

TODO:

  • complete metainformation (Børre)

Kåfjord

Promised to send us texts. Some texts have arrived, but nothing from Ája.

TODO:

  • contact Ája (Børre)
    • Not done

Sámi Instituhtta

TODO:

  • send renaming scripts to R. Valkeapää (Børre)

Čálliid Lágádus

Børre talked to them, they are positive, and will give us print-ready pdf's. http://www.calliidlagadus.org/

Árran

The negotiations are underway, they discuss it in a meeting today, and we have an appointment for later today or tomorrow (both their internal texts, and discussions on Báhko). All further negotiations will go through Bård Eriksen / Báhko.

TODO:

  • call Bård Eriksen (Børre)

5. Corpus infrastructure

General

Errors in the Antiword conversions found when parsing the xml corpus. Main problem is to skip headers and footers - now they are included as part of the text, and are intermingled within the regular text, e.g. in the middle of a sentence.

TODO (all of these in priority order, the third option is really a last resort):

  1. fine-tune the initial conversion in antiword (Børre or Saara)
  2. make file-specific fixes in the file-speciflc xsl file (by having our local xsl expert read our corpus, and spot obvious omissions in the conversion ( Saara)
  3. Manually fix the resulting files before sending off to analysis (??, this option should be postponed to just before the final, official release, and evaluated at that point)

User accounts and access

TODO:

  • discuss and decide upon the exact access policy we want to give corpus users; meeting on Tuesday May 23, at 9.30 (Børre, Sjur, Trond)
    • still not done, new date: Wednesday 14.6.2006, at 9.30
  • set upp the unix group structure to open for a new category of users: Linguists with access to the texts on the servers. Some of Børre, Saara, Trond.

More texts to the graphical corpus interface:

We need to get the infrastructure complete to be able to do this, then it should be a piece of cake.

Saaara has made corpus-analyze.pl, a script to analyse text while keeping the xml structure. It is working, but the output format needs some tweaking.

TODO:

  1. refine xml-tagged output (Saara and Tomi)
  2. add text to the server (Lars)

Aligner

Trond and Saara will continue this issue.

We need markup of parallelism in the corpus DTD, at least an indication of which documents belong together. Discussion to continue in the newsgroup (Saara has started it - please respond!).

Language recognition

Still waiting for more smj text to improve it.

Free and non-free texts

Anything? Final check with Børre and Saara - waiting for them to return.

Corpus summary

TODO:

  • trim generated corpus summary pages (Sjur)
    • suggestion: lump together files with content less than X paragraphs (X < 5?)

Proofed vs unproofed corpus files

TODO:

  • count the number of parallel files of type unproofed/proofread (Saara)
    • done, 35 such file pairs
    • delayed for the time being, move to Bugzilla (Sjur)
      • done

6. Infrastructure

Paradigm generation

Goal: Reuse Greenlandic code for paradigm generation.

We have now received the original PHP code from Tero/Per. It seems quite easy to adapt, although it will of course require work on our part.

TODO:

  • convert or adapt the received PHP to our needs (Tomi or Saara)
  • move xml-out option to Bugzilla (Sjur)
    • done

Hyphenator

Thomas is finished with adding ^ tags to the sme noun file.

Trond and Thomas have been working on the smj rule component, and have improved both the treatment of weak grade consonant clusters (preconsonantal geminates) and on some loan word patterns.

TODO:

  • correct the treatment of hyphenation of word boundaries and exceptions (fst gymnastics) (Sjur, Trond)
    • Still not done - postponed till next week (w 24)
  • Update the sme and sma rule sets with the insights gained from smj updates.

Automatic Bugzilla reminder for untouched bugs

We need to get a summary by e-mail for all bugs not touched in more than 5(?) weeks. Do we want it? Yes. It is possible? Børre will look around, if nothing found, he'll ask Thor-Øivind.

TODO:

  • set up Bugzilla to send automatic reminders for bugs not touched in a given timeframe; ask Thor-Øivind for help if needed (Børre)

7. Linguistics

Name double-tagging

Conclusion, in a principled fashion:

  1. hardcoded sem-tags win
  2. There is a sem-tag conversion procedure: according to a hierarchy of sem-tags: Any Plc can be interpreted as Sur, etc. (to be spelled out)

TODO:

  • when adding new names, only use one sem-tag unless there are known objects which belong to different sem classes(?) (all linguists)
  • write disamb rules to implement the system above (Trond, Linda)

North Sámi

Topic: Actio+compound - how productive? How much does it destroy speller performeance by being close to real spelling errors with a high editing distance to the correct form? 

Spelling error from typos.txt:
4       vuolggahansádji             vuolggasadji

vuolggahan
vuolggahan      vuolgga+N+Sg+Nom+Foc
vuolggahan      vuolggahit+V+TV+PrfPrc
vuolggahan      vuolggahit+V+TV+Ind+Prs+Sg1
vuolggahan      vuolggahit+V+TV+Actio+Acc
vuolggahan      vuolggahit+V+TV+Actio+Gen
vuolggahan      vuolggahit+V+TV+Actio+Nom

vuolggahansadji
vuolggahansadji vuolgga+N+SgNomCmp#ho+N+SgNomCmp#atnu+N+SgNomCmp#sadji+N+Sg+Nom
vuolggahansadji vuolgga+N+SgNomCmp#ho+N+SgNomCmp#atnu+N+SgNomCmp#sadji+N+Sg+Nom
vuolggahansadji vuolgga+N+SgNomCmp#ho+N+SgNomCmp#atnu+N+SgNomCmp#sadji+N+Sg+Nom
vuolggahansadji vuolggahit+V+TV+Actio#sadji+N+Sg+Nom
vuolggahansadji vuolggahit+V+TV+Actio#sadji+N+Sg+Nom
vuolggahansadji vuolggahit+V+TV+Actio#sadji+N+Sg+Nom

TODO:

  • investigate Actio compounding (Thomas)
  • discuss findings with Maaren, and later all of us (Thomas)

Lule Sámi

There are some open issues in the marginal area of the smj transducer:

  • numerals, e.g. Our poor treatment of number words becomes more visible as we get real texts (guhttatuvsánanielljatjuodegålmmålåkvihtta is not recognised, etc.).
  • names => waiting for the new name lexicon
  • compounds? Shortening here as well, but not in written language (some lexicalised exceptions, as in sme) => nothing to discuss/implement
  • loanwords? We should consider importing the ^LOAN words from sme and Lule-ify them. Otherwise, loan words should come from corpora.

TODO:

  • 50 unknown words left+2 abbr. +moaddi etc (numerals) need more checks ( Thomas)
    • progressing fine
  • add proper numeral analysis/treatment (Thomas)
  • add loanwords (e.g. latin -ere verbs) (Thomas)

8. Name lexicon infrastructure

TODO:

  1. finish refactoring for multiple collections in the search interfarce ( Sjur)
    1. progressing, investigating options
  2. develop the needed XQueries and interface (Sjur, Tomi)
    1. progressing, done some, commited (adding new term, create-termc-entry.xq)
  3. data synchronisation between risten.no and the cvs repo (Tomi)
    1. discussion started on eXist-list, nothing useful came up. We need to reformulate the question from our perspective, and bring it up again ( Sjur)

9. Public tender

We have received answers from LS, and are now wating for the PL answers. Their deadline is the coming Thursday.

TODO:

  • write clarification letters to PL (Sjur with help from Børre, Trond)
    • done
  • evaluate finally the offers based on the answers to the clarification letters

10. Other

Summer vacation

Who When
Børre August
Linda ?
Maaren ?
Saara July
Sjur at least 2 weeks in July, but still open
Thomas 3.7 - 7.8
Trond 3.7 - 14.8 (last two weeks off at summer school)
Tomi 8.7 - 16.7, 2 more weeks in July and/or August

Bug fixing

43 open Divvun/Disamb bugs (two down!), and 25 risten.no bugs

Guess: 1/3 of the bugs are fixed already (?)

Please help Saara with bug 279 (Perl locale). Not much help... Saara will contact Roy on this issue.

After the corpus issues have been somewhat settled, we should do a bug barnraising. ... and then a new one after the name lexicon is fixed.

Gobby

Installed to most computers (only Saara missing), now we need to test it in a multiuser scenario.

Trond has asked Lars Nygård and Tero Avellan to install Gobby as well. Per Langgård is already using it.

TODO:

  • install Gobby (Saara)
  • test it (Tomi, Trond, Sjur, Børre, Lars?)
  • if successful, document its use within our project (Børre)

SEE 2.5 extensions

TODO:

  • give the jspwiki syntax colouring mode to all (Sjur)
    • JSPWiki.mode checked in to cvs in gt/src/ - just double-click the mode!
  • write a perl script to extract all TODO items and sort them according to responsible person; wrap the script in a AppleScript available from the menu
  • other modes on the whish list:
    • twol mode
    • xfst mode
    • lexc mode
    • vislcg mode

11. Next meeting, closing

19.06.2006 09: 30

Sjur is away on Friday June 16.

Closed at 11: 21.

Appendix - task lists for the next week

Børre

  • corpus collection:
    • send out contracts with accompanying letter
    • Gather public texts, preferrably also parallel ones
    • Send out letters to the rest of the Iđut authors
    • send contract to Kurt Tore Andersen
    • call Brita Kåven again
    • contact Ája (Kåfjord)
    • Send renaming scripts to R. Valkeapää
    • call Bård Eriksen
  • corpus conversion:
    • Continue converting text from input format to our xml
    • convert nob and nno bible texts to be used as part of a parallel corpus
    • convert fin, swe to paratext or directly to our XML
    • review the paratext2xml converter
    • convert smj NT to paratext
    • complete Min Áigi metadata
  • corpus access:
    • meeting 9.6. t 9.30: discuss and decide upon the exact access policy we want
    • set upp the unix group structure for corpus users (also Saara, Trond)
  • set up Bugzilla automatic reminders for open issues; ask Thor-Øivind if needed
  • test Gobby (with others)
  • document use of Gobby within our project if above test is ok
  • fix bugs!

Maaren

  • On sick leave

Saara

  • Create a parallel corpora of the new testaments
  • add more texts to the graphical corpus interface
  • grammatical searchability in the graphical corpus interface
  • Implement parallel corpus upload in web upload script
  • Install Gobby
  • set upp the unix group structure for corpus users (also Børre, Trond)
  • Test the aligners once again
  • refine the xml output of the xml-tagged analyses
  • convert or adapt the received PHP for paradigm generation to our needs
  • fix bugs!

Sjur

  • public tender:
    • collect the last clarifying answers, and make a final proposal to the board
  • fst gymnastics to add hyphenation and word boundary marks to hyphenation transducer
  • name lexicon:
    • implement editing functions
    • finalise refactoring for multiple collections (regular search interface)
  • change corpus-summary processing to generate smaller pages
  • meeting 23.5. t 9.30: discuss and decide upon the exact access policy we want
  • test Gobby (with others)
  • fix bugs!

Thomas

  • investigate Actio compounding, first sme, later smj
  • discuss findings with Maaren, and later all of us
  • add proper numeral analysis/treatment to smj
  • add loanwords (e.g. latin -ere verbs) to smj
  • work on compounding
  • lule sámi incoming words
  • bug-fixing
  • sme G3 issue

Tomi

  • new proper name lexicon
    • data synchronisation of proper nouns between risten.no and CVS
    • XQuery refactoring and code development for our proper noun editor
    • new version of xml2lexc (based on ccat), should handle complex names correct: construct entries like we have now from the different parts of a complex name entry
  • read aligner docu, install, provide feedback
  • Set up the mechanism for the hash-mark transducer package
  • refine the xml output of the xml-tagged analyses
  • convert or adapt the received PHP for paradigm generation to our needs
  • test Gobby (with others)
  • fix bugs!

Trond

  • better smj NT text
  • get fin, swe, nob and nno NT and OT in paratext format
  • install aligner, test it and give feedback
  • fst gymnastics to add hyphenation and word boundary marks to hyphenation transducer
  • meeting 9.6. t 9.30: discuss and decide upon the exact access policy we want
  • set upp the unix group structure for corpus users (also Børre, Saara)
  • test Gobby (with others)
  • fix bugs!.