Meeting_2006-03-06

Meeting setup

  • Date: 06.03.2006
  • Time: 09.30 Norw. time
  • Place: Wherever we are : -)
  • Tools: iChat, SubEthaEdit

Agenda

  1. Opening, agenda review
  2. Reviewing the task list from two weeks ago
  3. Documentation - divvun.no
  4. Corpus gathering
  5. Corpus infrastructure
  6. Infrastructure
  7. Linguistics
  8. name lexicon infrastructure
  9. Spellers
  10. Other issues
  11. Summary, task lists
  12. Closing

1. Opening, agenda review, participants

Opened at 10: 04.

Present: Maaren, Sjur, Thomas, Tomi

Absent: Børre, Saara, Trond

Main secretary: all

Agenda accepted as is.

2. Reviewing the task list from the last meeting

Børre

On Winter holiday.

Maaren

  • work with risten.no
    • not done. I have worked with the top-ten list lately. Almost finished.
  • discuss with relevant people regarding seminar on proofing tools, normativity and SGL in February/March, including place.
    • Laila wants us to send the list to the SGL-members (as we have done before). Seminar is not actual. Yet.
  • get the normativity decissions from the December SGL meeting
    • I have send decissions to Thomas and asked Thomas to phone to Laila.

Saara

  • continue discussion on the new lexicon format
  • Fix the preprocess script and optimize it.
  • Try to add numeral treatment as part of the analyzator.
  • Move gt2ga.sh to G5 and implement copying of the gt-dir.
  • Create a parallel corpora of the new testaments.
  • Routine for adding new languages to the propernoun xml-structure.
  • Move to Bugzilla: the analyzer needs to be optimized.
  • Implement validation of xml corpus against the dtd.
  • Create a group for corpus users.
  • Finish corpus dtd documentation, dtd location and permlink reference
  • Finish gt/doc/ling/corpus_conversion_tech.html and rename to .xml
  • fix bugs!

Sjur

  • Follow up the lawyer treatment of the contracts
    • not done
  • Lule Sámi twol problems, with Thomas and Trond
    • not done
  • project planning with Trond, continued
    • not done
  • Follow up on place names from Norge Digitalt
    • not done
  • Evaluate SFST as speller (and analyzer) lexicon
    • not done
  • write a background document on the corpus contracts
    • not done
  • public tender:
    • review draft tender document from Finnut
      • done and published
    • the public tender would benefit a lot from anonymous cvs access (read-only) and also regular corpus access - started discussion last week with Børre and Trond (see notes further down)
  • smj G3 issue with Thomas and Trond
    • not done
  • sme G3 issue with Thomas and Trond
    • not done
  • call EDD/ Christian Emil Ore about national place name lexicon
    • not done
  • risten.no/proper noun lexicon development:
    • refactor code
      • not done
    • implement inheritance/collection overriding for xsl/css/xquery using sitemaps
      • tried to work out how to match parts of a request parameter, and continue based on the match - no success so far (but still studying)
    • data synchronisation between risten.no and the cvs repository
      • not done
  • fix bugs!

Thomas

  • work with Lule sámi 90% goal
    • worked and reached goal 92%
  • work on North Sámi compounding and derivation
    • worked
  • smj G3 issue with Sjur and Trond
    • not had time to
  • sme G3 issue with Sjur and Trond
    • not had time to

Tomi

  • move aspell UTF-8 suffix bug to Bugzilla
    • not done
  • corpus infrastructure:
    • dtd location (both public and internal)
      • not done
  • Document aspell infrastructure: finish doc/proof/spelling/X-spell/aspell.xml (it's almost done, but there are a couple of loose ends)
    • not done
  • new proper name lexicon
    • discuss the new lexicon format and other issues in the newsgroup
      • not done
    • Look into data synchronisation of proper nouns between risten.no and CVS
      • not done
    • XQuery refactoring and code development for our proper noun editor
      • not done
    • new version of xml2lexc (based on ccat), should handle complex names correct: construct entries like we have now from the different parts of a complex name entry
      • not done
  • fix bugs!

Trond

On Winter holiday.

3. Documentation

Changes and updates because of the Divvun public tender

  • document anonymous, read-only access to our cvs repo
  • probably a new main section (sub-tab?) on external access to all our resources
  • documentation on how to apply for a user account for the corpus repo
  • we need to finish the corpus dtd documentation

4. Corpus gathering

Collecting

See a previous meeting memo for what's to be done.

TODO: Send out the rest of the letters (Børre)

Odin

Waiting for Sæth to discuss with colleagues about how to implement the cooperation, and return to us.

TODO:

  • call Sæth (Børre)

Olavi Korhonen's Lule Sámi dictionary.

Korhonen and Oahpadusguovdásj have a shared copyright to the dictionary.

KIO Grafisk and the Iđut books

We need a test file in order to find out whether file conversion from Quark to InDesign works.

TODO:

  • Trond to ask for a Quark test file from his sister
  • Børre to ask KIO to send a more elaborate test file, representative for what we can expect, but without any copyrighted content (and please stress, that fonts will never leave the computer, they are part of the OS, and are not included in the Quark file). The file should include text with Sámi and Norwegian letters, and preferably some dummy pictures as well. Actually, Børre could create that file in Word and send it to KIO for them to move to Quark and send back.
  • Børre will send letters to the authors.

Bible texts

TODO:

  • review paratext2xml converter (Børre)
  • convert smj NT to paratext (Børre)
  • ask to get fin and swe NT and OT in paratext format. (Trond)
    • Still not done. Trond will contact Bibelselskapet for a new sme version, and for the NT nob and nno, and the fin and swe ones (after Wednesday)

5. Corpus infrastructure

We need more "version control" in the corpus work - we don't know which version of the XSL script was used (but roughly whether it was used or not). We need to document within the XSL file which version of all tools were used, including the XSL file (common section/template) itself.

Transferring the old gt/sme/corp files to the new corpus repo:

  • for the biggest top ten (or so) the orig. should be located and copied to the new corpus repo
  • then these files should be removed from gt/sme/corp/
  • all small files could just be forgotten/ignored

TODO:

  • Access control to corpus repo resolved through Unix groups: one group for corpus maintainers with write access, and another for corpus users, with read-only access. Still not done.
    • Saara will ask Roy to do what should be done.

Further discussion about corpus analysis and computer use:

The new G5 is tremendeously faster than cochise, thus we want to use it. But cochise will continue to be our main corpus repo.

  • the corpus/gt/ dir will be synchronised with the G5
    • TODO: set up copying script (Saara)
  • we need to develop strong enough security routines for the G5 to fulfill our obligations towards the text licensers
    • TODO: Børre to move this to bugzilla
  • we are still using only one processor when analysing - making some simple multiprocessor analysis script will increase the speed of the analysis a lot
    • script implemented, but not tested due to copying not in place yet (see above).

New tasks:

  • corpus dtd documentation:
    • structure, content/model and location of the dtd (location = permlink) http: //giellatekno.uit.no/dtd/corpus.dtd TODO: Saara to write and finish the docu, also check the dtd link
      • the link isn't working
  • finalize gt/doc/ling/corpus_conversion_tech.html (and possibly convert it to xml) (Saara)
  • add xml validation against our dtd to the corpus conversion process ( Saara)

Changes and updates because of the Divvun public tender

  • routines for setting up new users of our corpus
    • create the final text corpus license: the regular end user computer license towards UiT/IT, to get an account and access to the corpus repository, be it through a web interface or the command line. We can probably as Roy Dragseth et co about a useful template.
    • Børre will be the recipient of the SD end user license (approving corpus users on behalf of SD). Who should be the recipient on behalf of UiT/IT?
    • who will do the actual account setup? What type of account is needed?
  • an automatic build of the content of our corpus repo:
    • for each text:
      • license attached to each text
      • length (words, characters, sentences, paragraphs)
      • (source) language and other properties of the text
      • for freely available texts, a download link?
    • an overview document with statistics for the whole corpus
    • the automatic build should generate one (or several) Forrest XML document(s) containing the above info

6. Infrastructure

We need to set up anonymous, read-only access to our cvs repo as outlined by our friend in Skolelinux. This is necessary both for the Divvun public tender, as well as for other cooperation with external bodies (e.g. Skolelinux). This setup needs to be accompanied by documentation for access, as well as for building, prerequisites, etc.

We might even consider seting up patching routines, such that people with read-only access can send in patches for bugs and enhancements they would like to contribute.

Our open-source policy now demands that we really take the step, and not only talk open-source: -)

Should we set up view-cvs (web interface to cvs)?

Problems:

  • IP-problemacy (Do we have the complete rights to the content, or does the work have a license that allows us to publish the them)
    • Sámi place names - we need to ask our sources:
      • Statens kartverk (Trond) - materialet er fritt, jf Stortinget!
      • Finnish same (Thomas) - we don't have it, so this is no issue. Reminder: we need to get it!
    • Are there any parts in smj, sms, etc that shouldn't be publicly available?
      • What about the inc-*.txt files in smj/src/ ? Are they the Anders K source files? Not exactly, but they are derived from it. We will have to remove them from the CVS repository when we're finished with them.
      • do we have any corpus files in smX/corp/ with copyright/unchecked lisence? Yes, Trond needs to check this this week.
    • No other problems? Pekka is fine with this.
      • = No other problems.
  • Do we dare to show our work?
    • yes

Howto/who:

  • what do we need?
    • web interface? maybe
    • command line check-out? yes (Roy Dragseth / Børre)
    • need to be able to restrict anon. cvs to only specific modules ( Roy Dragseth / Børre)

7. Linguistics

North Sámi

We want to get the decisions from the SGL meeting in December.

  • Laila has sent those to Thomas by email ...a long time a go...
  • Thomas will include the responses in our normativity document.

SGL do not want us to send "unnecessary" things like rievan - rieban, politihkka - bolitihkka, pievlat - bievlat. And I agree with Laila. We are wasting their time. We have to write rieban-politihkka-bievlat. These are decided already year 1982-1988 1992-94. We have to read what Ole Henrik Magga has written long time a go about the decissions they (Finland, Norway and Sweden) made 1982-1988, and 1992 (?).

We have to make a list and send the list to SGL. They do have a meeting probably in May.

TODO:

  • Find the book by O. H. Magga on normativity desicions made in the 80'ies, and documentation on all later decisions:
    • Dieđut nr 2 1985: GIELLA, dutkan, dikšun ja oahpaheapmi.Sámi instituhtta. s. 32 - 74 (davvisámi čállinvuohki - Ole Henrik Magga), s. 68-74 (Pekka Sammallahti)
    • documentation on later decisions from Laila, Maaren will get copies and send them to Thomas
  • make sure we only send them issues that are really open
  • new issues need probably to be better explained from the point of view of spelling correction and end user reactions - that is, why we think it is an issue in need for SGL clarification

TODO:

  • get the decisions (Maaren)
    • done

Lule Sámi

Goal for March 1st: Reach the 90 percent lexical recall limit.

TODO:

  • Trond and Thomas: work on the 90 % limit
    • we reached 92%!
  • Trond: Write a report to NFR.
    • done

8. Name lexicon infrastructure

Complex names

  • make sure xml2lexc can handle complex names in ways compatible with our present tool chain (=reconstruct the lexc format we have now) (Tomi)
    • the resulting file format should be identical to our present prop-name file (=lexc), that can then be converted to our new xml format using the same script as for the regular names (Tomi or Saara, but only when the technical details are settled)

TODO:

  • Move this issue to bugzilla (Børre)

XML format

TODO on eXist as editor:

  1. refactor and prepare risten.no for multiple collections:
    1. develop the Cocoon sitemap to delegate requests to the proper folder level, such that the most specific code is always used
    2. refactor the code into more and more specific components according to our folder hierarchy
  2. develop the needed XQueries and interface (Sjur, Tomi)
  3. data synchronisation between risten.no and the cvs repo (Tomi)
  4. test whether eXist as editor is actually working well (linguists)

Data synchronisation task list/specification:

  • the xml file needs to be stored/updated in cvs
  • there should be no diffs on whitespace and sorting order (to ensure we get only substantial diffs, and no noise)
  • the prop name update cycle should something like:
    • dump the xml from eXist (in proper sorting order)
    • check whether there are diffs against cvs; continue only if there are
    • update from cvs
    • error check: are there conflicts? if yes, send report to <somebody>
    • are there still diffs? if yes, continue:
    • check in/commit w. generated comment
    • error check: is the document valid and conformant xml? if no, stop and send a report to <somebody>
    • reimport the xml file into eXist
  • question: do we need to lock the file in eXist through this update cycle?
  • the update cycle should be a nightly cron job

9. Spellers

Nothing until the new proper noun lexicon is in place.

10. Other

SGL Seminar

SGL don't want a seminar - at least not yet.

Bug fixing

35 open Divvun/Disamb bugs, and 24 risten.no bugs

11. Summary, task list

Børre

  • send out contracts with accompanying letter
  • Gather public texts, preferrably also parallel ones
  • Continue converting text from input format to our xml
  • convert nob and nno bible texts to be used as part of a parallel corpus
  • review the paratext2xml converter
  • convert smj NT to paratext
  • Call Ove Sæth
  • Move complex name lexicon issue to bugzilla
  • Ask KIO Grafisk to make a test Quark document based on a Word document from us
  • Send out letters to the Iđut authors
  • Add corpus security re G5 syncing as an issue to Bugzilla
  • fix bugs!

Maaren

  • work with the top-ten list
  • send copies of normativity decisions from 1985 to 1992 (2003?) to Thomas

Saara

  • continue discussion on the new lexicon format
  • Fix the preprocess script and optimize it.
  • Try to add numeral treatment as part of the analyzator.
  • Move gt2ga.sh to G5 and implement copying of the gt-dir.
  • Create a parallel corpora of the new testaments.
  • Routine for adding new languages to the propernoun xml-structure.
  • Move to Bugzilla: the analyzer needs to be optimized.
  • Implement validation of xml corpus against the dtd.
  • Create a group for corpus users.
  • Finish corpus dtd documentation, dtd location and permlink reference
  • Finish gt/doc/ling/corpus_conversion_tech.html and rename to .xml
  • fix bugs!

Sjur

  • Follow up the lawyer treatment of the contracts
  • Lule Sámi twol problems, with Thomas and Trond
  • project planning with Trond, continued
  • Follow up on place names from Norge Digitalt
  • Evaluate SFST as speller (and analyzer) lexicon
  • write a background document on the corpus contracts
  • public tender:
    • answer requests/questions
    • set up anon. read-only cvs with Børre
    • corpus repo access
  • smj G3 issue with Thomas and Trond
  • sme G3 issue with Thomas and Trond
  • call EDD/ Christian Emil Ore about national place name lexicon
  • risten.no/proper noun lexicon development:
    • refactor code
    • implement inheritance/collection overriding for xsl/css/xquery using sitemaps
  • fix bugs!

Thomas

  • add incoming Lule sámi words
  • include the SGL: decisions in our normativity document
  • include normativity desicions made by Magga and Sammalahti in our normativity document
  • follow-up on the Sámi place names in Finland
  • work on North Sámi compounding and derivation
  • smj G3 issue with Sjur and Trond
  • sme G3 issue with Sjur and Trond

Tomi

  • move aspell UTF-8 suffix bug to Bugzilla
  • corpus infrastructure:
    • dtd location (both public and internal)
  • Document aspell infrastructure: finish doc/proof/spelling/X-spell/aspell.xml (it's almost done, but there are a couple of loose ends)
  • new proper name lexicon
    • discuss the new lexicon format and other issues in the newsgroup
    • Look into data synchronisation of proper nouns between risten.no and CVS
    • XQuery refactoring and code development for our proper noun editor
    • new version of xml2lexc (based on ccat), should handle complex names correct: construct entries like we have now from the different parts of a complex name entry
  • fix bugs!

Trond

  • Bring Lule Sámi up to 90 %
  • Write Lule Report to NFR
  • Apply for strategy funds
  • Work on corpus texts with Børre.
  • Contact the Finnish and Swedish Bible societies to get Bible texts.
  • Look at ga/ directory issue with Saara
  • Ask for a Quark test file from his sister
  • News group discussion followup.
  • clean all texts from gt/smX/corp - they need to be in our corpus repo before we open our cvs for anonymous access
  • fix bugs!.

12. Next meeting, closing

13.03.2006 09: 30

Closed at 12: 07