NDS Updating Dictionaries

For the impatient: The short version:

    fab DICT compile
    fab DICT test_configuration
    fab DICT restart_service

The longer explanation:

1.) Log in to the server via SSH

Log in to gtdict, and thereafter do sudo su neahtta

Note that when logged in as the NDS user, the python virtualenv should be activated automatically, and you will see this before the command prompt:

    (neahtta_env)[neahtta@gtdict ~]$

(If you do not see this, do the following commands from the home directory of neahtta: cd ~ && source env/bin/activate.)

When you see (env) in the command prompt, continue.

2.) Go to the neahtta catalogue and run the Fabric process

	cd ~/neahtta/
	fab DICT compile_dictionary

Replace DICT below with sanit, baakoeh, etc. (to fab sanit compile_dictionary etc.)

If you have problems here, make sure that the environment variables for GTHOME, and GTCORE are set, however the neahtta user should automatically be configured properly. Either you will see errors, or you can check with echo $GTHOME. The neahtta user has these set automatically in its bash profile.

3.) Check that there were no errors, also wc -l dicts/*.xml to make sure there is content in the files.

If there is an error in an XML file used in compilation, Saxon will print out the file name and line that was problematic. Before compilation, a backup file will be stored, so if the compilation process overwrites this with a blank file, you may revert to a previous version. Backup files are named *.bak, and include a timestamp.

This process compiles all dictionaries to dicts/, which is the place that most instances of NDS rely on, following the relevant configuration file in configs/DICT.config.yaml. This will usually be enough, but if updates do not seem to be visible on the web, it is a good idea to check that the dictionaries are in the locations that the config expects, and alternatively restarting the server process.

NB: The files checked in to SVN are different from those actually used in production on the server, this is to prevent accidental overwritings via svn up. Thus, you will need to edit and check in configs/DICT.config.yaml.in, which is fine for use in development work, but the servers instances will be running from confgis/DICT.config.yaml.

4.) Test the configuration files

An automatic tool to check that everything went well is also available.

Running the following command will evaluate the config, test dictionaries, and then print FST paths and last updated date. If an FST is missing from its expected path, it will be listed as MISSING. If you see any errors at the end of the process, or worse, Python errors, something is wrong and you should avoid restarting until this is corrected.

    fab DICT test_configuration

5.) Restart the server process

When everything is working, run the following:

    fab DICT restart_service

There are two ways to update the FSTs. For both of these options, you must know first where the FSTs for each dictionary and language should lie. FST locations are defined in the relevant config file in configs/DICT.config.yaml, in the Morphology section near the top. (Note the difference mentioned above between .yaml.in and .yaml.

As above, you can use the test command to see if the files were updated.

    fab DICT test_configuration

If you see any errors, be sure to correct them.

Updating on your own

The first way to update FSTs is to do so on your own, using whichever method you are comfortable with, typically following the usual procedure for $GTSVN/langs/, and then copying them manually to the specified locations.

To find the FST locations:

    fab DICT test_configuration

This will output the following:

    SoMe:
      FOUND:   /opt/smi/sme/bin/analyser-dict-gt-desc-mobile.xfst
      UPDATED: Tue Nov  4 15:47:31 2014

      FOUND:   /opt/smi/sme/bin/generator-dict-gt-norm.xfst
      UPDATED: Tue Nov  4 15:47:31 2014


    sme:
      FOUND:   /opt/smi/sme/bin/analyser-dict-gt-desc.xfst
      UPDATED: Tue Nov  4 15:47:31 2014

      FOUND:   /opt/smi/sme/bin/generator-dict-gt-norm.xfst
      UPDATED: Tue Nov  4 15:47:31 2014

    ... snip ...

When you compile the analyzers on your own, copy them to these paths, and test that their permissions allow them to be accessible to the neahtta user.

Updating via Fabric

NB: on gtdict this will work only by removing 'gtdict.uit.no' in the variable no_fst_install in the fabfile.py.

Fabric provides a quick shortcut to the build processes for each language.

When you are logged in and in the ~/neahtta/ directory, you may run these commands.

Compiling FSTs with Fabric is as follows: you can compile all FSTs individually with fab DICT compile_fst:ISO, or, fab DICT compile where DICT is the short name for the project. The latter will compile all FSTs belonging to the DICT, together with the lexicon.

    fab DICT compile_fst:ISO
    fab DICT compile

When you run these, follow the output to make sure no errors have occurred. The process will automatically install the new FST files in their respective locations, but only if the compilation process was successful.

Updates to FSTs do not require restarting the server, and changes will be made live immediately. If you update the lexicon however, you shouold restart the process.

Either of these will compile the FSTs in their original locations as defined in the Makefile (which simply follows the procedures for $GTSVN/langs), but it will also copy FST files to temporary locations within the dictionary config directory as a prerequisite to the install targets.

Go to neahtta and configure the dictionaries

    cd ~/neahtta/
    fab DICT test_configuration

Here, replace DICT with the relevant name in configs/ that you are working on (the list of DICTs above).

If everything is good and there are no errors, you'll see FST paths, and some happy cyan text at the end.

Either use the fab process, or relevant system commands.

    cd ~/neahtta/
    fab DICT restart_service

NB: you may be prompted for the neahtta sudo password, and if this doesn't work, something is broken and developers must fix it.

The Fabric system.

The Fabric system itself (cf fabric.org is run, and documented, in neahtta/fabfile.py. This file contains both the actual commands and some documentation.

Using Fabric

Most of the process here is delegated to other commands, and this is because Fabric is a system for executing commands locally or remotely on other systems.

Two benefits for people who need to run these processes frequently are:

• chaining commands together
• running remotely

Commands can also be chained, for example (here with guusaaw as DICT):

    fab sanit compile test_configuration restart_service

This would execute everything after sanit in order: compilation, testing, and restart. Chaining restart_service is not recommended, one should inspect the output of test_configuration (that everything is ok) before restarting.