NDS Updating Dictionaries

This page documents both lexicon and fst updating, and restarting of the server. One may update either lexica or fst or both, but in both cases configuring and resetting of the surver must be run.

Administering NDS requires the use of Fabric, which is a system for performing processes remotely.

All of the examples here use the gtweb dictionaries, and require logging in as the neahtta user. Note that on gtoahpa, the compile process is restricted, so that only the lexicon will be able to be compiled, but not the FST files. FSTs must be compiled manually (see below in Updating the FSTs).

NB: Neahttadigisánit (sme, sma, etc.) or sanit.oahpa.no/sánit.oahpa.no is currently on gtoahpa, but other services are located on gtweb: sanat.oahpa.no, valks.oahpa.no. For these you will need to log into gtweb, though the following examples may show gtoahpa as the hostname in the command prompt

Updating the lexica on gtweb and gtoahpa

For the impatient: The short version:

fab PROJECT compile
fab PROJECT test_configuration
fab PROJECT restart_service

The longer explanation:

1.) Log in to the server via SSH as the NDS user (user name:neahtta)

If you have root access, log in to gtweb, and thereafter do sudo su neahtta If you do not have root access, you can still log in by having your public SSH key added to neahtta's authorized_keys. See SSH on how to make an SSH keypair if you don't have one already, then ask someone with access (e.g. Kevin/Trond/Jaska/Ryan) to append your public key to the /home/neahtta/.ssh/authorized_keys file.

Note that when logged in as the NDS user, the python virtualenv should be activated automatically, and you will see this before the command prompt:

    (env)[neahtta@gtweb ~]$ 

(If you do not see this, do the following commands from the home directory of neahtta: cd ~ && source env/bin/activate.)

When you see (env) in the command prompt, continue.

2.) Go to the neahtta catalogue and run the Fabric process

	cd ~/neahtta/
	fab DICT compile_dictionary

Replace DICT below with sanat, vada, etc. (to fab vada compile_dictionary etc.)

If you have problems here, make sure that the environment variables for GTHOME, and GTCORE are set, however the neahtta user should automatically be configured properly. Either you will see errors, or you can check with echo $GTHOME. The neahtta user has these set automatically in its bash profile.

3.) Check that there were no errors, also wc -l dicts/*.xml to make sure there is content in the files.

If there is an error in an XML file used in compilation, Saxon will print out the file name and line that was problematic. Before compilation, a backup file will be stored, so if the compilation process overwrites this with a blank file, you may revert to a previous version. Backup files are named *.bak, and include a timestamp.

This process compiles all dictionaries to dicts/, which is the place that most instances of NDS rely on, following the relevant configuration file in configs/DICT.config.yaml. This will usually be enough, but if updates do not seem to be visible on the web, it is a good idea to check that the dictionaries are in the locations that the config expects, and alternatively restarting the server process.

NB: The files checked in to SVN are different from those actually used in production on the server, this is to prevent accidental overwritings via svn up. Thus, you will need to edit and check in configs/DICT.config.yaml.in, which is fine for use in development work, but the servers instances will be running from confgis/DICT.config.yaml.

4.) Test the configuration files

An automatic tool to check that everything went well is also available.

Running the following command will evaluate the config, test dictionaries, and then print FST paths and last updated date. If an FST is missing from its expected path, it will be listed as MISSING. If you see any errors at the end of the process, or worse, Python errors, something is wrong and you should avoid restarting until this is corrected.

    fab DICT test_configuration

5.) Restart the server process

When everything is working, run the following:

    fab DICT restart_service

Updating the FSTs

There are two ways to update the FSTs. For both of these options, you must know first where the FSTs for each dictionary and language should lie. FST locations are defined in the relevant config file in configs/DICT.config.yaml, in the Morphology section near the top. (Note the difference mentioned above between .yaml.in and .yaml.

As above, you can use the test command to see if the files were updated.

    fab DICT test_configuration

If you see any errors, be sure to correct them.

Updating on your own (NB:necessary on gtoahpa)

The first way to update FSTs is to do so on your own, using whichever method you are comfortable with, typically following the usual procedure for $GTSVN/langs/, and then copying them manually to the specified locations.

To find the FST locations:

    fab DICT test_configuration

This will output the following:

    SoMe:
      FOUND:   /opt/smi/sme/bin/analyser-dict-gt-desc-mobile.xfst
      UPDATED: Tue Nov  4 15:47:31 2014

      FOUND:   /opt/smi/sme/bin/generator-dict-gt-norm.xfst
      UPDATED: Tue Nov  4 15:47:31 2014


    sme:
      FOUND:   /opt/smi/sme/bin/analyser-dict-gt-desc.xfst
      UPDATED: Tue Nov  4 15:47:31 2014

      FOUND:   /opt/smi/sme/bin/generator-dict-gt-norm.xfst
      UPDATED: Tue Nov  4 15:47:31 2014

    ... snip ...

When you compile the analyzers on your own, copy them to these paths, and test that their permissions allow them to be accessible to the neahtta user.

Updating via Fabric

Fabric provides a quick shortcut to the build processes for each language.

When you are logged in and in the ~/neahtta/ directory, you may run these commands.

Compiling FSTs with Fabric is as follows: you can compile all FSTs individually with fab DICT compile_fst:ISO, or, fab DICT compile where DICT is the short name for the project. The latter will compile all FSTs belonging to the DICT, together with the lexicon.

    fab DICT compile_fst:ISO
    fab DICT compile

When you run these, follow the output to make sure no errors have occurred. The process will automatically install the new FST files in their respective locations, but only if the compilation process was successful.

Updates to FSTs do not require restarting the server, and changes will be made live immediately. If you update the lexicon however, you shouold restart the process.

Either of these will compile the FSTs in their original locations as defined in the Makefile (which simply follows the procedures for $GTSVN/langs), but it will also copy FST files to temporary locations within the dictionary config directory as a prerequisite to the install targets.

Testing the configuration

Go to neahtta and configure the dictionaries

 
    cd ~/neahtta/
    fab DICT test_configuration

Here, replace DICT with the relevant name in configs/ that you are working on (the list of DICTs above).

If everything is good and there are no errors, you'll see FST paths, and some happy cyan text at the end.

Resetting the server

Either use the fab process, or relevant system commands.

 
    cd ~/neahtta/
    fab DICT restart_service

NB: you may be prompted for your the neahtta sudo password, and if this doesn't work, something is broken and developers must fix it.

Production server notes

Production servers currently check code out from tags/apps/dicts/nds/nds-stable-prod, and configuration subdirectories are checked out from trunk (svn switch). This means that only important prod-ready changes will be available from an svn up, but linguists may still work in configuration directories and update them when they will.

More on Fabric

The Fabric system.

The Fabric system itself (cf fabric.org is run, and documented, in neahtta/fabfile.py. This file contains both the actual commands and some documentation.

Using Fabric

Most of the process here is delegated to other commands, and this is because Fabric is a system for executing commands locally or remotely on other systems.

Two benefits for people who need to run these processes frequently are:

  • chaining commands together
  • running remotely

Commands can also be chained, for example (here with guusaaw as DICT):

    fab guusaaw compile test_configuration restart_service

This would execute everything after guusaaw in order: compilation, testing, and restart. Chaining restart_service is not recommended, one should inspect the output of test_configuration (that everything is ok) before restarting.

More comments on the CENTOS (gtoahpa) operative system

In order to start with reboot on CENTOS:

    chkconfig --list  (list all)
    chkconfig SERVICE on

where SERVICE is a variable for nds-sanit.service, etc.