ECB Gesmes/CB Checker

ECB Statistical Information Systems	Document Reference: ECB/SIS/2006/050
User Manual for the ECB Java Suite
Version 1.3	Issue: 26 May 2014

1. Introduction

This document provides instructions for a quick installation and use of the ECB Java Suite. The ECB Java Suite software allows to perform syntactic and semantic (against the provided structural definitions files) checks of SDMX messages. The software is written in Java and it requires the Java Runtime Environment to be installed on user's PC (see Section Minimum system Requirements).

The package comprises the following files:

	What is it?	Essential for checker operation?
Core Java application
`gesmescb.jar`	Contains the actual Java class files. JAR means "Java Archive".	Yes.
Checker specific (GUI version)
`awui.cnf`	Persistent configuration memory for the checker GUI. Updated automatically to reflect option changes.	No, but without it, use of the GUI may be less convenient.
Checker specific (command line version)
`checker.bat`	MSDOS shell command which encapsulates the call to Java with correct parameters for checking.	On Windows yes, UNIX no.
`checker.sh`	UNIX shell command which encapsulates the call to Java with correct parameters for checking.	On UNIX yes, Windows no.
`checker.cnf`	Configuration file for checking.	Yes.
Loader specific
`loader.bat`	MSDOS shell command which encapsulates the call to Java with correct parameters for loading to FAME.	No. It might be useful only for institutions using FAME; it allows loading onto FAME databases
`loader.sh`	UNIX shell command which encapsulates the call to Java with correct parameters for loading to FAME.	No. It might be useful only for institutions using FAME; it allows loading onto FAME databases.
`loader.cnf`	Configuration file for loading to FAME	No. It might be useful only for institutions using FAME; it allows loading onto FAME databases.
Structural Definitions
`StructuralDefinitions`	Directory for structural definitions of a maintenance agency in Gesmes/TS format. Here you must put the structural definitions files as published by e.g. the BIS or ECB. The Structural definitions must be those in Gesmes/TS or Gesmes/TS format.	Yes: essential for the semantic checks. You must put here an up-to-date file.
Documentation
`readme.txt`	Short information about the current version.	No.
`UserManual.html`	This file; installation, running, and maintenance of the checker.	No.
`checkerdirs.png`	Diagram for `UserManual.html`.	No.
`license.txt`	ECB public licence.	No.
Related to JAR signature
`key.txt`	Short guide to check integrity of `gesmescb.jar`.	No.
`ecb-ioana.key`	Public key for use with `jarsigner`.	No. Useful to check integrity of updates of gesmescb.jar.

The program can be installed on a PC or on a UNIX system. This guide only describes the PC installation, but the setup on UNIX is similar. In place of checker.bat you then use checker.sh.

2. Installing the ECB Java Suite

2.1. Installing Java

Precondition for using the software is a set up Java Runtime Environment on your computer. You can test if you have one from the command prompt by typing

java -version

and the return key. If you have Java properly installed, you should get a message like

java version "1.2.2"

Classic VM (build JDK-1.2.2-W, native threads, symcjit)

Otherwise, you will get an appropriate error message, e.g.

The name specified is not recognized as an

internal or external command, operable program or batch file.

and you will need to get Java properly installed. Note that the PATH system variable must include a path to the Java executable. You may want to ask your local help desk for assistance on setting up Java.

2.2. Installing the software

Let's assume that we want to use the Suite on a PC on drive C. The ECB software can be stored on a directory called, for example, C:\ECBchecker. Then all you have to do is simply copy the files checker.bat, checker.cnf, and gesmescb.jar to this directory. Of course you may copy more files, particularly you may have to copy license.txt for legal reasons.

2.3. Installing the structural definitions

The structural definitions files need to be saved somewhere, e.g. under C:\StructuralDefinitions. So, the following tree structure could be used:

Note that the structural definitions are no longer part of the checker distribution. You must supply them yourself to the directory C:\StructuralDefinitions. Refer to your structural definitions maintenance agency about how to obtain the structural definitions.

2.4. Upgrading the checker software

If you have already a version of the checker installed, you may not want to erase this installation and install from scratch. One reason for this can be that you have customised one or more of the files checker.cnf, checker.bat, or checker.sh to your local needs and want to keep these modifications. Then all you should do is to copy the files gesmescb.jar, checker.bat (checker.sh for Unix) to your checker directory overwriting the previous version of this file with the same name.

2.5. Updating structural definitions

From time to time your maintenance agency will issue a new version of its structural definitons. To let the checker use these, do the following:

Put the new structural definitions file ESCByyyy'_mm'_dd'.gsm into the StructuralDefinitions directory.
Remove the old file (e.g. C:\StructuralDefinitions\ESCB2002_09_11.gsm).

2.6. Using the ECB Java Suite (command line version) with multiple maintenance agencies

Structural definitions from more than one maintenance agency can be put into the directory StructuralDefinitions. When the checker runs, it first reads all files from this directory. It will find out from the data file you check which structural definitons to use.

In normal circumstances this approach is simple and sufficient. However two problems can emerge: The structural definitions files are loaded in the order that your operating system expands "*.gsm". If your structural definitions files have dependencies and require a special order to be kept, this can cause a problem. Also if you have a vast set of structural definitions files, reading all of them at program startup may be inefficient or even impossible.

In these cases, several checker script (.sh or .bat) files can be created, with each one of them using different structural definitions files. For example:

The script checker.bat (or checkerECB.bat) could contain the following command and arguments:

·

·         java -cp gesmescb.jar -DInt.ecb.logging.config.class=Int.ecb.gesmescb.util.LoggingPropertiesReader -DInt.ecb.logging.config.file=checker.cnf -DInt.ecb.prefs.config.file=checker.cnf Int.ecb.gesmescb.application.Loader ..\StructuralDefinitions\ESCByyyy_mm_dd.gsm %1 %2 %3 %4 %5 %6 %7 %8 %9

Thus, the command checkerECBfilename would check against the ECB's structural definitions.

If we assume that EUROSTAT has included its structural definitions in the file EUROSTATyyyy_mm_dd.gsm, then we could create a file called checkerEUROSTAT.bat that contains the command and arguments:

·

·         java -cp gesmescb.jar -DInt.ecb.logging.config.class=Int.ecb.gesmescb.util.LoggingPropertiesReader -DInt.ecb.logging.config.file=checker.cnf -DInt.ecb.prefs.config.file=checker.cnf Int.ecb.gesmescb.application.Loader ..\StructuralDefinitions\EUROSTATyyyy_mm_dd.gsm %1 %2 %3 %4 %5 %6 %7 %8 %9

Thus, the command checkerEUROSTAT filename would check against the EUROSTAT structural definitions.

If we assume that the BIS has included its structural definitions in the files BIS1.gsm and BIS2.gsm, then we could create a file called checkerBIS.bat that contains the command and arguments

·

·         java -cp gesmescb.jar -DInt.ecb.logging.config.class=Int.ecb.gesmescb.util.LoggingPropertiesReader -DInt.ecb.logging.config.file=checker.cnf -DInt.ecb.prefs.config.file=checker.cnf Int.ecb.gesmescb.application.Loader ..\StructuralDefinitions\BIS1.gsm ..\StructuralDefinitions\BIS2.gsm %1 %2 %3 %4 %5 %6 %7 %8 %9

Thus, the command checkerBIS filename would check against the BIS structural definitions.

If we assume that ECB, EUROSTAT, and the BIS have included their structural definitions in files named as above, then we could create a file called checker3.bat that comprises the command and arguments:

·

·         java -cp gesmescb.jar -DInt.ecb.logging.config.class=Int.ecb.gesmescb.util.LoggingPropertiesReader -DInt.ecb.logging.config.file=checker.cnf -DInt.ecb.prefs.config.file=checker.cnf Int.ecb.gesmescb.application.Loader ..\StructuralDefinitions\ECByyyy_mm_dd.gsm ..\StructuralDefinitions\EUROSTATyyyy_mm_dd.gsm ..\StructuralDefinitions\BIS1.gsm ..\StructuralDefinitions\BIS2.gsm %1 %2 %3 %4 %5 %6 %7 %8 %9

Thus, the command checker3 filename would check against ECB, EUROSTAT, or BIS structural definitions according to which maintenance agency is specified in the Gesmes/TS file to check. This works because the plain Java application in gesmescb.jar takes all arguments except the last one as structural definitions and only the last one as the file to check (or load). (Note for experienced users: You can also specify structural definitions on the command line as further arguments before the file to check:

checker3.bat mystrucdefs.gsm c:\testdir\test.gsm

In other words, the commands checkerECB, checkerEUROSTAT, checkerBIS would allow to check against only the structural definitions of the ECB, Eurostat and the BIS respectively, the command Checker3, however, would be able to use the structural definitions of all these three organisations.

3. Using the ECB Java Suite

3.1. Running the checker GUI

Since recently, the ECB Java Suite can be run using an interactive graphical user interface (GUI). Browse the directory ECBchecker and start gesmescb.jar therein by (double-)click. After you set the default structural definitions, you can drag and drop Gesmes/TS files to the GUI's text area to check them. Consult the online help for further directions.

The GUI is meant as an easy-to-use variant to check Gesmes/TS data. If this is enough for you, feel free to skip the rest of this chapter. There, you learn how to exploit from the command line features not available in the GUI.

For checker and converter part, use the package sdmxTools and read the Readme.txt file.

3.2. Running the Suite from command line

Let's assume an installation as proposed. And consider a file called test.gsm in directory C:\testdir that needs to be checked. For this we need to go to the command prompt (MSDOS prompt) and enter the commands (without the prompt):

C:\>cd ECBChecker

C:\ECBChecker>checker C:\testdir\test.gsm

In fact, the command which is executed (checker.bat) makes use of the following longer command:

java -cp gesmescb.jar -DInt.ecb.logging.config.class=Int.ecb.gesmescb.util.LoggingPropertiesReader -DInt.ecb.logging.config.file=checker.cnf -DInt.ecb.prefs.config.file=checker.cnf Int.ecb.gesmescb.application.Loader ..\StructuralDefinitions\*.gsm %1 %2 %3 %4 %5 %6 %7 %8 %9

The argument ..\StructuralDefinitions\*.gsm before the parameter argument(s) (starting with the % sign or $ in UNIX) specifies the location and the file name of the structural definitions file to be used according to the proposed installation. So, if the user wanted to use another structural definitions file or another directory where the structural definitions would be stored, he/she would have to edit the file checker.bat accordingly before execution.

The output (result of the check) is sent to the screen. See Section Example for sample output messages.

3.3. Calling the Suite from another directory

By default, the checker can only be called from the directory where it is installed. This is inconvenient. If you want to overcome this problem, you need to edit the file checker.sh or checker.bat. Modify all references to files to use an absolute path name. E.g. if you have installed the checker directory tree in C:\, the command line

java -cp C:\ECBchecker\gesmescb.jar -DInt.ecb.logging.config.class=Int.ecb.gesmescb.util.LoggingPropertiesReader -DInt.ecb.logging.config.file=C:\ECBchecker\checker.cnf -DInt.ecb.prefs.config.file=C:\ECBchecker\checker.cnf Int.ecb.gesmescb.application.Loader C:\StructuralDefinitions\*.gsm %1 %2 %3 %4 %5 %6 %7 %8 %9

will do the job.

3.3. The log file

The output (result of the check) is sent to the screen. In many cases it is useful to have the output go to a file, too. Often, a simple output redirection like

checker C:\testdir\test.gsm 2> log.txt

will do the job. If, however, you need log files on a regular basis, you may want to change the configuration so that a log file is generated with each run. To do so, change the logging setup in the configuration file checker.cnf to read:

# Log to console

# handlers = Int.ecb.logging.ConsoleHandler

# Log to console and file

handlers = Int.ecb.logging.FileHandler Int.ecb.logging.ConsoleHandler

Then, due to the line

Int.ecb.logging.FileHandler.pattern=testlog.log

logs will go to testlog.log. Of course you can change this configuration to use a different name. If you want to use dynamically generated names for the log files, you should edit the file checker.bat or checker.sh. For this, please observe the pattern suggested in the section about the acknowledgement file.

3.4. The acknowledgement file

The configuration file checker.cnf can be changed for producing an acknowledgement file. By default, the configuration line to write a file ack.txt is commented out. All you have to do is remove the hash mark so that the line becomes:

Int.ecb.gesmescb.error.AcknowledgementManager.acknowledgementFile = ack.txt

Of course, the name of the acknowledgement file can be changed in the configuration. Using a command line

java -cp gesmescb.jar -DInt.ecb.logging.config.class=Int.ecb.gesmescb.util.LoggingPropertiesReader -DInt.ecb.logging.config.file=checker.cnf -DInt.ecb.prefs.config.file=checker.cnf -DInt.ecb.gesmescb.error.AcknowledgementManager.acknowledgementFile=%2 Int.ecb.gesmescb.application.Loader ..\StructuralDefinitions\*.gsm %1

in place of that in checker.bat, the acknowledgement file could even be determined on the command line. Such a script would take two arguments, first the input filename, second the acknowledgement filename.

This construct demonstrates the general pattern to transfer control from the configuration file to the script or the command line. Note that the pattern can be applied analogously to checker.sh on a UNIX platform.

4. Writing data

The Gesmes/TS suite can not only used to check, but also to translate Gesmes/CB and Gesmes/TS data. There are a few writing modules buildt in. Others you can develop yourself and plug in. This section describes the available modules. Please refer to developer documentation for further information. You kan skip this chapter if you do not intend to use the checker to create FAME databases from Gesmes/TS files.

4.1 Using the FAME Loader

If you have in addition to the requirements for the checker also FAME/TimeIQ

(version 1.2) installed on your computer, you can build FAME databases using the loader. Just make sure that your CLASSPATH includes the paths of the files timeiq.jar, TimeIQLicense.jar.

The loader can then be called analogously to the checker:

C:\ECBChecker>loader C:\testdir\test.gsm

The configuration file loader.cnf instructs the loader to write to the database file output.db. If needed, you can modify the filename in that configuration or file. The better solution ion most cases will be to edit the script file loader.bat or loader.sh so that it takes the database filename as an additional argument. The call to Java could look like:

java -cp $CLASSPATH:gesmescb.jar -DInt.ecb.logging.config.class=Int.ecb.gesmescb.util.LoggingPropertiesReader -DInt.ecb.logging.config.file=loader.cnf -DInt.ecb.prefs.config.file=loader.cnf -DInt.ecb.gesmescb.fame.SimpleESDBWriter.destination=%2 Int.ecb.gesmescb.application.Loader ../StructuralDefinitions/ESCB2002_05_28.gsm %1

Such a script would take two arguments, first the input filename, second the output database filename.

Finally be aware that the difference between the Checker and the Loader is essentially a different configuration. So first make sure the Checker runs according to your expectations. If it does not, the Loader will surely not do so, either.

4.2 Writing CSV

Some non-Gesmes applications out in the wild sometimes need CSV (comma separated values) file format. There is one module available to write tiome series in one particular flavour: The EurostatWriter. To activate it, add the following lines to the configuration file checker.cnf:

Int.ecb.gesmescb.application.Loader.useEurostatWriter = true

Int.ecb.gesmescb.csv.EurostatWriter.destination = output.csv

Then all time series will go to the file ouput.csv. Note that attributes will not be written!

4.3 Writing SDMX-ML

To write SDMX-ML Version 1.0 compact format, use the SDMXMLWriter. Its current version does not enforce the uniqueness constraints, i.e. that each group or series appears only once in the file. To activate it (and write to the file sdmx.xml), add the following lines to the configuration file checker.cnf:

Int.ecb.gesmescb.application.Loader.useSDMXMLWriter = true

Int.ecb.gesmescb.xml.SDMXMLWriter.destination = sdmx.xml

Int.ecb.gesmescb.xml.SDMXMLWriter.encoding = UTF-8

Int.ecb.gesmescb.xml.SDMXMLWriter.indent = tab

You can also provide schema locations for the SDMX schemas:

Int.ecb.gesmescb.xml.SDMXMLWriter.message.schema = https://stats.ecb.int/stats/vocabulary/sdmx/1.0/SDMXMessage.xsd

Int.ecb.gesmescb.xml.SDMXMLWriter.structure.schema = https://stats.ecb.int/stats/vocabulary/sdmx/1.0/SDMXStructure.xsd

In addition, you must add a line for each maintenance agency you use to set the namespace URI for key families. Again, you can optionally provide schema locations for the key families. This section could look like:

Int.ecb.gesmescb.xml.KeyFamilyURIMapping.ECB.*.namespace = http://www.ecb.int/vocabulary/stats/kfidx()

Int.ecb.gesmescb.xml.KeyFamilyURIMapping.ECB.*.schema = http://www.ecb.int/vocabulary/stats/kfidx()/date()/sdmx-compact.xsd

Int.ecb.gesmescb.xml.KeyFamilyURIMapping.BIS.*.namespace = http://www.bis.org/vocabulary/stats/kfidx()

5. For previous users of GESLATE and POLYVAL

Below you find a list of the main differences between the "ECB Gesmes/TS Checker" and the Geslate/Polyval checking software:

	Geslate/Polyval Checker	ECB Gesmes/TS Checker
Can it check messages based also on other Gesmes profiles than Gesmes/CB and Gesmes/TS?	Yes (after an appropriate configuration of the "ddf" file).	No (the ECB Checker is Gesmes/CB and Gesmes/TS specific).
Semantic checks	Against a structural definitions database (see below)	Against a structural definitions file (see below)
Structural definitions repository	The structural definitions database is created and updated using specially written structural messages (the ECB used to distribute them in the past under the file names `ECByyCnn.gsm` and `ECByyFnn.gsm` for the code lists and structure definitions respectively). These files need to be "passed through" the checker in order to update the structural definitions.	The ECB Gesmes/TS Checker, in order to perform the semantic checks, reads/uses simply a "structural definitions" file in Gesmes/TS format (a kind of structural definitions database "on-the-fly"). For the ECB data exchanges, this structural definitions input file is the genuine `ESCByyyy_mm_dd.gsm` (old naming convention was `ECByyAnn.gsm`). As soon as it stored somewhere and the checker can locate and "read" it, probably in the same directory together with structural definitions from other maintenance agencies, it could be considered that the set of these files comprise the structural definitions of the software.
Possibility to use different versions of structural definitions coming from the same maintenance agency	No.	Yes (`checker.bat` or `checker.sh` must be copied and the copies edited).
List of allowed structural definitions agencies	A special file is needed with a list of the organisations that are considered legitimate structural definitions maintenance agencies.	Not needed. The structural definitions maintenance agency is specified in the input structural definitions file (see above) and should match with the agency specified in the message to be checked.
Input files on the command line	structural definitions file: in Gesmes format, but not Gesmes/TS Gesmes/TS file to be checked	structural definitions file in Gesmes/TS format Gesmes/TS file to be checked Header input file as a template for the acknowledgement file.
Output files	Log file Header file	Very detailed (messy) log file Acknowledgement file (optional)
Execution method	Command line.	Command line.
Switches on the command line	None	None recommended. (Expert only.)
Configurable options and arguments	Semantic checks: on/off Header file for data exchanges with the ECB: configurable settings Location of files (input, log, header)	Semantic checks off/errors only/errors and warnings Header file for data exchanges with the ECB Location of files (header, input, log, acknowledgement) Output in CSV or XML
Indicative performance (Windows NT, 320 MB, Pentium 3, 500MHz; checking 1.5 MB Gesmes/TS file)	5 minutes 15 seconds	16 seconds

6. Minimum system requirements

Memory (RAM)	32 MB
Operating System	Windows NT or XP, UNIX, other systems running Java.
Java Runtime Environment	Standard Edition, version 1.2 or later

7. Example

7.1. Sample call

checker test.gsm

7.2. Sample output

Enter the new time:

Log opened: 21/07/06 10:25

INFO::ECB's Gesmes/TS Suite

INFO::Implementation Version: 1.2.26

INFO::Configured to {check}

INFO::Starting to parse GESMES/CB(EDIFACT) input from test.ges.

INFO::Interchange: IREF011294

INFO::Sender: 4F4

INFO::Receiver: 4F0

INFO::Message: MREF000001

INFO::Sender: 4F0

INFO::Receiver: 4F0

INFO::Subject: SEC Data

INFO::Processed 1 megabyte.

INFO::Ending to parse GESMES/CB(EDIFACT) input from test.ges. Processed 1622805 byte.

INFO::Closed GESMES/CB(EDIFACT) input from test.ges.

INFO::Finished processing test.ges

No problems

7.3. Interpreting the report

In analysing the output, you focus on the lines from:

INFO::Starting to parse GESMES/CB(EDIFACT) input from test.gsm.

Possible warnings before this line are related to the structural definitions files. In the lines we focusing on, no WARNING or even less a SEVERE problem has occurred, the file test.gsm therefore has passed all checks.

Note that the checker program returns a nonzero value if a SEVERE problem was encountered and 0 otherwise. Hence, WARNINGs are essentially accepted, but you are made aware of their existence.

7.4. Severe semantic problems

An example of a severe problem report would be the message:

SEVERE::CodeList "CL_FREQ" does not contain value "G" in key "G.DK.1000.F33.N.2.EUR.U.Z".

To find out the exact location in the checked file, you would have to search in the input Gesmes/TS file for the key string G:DK:1000:F33:N:2:EUR:U:Z.

7.5. Syntactic problems

If the checked file contains syntactic errors, no semantic checking is possible. An error report is given which indicates the location in the file where the offending input was noticed. For instance, replace the segment tag ARR by ART in one segment to yield:

ART++Q:DK:1000:F33:N:4:EUR:U:Z:19934:608:12.7:A:F'

Then a message like

SEVERE::Parsing problem: Verify "UNT+" failed. In 1st message, Group 20, UNT-segment. File test.gsm, character 312, line 16, pseudocolumn 1, text "ART++Q:DK:1000:F33:N:4:EUR:U:Z:19934:608:12.7:A:F'ARR++Q:DK:...". [Verify "UNT+" failed.]

would be raised. As you see, it is not easy to find back the origin of the problem from this message. What happened is that the checker when reading all ARR segments encountered a segment other than ARR and therefore left the ARR reading loop. Since ART is not FNS, the only segment which could validly occur in this position is UNT; and that is what the checker expects. Interpreting syntactic problem messages unfortunately requires a lot of fantasy, as you see. But note that the position of the offending input is given in quite a detail. The message indicates line and number of character in the line and also tries to give the beginning of the text following the offending input. It is from this text sample that you best start when trying to find the cause of a syntax error (i.e. a parsing problem).

8. Feedback to developers and reporting bugs

Should you have any comments, remarks, suggestions, bug reports, please post them to the Gesmes/TS Suite developers' forum. If urgent response to a bug report is required, you may also mail to the hotline SIS.External@ECB.int.

When you report a bug, please tell the version of the checker, the operating system and version of Java that you are using. The version of the checker you see best in the log output on the screen. You can find out the version of Java by entering the command:

java -version

9. Disclaimer

Trademarks used in this document are owned by their respective owners. The missing reference to the individual owner(s) does not permit any assumption of the contrary.

User Manual for the ECB Java Suite

3. Using the ECB Java Suite

3.1. Running the checker GUI

3.2. Running the Suite from command line