Submitting Data to the MOTIF Database


 

Preparing the data

Before submitting data to the MOTIF database, you should should follow the steps outlined in the Experimental Design section:

Do not forget to check...

Data file names

Please note that UNLIKE IPCC we request that THE DATA FILES HAVE UNIQUE NAMES across the WHOLE database, and use the following data file name template:

    varname_table_experiment_modelorlab[_time-range].nc

table being one of the PMIP2 CMOR tables short names: A_FX, A_SE, A_MO, A_DA, O_SE, O_AN, O_MO, S_SE, S_AN, S_MO, I_SE, I_DA, I_MO

experiment being one of: pmip2_0k_oa, pmip2_0k_oav, pmip2_6k_oa, pmip2_6k_oav, pmip2_21k_oa, pmip2_21k_oav

modelorlab should be composed of uppercase and/or lowercase characters, numbers and/or the '-' (dash) and/or '.' (dot) characters.

If a data file has to be split into several sub-files along the time axis, use either
    _yyyy-YYYY
if the file covers the whole years yyyy to YYYY, or
    _yyyymmdd-YYYYMMDD

There is no convention for year dates to use for the paleo experiments, so you might as well start all the paleo experiments on January 1st 0001 (i.e. 0001 or 00010101 and the units of the time axis should be 'days since 0001-01-01').
You are of course free to follow whatever dating scheme you want in your actual runs, but unified dates in the database may make some comparison/processing steps a bit easier...

The logic behind the file names

There is some logic in the file naming scheme that may be useful for programmers! The major fields of a file name are separated by '_' characters, which makes it easy to extract the different fields of a file name by just splitting it where there are '_' characters...

shell scripts users:

> fname="ts_A_MO_pmip2_0k_oa_HadCM3-MOSES2_0000-0029"
> echo $fname | sed 's/_/  /g'
ts  A  MO  pmip2  0k  oa  HadCM3-MOSES2  0000-0029
> model=`echo $fname | cut -d_ -f7`
> echo $model
HadCM3-MOSES2
python scripts users:
>>> fname = "ts_A_MO_pmip2_0k_oa_HadCM3-MOSES2_0000-0029"
>>> ffields = fname.split('_')
>>> ffields
['ts', 'A', 'MO', 'pmip2', '0k', 'oa', 'HadCM3-MOSES2', '0000-0029']
>>> model = ffields[6]
>>> model
'HadCM3-MOSES2'

Do not forget to check...

Try to use correct file names (even if the database file insertion script can handle most common error and will correct the file names on the fly)!

Also, make sure that your data files generating codes/scripts that can handle a new naming scheme, if we have to make some changes (minor changes, hopefully)...

Based on the data that has already been submitted to the database, you should pay attention to the following details:

Sending the data

The data files will have to be sent to LSCE by anonymous ftp and/or scp (if you already have an account at LSCE).

Data transfer by tape should only be used as a last resort (if transfer over the network appears to be extremely slow), after agreeing with the database administrator on what tape format should be used.

Remember that we are not requesting as many vertical levels for the MOTIF project as for IPCC, so the amount of data to transfer is (much) smaller.

The data files should be stored in the following directory hierarchy

    modelorlab/yyyymmdd/experiment/table
        where yyyymmdd is the submission date

and sent either as is (the hierarchy is replicated directly in the directory where the data is uploaded) or in a compressed tar file with the name

    modelorlab_yyyymmdd_experiment_table.tar.gz

        where yyyymmddA is the submission date followed by a letter (a to z) if several tar files are sent

whichever is more convenient.

Whatever transfer method is used, always keep an exact copy of the data you send, in case it needs to be sent again or reprocessed from scratch (we are not expecting a major database crash, but it is always best to be prepared... :).

And it is of course a good idea to keep all the programs/scripts you used to prepare your data in case your files need to be processed again (either because somebody find some errors in your files, or because the output standards need to be changed).

Sending the data by anonymous ftp

You should carefully follow those steps when you use this submission method:

Sending the data by scp

You should carefully follow those steps when you use this submission method:

Checking the processed data

...


Home Top Site Map @ Last updated \2008/06/19 09:16:54