# CODA IDL

The CODA IDL interface consists of just a handful of IDL 'named structures' and functions. A primary design goal was to simplify, as much as possible, the number of different types and functions a user would have to master in order to effectively use the library.

## Contents

- CODA Definition Path
- Named Structures
  - `CODA_DATAHANDLE`
  - `CODA_ERROR`
  - `CODA_NO_DATA`
- Functions and Procedures
  - `CODA_VERSION`
  - `CODA_GETOPT`
  - `CODA_SETOPT`
  - `CODA_OPEN`
  - `CODA_OPEN_AS`
  - `CODA_CLOSE`
  - `CODA_PRODUCT_CLASS`
  - `CODA_PRODUCT_TYPE`
  - `CODA_PRODUCT_VERSION`
  - `CODA_FETCH`
    - The first argument
    - Specification arguments
    - Return type
      - Fetching simple (non-composite) types
      - Fetching records
      - Fetching arrays
  - `CODA_FETCH_DATAHANDLE`
  - `CODA_EVAL`
  - `CODA_ATTRIBUTES`
  - `CODA_FIELDAVAILABLE`
  - `CODA_FIELDCOUNT`
  - `CODA_FIELDNAMES`
  - `CODA_SIZE`
  - `CODA_DESCRIPTION`
  - `CODA_UNIT`
  - `CODA_TIME_TO_STRING`
  - `CODA_IS_NO_DATA`
  - `CODA_IS_ERROR`
  - `CODA_UNLOAD`

## CODA Definition Path

Note that in order to access products whose formats are defined using .codadef files, you should let CODA now where these .codadef files are stored. By default the CODA IDL interface will look for .codadef files in a directory relative to the location of the CODA IDL DLM file (`../../../share/coda/definitions`; note that software that embed CODA may sometimes override this default location). However, this will only work if you have set the IDL_DLM_PATH environment variable as specified in the CODA installation instructions for the IDL interface (otherwise no default location will be set).

You can override the default location by setting the CODA_DEFINITION environment variable. This environment variable should be a ':' separated (';' on Windows) list of absolute paths to directories containing .codadef files or absolute paths to .codadef files themselves (or a mix of those).

## Named Structures

The CODA IDL interface defines three 'named structures'. This section discusses them in detail. For reference, the three types are shown in the table below.

name | fields | type | usage
--- | --- | --- | ---
`CODA_DATAHANDLE` | `PF_ID` `OPAQUE` | `ULONG64` not user-accessible | represents a data-item in a product-file
`CODA_ERROR` | `ERRNO` `MESSAGE` | `INT` `STRING` | CODA call return status (can also indicate success)
`CODA_NO_DATA` | `OPAQUE` | `BYTE` | represents an empty data item; the `OPAQUE` field is a dummy value and should not be used

### The `CODA_DATAHANDLE` named structure

The `CODA_DATAHANDLE` type represents a piece of data within a currently opened product-file; internally, it maintains a pointer to a certain offset within the data product, as well as information on the type of expression residing at that offset.

`CODA_DATAHANDLE` variables are used extensively 'under the hood' of the CODA IDL interface; however, there are two closely related cases where this type will also be visible at the IDL user level.

If a call to `CODA_FETCH` is made that should return an 'array of array' or 'array of record' type, the call will return an 'array of `CODA_DATAHANDLE`' instead.

The reason for this behavior is that the sophisticated type system supported by CODA cannot be mapped in full generality onto the IDL type system from within IDL DLMs. Within a DLM it is not posible to create IDL arrays where different elements have different storage sizes (and DLMs do not have public access to the 'pointer interface' that can be used from within IDL code). Due to the generality of the CODA interface it is currently also not possible to make an exception for the possible cases where the records/arrays might per-chance have the same storage size for each array element. For consistency, we have therefore chosen to handle all these cases uniformly, always returning an 'array of `CODA_DATAHANDLE`' when an 'array of array' or an 'array of record' is fetched. You can then manually traverse the 'array of `CODA_DATAHANDLE`', and use `CODA_FETCH` (providing a `CODA_DATAHANDLE` as first argument) to obtain the individual records or arrays represented by the `CODA_DATAHANDLE` elements.

Although the proper use of `CODA_DATAHANDLE`-arrays is probably the most subtle and difficult aspect of using the CODA IDL interface, it has proved to be quite easy to get used to. In fact, when used properly, the use of an 'array of `CODA_DATAHANDLE`' often improves performance as the `CODA_DATAHANDLE` elements effectively cache the information needed to access the data, thereby speeding up subsequent accesses.

The `CODA_DATAHANDLE` structure has two fields:

- The `PF_ID` is a product-file ID as returned by the `CODA_OPEN` function (see below). At the IDL user level, you are free to inspect its value. However, do not change its value.
- The `OPAQUE` field is used internally to maintain pointer and data type information. This field should not be accessed from within IDL; most importantly, its value should not be changed.

The meaning of a `CODA_DATAHANDLE` becomes undefined after the product to which it is associated is closed; You should not use it in subsequent CODA IDL interface operations. Doing so will generate an error.

### The `CODA_ERROR` named structure

The `CODA_ERROR` structure may be returned by any CODA IDL interface function, usually indicating that the operation failed for some reason. Despite the name, you should think of a `CODA_ERROR` as a 'status report' of the last CODA IDL interface call; the status may, in fact, indicate successful completion.

For example, the `CODA_CLOSE` function will always return a value of type `CODA_ERROR`, even when the operation succeeds.

The `CODA_ERROR` has two fields. The `ERRNO` field gives a numerical error code. This code is guaranteed to be either 0 (zero), indicating success, or a negative number, indicating failure.

The `MESSAGE` field is a STRING that provides a human-readable description corresponding to the `ERRNO` field. Some errors originate from the underlying C library that the CODA IDL interface uses, others are specific to IDL. For example, if one of the parameters you pass to a CODA IDL interface function is incorrect you will get a `CODA_ERROR` indicating the type of error.

### The `CODA_NO_DATA` named structure

The `CODA_NO_DATA` is used to represent unavailable record fields or zero-element arrays in IDL. CODA is capable of representing arrays that have zero indices in any dimension but such a data structure cannot be represented in IDL. Whenever CODA needs to represent a zero-element array, an instance of type `CODA_NO_DATA` is used instead.

The `CODA_NO_DATA` has one field. The `OPAQUE` field is present only to assure that the structure has a field (which is required by IDL). It does not contain any useful data.

## Functions and Procedures

### Function `CODA_VERSION(): STRING`

The `CODA_VERSION` function returns a string containing the current version number of CODA. The version number is always of the format 'x.y.z', i.e., major, minor, and revision numbers, separated by dots.

### Function `CODA_GETOPT(STRING option_name): INT`

The `CODA_GETOPT` function returns the numerical value of a CODA IDL interface option. The following options are supported:

option name | possible values | default value | meaning
--- | --- | --- | ---
FilterRecordFields | 0 or 1 | 1 | if set, the CODA IDL interface will skip spare fields and fields that have a fixed value (such as header labels) when retrieving records, avoiding unnecessary clutter.
PerformConversions | 0 or 1 | 1 | if set, CODA will perform unit/value conversions for fields that have a 'conversion' defined in the CODA product format definitions.
PerformBoundaryChecks | 0 or 1 | 1 | if set, CODA will perform boundary checking on array accesses. This makes CODA more robust, at the cost of some performance. It is recommended to disable boundary checks only for thoroughly tested programs; disabling boundary checks can lead to wrong results and/or IDL crashes if arrays are accessed beyond their boundaries.
SwapDimensions | 0 or 1 | 1 | if set, CODA will swap the dimensions of multi-dimensional arrays (i.e. performing a multi-dimensional transpose on the data) so the array dimensions that are found in e.g. the CODA format definitions can be used as-is on the data. This distinction is needed because IDL uses Fortran-style ordering for array indices whereas CODA uses C-style array dimensioning ordering as the standard. If you disable this option, CODA will no longer transpose the data itself, but will invert the ordering of the array indices you pass to e.g. CODA_FETCH and the dimension sizes that are returned by CODA_SIZE.
TimeUnitDays | 0 or 1 | 0 | if set, the CODA IDL interface will represent all time values as days since 1-1-2000 instead of seconds since 1-1-2000. This is true both for fetched values and for values passed to `CODA_TIME_TO_STRING`. Use of this functionality is discouraged, as this will make data exchange between programs more difficult.
UseMMap | 0 or 1 | 1 | By default CODA uses a technique called 'memory mapping' to open and access data from product files. Using mmap greatly outperforms the default approach of reading data using the open()/read() combination. The downside of mapping a file into memory is that it takes away valuable address space. When you run a 32-bit Operating System your maximum addressable memory range is 4GB and if you simultaneously try to keep a few large product files open your memory space can quickly become full. Opening additional files will then produce 'out of memory' errors. Note that this 'out of memory' situation has nothing to do with the amount of RAM you have installed in your computer. It is only related to the size of a memory pointer on your system, which is limited to 4GB. If you are using CODA in a situation where you need to have multiple large product files open at the same time you can turn of the use of memory mapping by disabling this option. If you change the memory mapping option, the new setting will only be applicable for files that will be opened after you changed the option. Any files that were already open will keep using the mechanism with which they were opened.
UseSpecialTypes | 0 or 1 | 1 | if you disable this option, CODA will use the base type of a special type (and not the special type itself) when reading data or retrieving information about a data item. See the CODA Product Format Definitions documentation for more information about special types.
Verbose | 0 or 1 | 1 | if set, the CODA IDL interface will echo 'error' return values to the IDL command line just before returning them to the IDL user level. This can be helpful in detecting problems.

### Function `CODA_SETOPT(STRING option_name, INT new_value): INT`

The `CODA_SETOPT` function is used to set the CODA IDL interface option to a new value. See the `CODA_GETOPT` description above for a list of allowed options and values, and their associated meanings.

The `CODA_SETOPT` function will return the previous value of the option.

### Function `CODA_OPEN(STRING filename): ULONG64`

`CODA_OPEN` will attempt to open the file (which may fail, for example, if you specify a file that does not exist).

If `CODA_OPEN` succeeds, it returns a unique positive 64-bit unsigned integer that can subsequently be used to identifty the file in `CODA_FETCH` and `CODA_CLOSE` calls. If it fails a `CODA_ERROR` structure is returned giving the reason for failure.

It is perfectly OK to have multiple files open at the same time. Still, you are advised to close files as soon as you are done with them, since each of them consumes a vital system resource: address space. Note that this is not the same as 'memory': a data product is not read in its entirety into memory. A typical opened product file should consume a couple of tens of kilobytes of memory. However, the entire product file is virtually mapped into memory (a well-known technique that provides high-performance I/O access to a file). On a modern 32-bit computer (e.g., an Intel x86-class computer running Linux or Windows), up to 3 gigabytes worth of files may be mapped into memory at any one point in time.

In the CODA IDL interface, you can have up to 100 files open simultaneously. An attempt to open more files will yield an error.

### Function `CODA_OPEN_AS(STRING filename, STRING product_class, STRING product_type, INT version): ULONG64`

`CODA_OPEN_AS` will try to open the specified file for reading similar to `CODA_OPEN`, but instead of trying to automatically recognise the applicable product class/type/version as `CODA_OPEN` does, this function will impose the format definition that is associated with the given product_class, product_type, and version parameters.

Note that you normally won't need this function as CODA will be able to recognize which format definition to use automatically. However, for the rare occasions where `CODA_OPEN` is not sufficient, you can use this function to force the use of a specific format definition.

You can specify -1 for the version to request the latest available version of the format definition.

### Function `CODA_CLOSE(ULONG64 pf_id): CODA_ERROR`

The `CODA_CLOSE` function is used to release a previously-opened product file. Its one argument should be a 64-bit unsigned integer as returned by a previously executed successful `CODA_OPEN` call.

### Function `CODA_PRODUCT_CLASS(ULONG64 pf_id): STRING`

The `CODA_PRODUCT_CLASS` function returns a string containing the product class of a product file if it has one (otherwise an empty string is returned).

### Function `CODA_PRODUCT_TYPE(ULONG64 pf_id): STRING`

The `CODA_PRODUCT_TYPE` function returns a string containing the product type of a product file if it has one (otherwise an empty string is returned).

### Function `CODA_PRODUCT_VERSION(ULONG64 pf_id): INT`

The `CODA_PRODUCT_VERSION` function returns an integer denoting the product format version. If the product does not have a format version number this functiion will return -1.

### Function `CODA_FETCH(...)`

The `CODA_FETCH` function is used to fetch data from somewhere in an open product-file and turn it into an IDL variable.

#### `CODA_FETCH`: the first argument

Several forms of `CODA_FETCH` exist that are distinguished by the first argument argument. Depending on the form of the call, `CODA_FETCH` will try to read data from different header-parts or datasets in the product files. This is summarized below.

call | Fetch-root
--- | ---
`CODA_FETCH(pf_id)` | Entire product
`CODA_FETCH(pf_id, [, what, ...])` | Data (from product root)
`CODA_FETCH(data_handle [, what, ...])` | Data described by handle

As noted before, in some cases the `CODA_FETCH` call may return an array of `CODA_DATAHANDLE` structures. The last form of the `CODA_FETCH` call allows you to traverse such an array and fetch the data described by each of the data-handles in turn.

#### `CODA_FETCH`: data-item specification arguments

After the first argument you can specify more arguments that tell `CODA_FETCH` to retrieve only a specific part of the data pointed to by the first argument. It is possible to have more than one such 'specification argument' in case of nested data-types.

Three forms of 'specification arguments' exist:

- Array index: You need to provide an array index that has the same number of elements as there are dimensions in the array that you are referring to. So if the array is two dimensional you have to pass indices for both dimensions like `[4,5]`. If the array is one dimensional you can just provide a single index value (without the '[]') to get to the k-th element. You can use `coda_size` to check the number of dimensions of an array and the size of each dimension.
- Field name: To go to a certain field inside the record that you are pointing to, you can provide the field name as a string.
- Path: You can provide a string containing a path reference, such as `"measurements[0]/time"`. Using paths will also allow you to navigate to attributes by using `"@"` as a path component. For instance, '`"temperature", "@", "units"`' or `"temperature@units"` will point to the units attribute of the temperature data. Note that array indices used in these string paths need to be 0-based indices on the flattened view of an array. This means that if an array is defined as having more than one dimension then the index as used in a path expression should be between 0 and the total number of array elements (exclusive). For example, for a `[10,8]` array, the index should be >= 0 and <= 79. For, instance `coda_fetch(pf, 'variable', [0,0])` is equivalent to `coda_fetch(pf, 'variable[0]')`

An example will help to understand all this. Suppose we have a product that contains a dataset called 'MEASUREMENTS', having 1000 records. Each of these records contains a field called 'samples' that is a 2-dimensional 101x101 array of doubles.

Now suppose we want to fetch the central 'sample' contained in record #10. The following `CODA_FETCH` would do the trick:

`result = CODA_FETCH(pf_id, 'MEASUREMENTS', [10], 'samples', [50,50])`

Alternatively, since the '[10]' specifies the index of a one-dimensional array, we could also write

`result = CODA_FETCH(pf, 'MEASUREMENTS', 10, 'samples', [50,50])`

Or we could use a single string parameter to provide the full path

`result = CODA_FETCH(pf, 'MEASUREMENTS[10]/samples[5100]')`

For the `CODA_FETCH` and `CODA_FETCH_DATAHANDLE` functions (but not for the other CODA IDL interface functions that use data-item specification arguments) there is an additional feature. If you provide a `-1` for one or more of the dimensions of an array you will fetch all elements in this dimension. For example, with

`result = CODA_FETCH(pf, 'MEASUREMENTS', -1, 'samples', [50,50])`

you can fetch all central samples of all measurements into a single array. Similarly, you can use

`result = CODA_FETCH(pf, 'MEASUREMENTS', 0, 'samples', [-1,0])`

to fetch the first samples column of the first measurement.

The `-1` parameter may only be used for one array in a fetch statement (e.g. `CODA_FETCH(pf, 'MEASUREMENTS', 0, 'samples', [-1,-1])` is allowed, but `CODA_FETCH(pf, 'MEASUREMENTS', -1, 'samples', [-1,0])` is not). Also, the -1 index only works when passing it as an explicit argument (i.e. calling `coda_fetch(pf, "dataset[-1]/dsr_time[5100]")`, where the -1 index is part of a string argument, will not work; calling `coda_fetch(pf, "dataset", -1, "dsr_time[5100]")` will work).

#### `CODA_FETCH` return type

The IDL type of the `CODA_FETCH` return-value depends on the CODA data type of the data item being fetched. Three cases are distinguished:

##### Return types for simple (non-composite) types

If the data item has a basic type the return type is determined according to the following table.

CODA read type | IDL Type | remark
--- | --- | ---
int8 | INT |
uint8 | BYTE | a BYTE is unsigned in IDL
int16 | INT |
uint16 | UINT |
int32 | LONG |
uint32 | ULONG |
int64 | LONG64 |
uint64 | ULONG64 |
float | FLOAT |
double | DOUBLE |
char | STRING |
string | STRING |
bytes | BYTE | This will actually translate into an array of type BYTE

`CODA_FETCH` return types for basic data types

##### Return types for Record types

Record types will be returned as anonymous (unnamed) IDL structures, where each field will correspond to a field in the original data set. If a field has the hidden property and if the FilterRecordFields option is set then the field will not be copied into the IDL structure.

For basic types, the IDL field type will be determined according to the 'basic data types' table given above. Fields that are themselves records are expanded as 'nested records'. Fields that are themselves arrays are expanded as 'array types', as discussed below.

If a record contains no fields (i.e. an empty record) then CODA will return a `CODA_NO_DATA` named structure instead of an empty anonymous struct.

##### Return types for Array types

Array types will be returned as multi-dimensional IDL arrays, with the IDL array base-type derived from the base-type of the CODA type according to the 'basic data types' table given above. There are, however, some complications that IDL programmers need to be aware of when the base type itself is either an array or a record.

For array and record base-types (i.e., 'array of array', 'array of record'), there is, in general, no equivalent type in IDL. As explained in the section about the `CODA_DATAHANDLE` structure above, both these cases will yield an 'array of `CODA_DATAHANDLE`' instead.

If an array is empty (i.e. the size of one of its dimensions is 0) then CODA will return a `CODA_NO_DATA` named structure instead of a multi-dimensional IDL array.

### Function `CODA_FETCH_DATAHANDLE(...): CODA_DATAHANDLE`

This function takes the same kind of argument as `CODA_FETCH`; however, instead of returning the specified data from the product, it always returns a datahandle to the specified data (even if it would be easily representable in IDL).

### Function `CODA_EVAL(STRING expression, ...)`

This function takes a CODA expression in the form of a string and an optional product location. The product location is the same kind of argument (list) as used for `CODA_FETCH`. If the CODA expression can be evaluated statically, then the product location argument is not required, otherwise the product location argument is mandatory.

The syntax for the CODA expression language can be found in the CODA Expressions documentation. Note that evaluation of 'void' expressions is not supported.

### Function `CODA_ATTRIBUTES(...)`

This function takes the same kind of argument as `CODA_FETCH`; however, instead of returning the specified data from the product, it returns a record containing all attributes of the specified data item. The record can be empty if no attributes are available.

### Function `CODA_FIELDAVAILABLE(...): INT`

This function takes the same kind of argument as `CODA_FETCH`; however, instead of returning the specified data from the product, it returns the availabillity of a record field. It returns 1 if it is available and 0 if it is not.

If the arguments to `CODA_FIELDAVAILABLE` do not point to a record, a `CODA_ERROR` structure is returned.

### Function `CODA_FIELDCOUNT(...): INT`

This function takes the same kind of argument as `CODA_FETCH`; however, instead of returning the specified data from the product, it returns the number of fields in a record.

If the arguments to `CODA_FIELDCOUNT` do not point to a record, a `CODA_ERROR` structure is returned.

### Function `CODA_FIELDNAMES(...): STRING-array`

This function takes the same kind of argument as `CODA_FETCH`; however, instead of returning the specified data from the product, it returns the field-names of the fields in a record.

If the arguments to `CODA_FIELDNAMES` do not point to a record, a `CODA_ERROR` structure is returned.

### Function `CODA_SIZE(...): INT-array`

This function takes the same kind of argument as `CODA_FETCH`; however, instead of returning the specified data from the product, it returns the actual dimensionality of a multi-dimensional array.

If the arguments to `CODA_SIZE` do not point to an array, a `CODA_ERROR` structure is returned.

### Function `CODA_DESCRIPTION(...): STRING`

This function takes the same kind of argument as `CODA_FETCH`; however, instead of returning the specified data from the product, it returns the description of a data item.

### Function `CODA_UNIT(...): STRING`

This function takes the same kind of argument as `CODA_FETCH`; however, instead of returning the specified data from the product, it returns the unit of a data item.

### Function `CODA_TIME_TO_STRING(numerical argument): STRING`

CODA by default stores time values as a double denoting the number of seconds since Midnight, January 1st, 2000 UTC.

The `CODA_TIME_TO_STRING` function takes a `DOUBLE`-encoded time value and converts it to a human-readable string of the form "2000-01-01 00:00:00.000000"

### Function `CODA_IS_NO_DATA(...): INT`

The `CODA_IS_NO_DATA` function takes one argument; it returns a value of TRUE (1) if and only if the argument is a `CODA_EMPTY_ARRAY` structure.

- its `ERRNO` field is not equal to zero

### Function `CODA_IS_ERROR(...): INT`

The `CODA_IS_ERROR` function takes one argument; it returns a value of TRUE (1) if and only if

- the argument is a `CODA_ERROR` structure
- its `ERRNO` field is not equal to zero

If either of these is not true, `CODA_IS_ERROR` returns FALSE (0).

Using `CODA_IS_ERROR`, you can make your CODA-based IDL programs robust (i.e., resistant to failures). If you test the result of any CODA IDLinterface call with `CODA_IS_ERROR`, you should be able to intercept any error.

### Procedure `CODA_UNLOAD`

The `CODA_UNLOAD` procedure will close all opened product files and unload any chached CODA product format definitions from memory.

The CODA product format definitions are loaded on-demand if any CODA function that needs it is invoked.

This function may be (slightly) useful on systems with little memory. You could open a product file, fetch the important data, close the file, and then unload the product format definitions to free extra memory before starting data-processing. Other than that, this function is of little practical use.