The eXtremeDB Feed Handler API Overview

The Feed Handler Application Programming Interface allows the users to create their own extension modules that handle market data feeds not currently supported by the eXtremeDB Feed Handler.

The modules are implemented as shared libraries on Linux and macOS, and as DLLs on Windows. They should be linked with the Feed Handler library, libfh, as well as any third-party libraries and frameworks required to handle specific feeds.

The libfh library exposes a number of C functions that facilitate module configuration, database schema definition, data storage, etc. These functions are referred to as the “Feed Handler API” (or simply API for brevity).

Furthermore, each module is expected to export a certain predefined set of C functions. The Feed Handler application will call these functions to manage the module. These functions are referred to as the “Feed Handler Module API” (or ModAPI).

The modules can be implemented in C or C++, or in any other language of choice, as long as it allows for usage of C APIs, and provides means to create shared libraries or DLLs.

(See page Feed Handler C API for detailed description of the individual functions.)

Feed Handler Module API Usage

Module Lifetime Overview

Upon launch, the Feed Handler application loads and parses its configuration file. The configuration file lists handler modules that do the actual job of receiving and handling data, and their configurations. For each module, the Feed Handler application loads its shared library. It uses the name of the module (as specified in the “name” field) as the base name of the module library. For example, if the name is “testfeed”, the Feed Handler will look for the libtestfeed[_debug].so library on Linux/macOS and testfeed[_debug].dll on Windows. The debug versions of the libraries are loaded by the debug version of the Feed Handler executable. Standard shared library search paths are used for each platform.

After loading the module library, the Feed Handler application searches it for the ModAPI functions and checks the module’s version (see the mco_fh_mod_api_version() function description). If the check passes, it calls the module’s mco_fh_mod_create() method.

Next, the application initializes the module. The configuration file defines the following entities:

• Data sources: service endpoints providing market data of interest; the exact semantics of this field depend on the module and are described in its documentation.
• Subscriptions: market data symbol names and the names of the data sources that provide them.

The application uses the mco_fh_mod_add_data_source() and mco_fh_mod_add_subscription() ModAPI functions to pass these entities to the module.

Then the application calls the mco_fh_mod_init() ModAPI function. The module is expected to connect to the data sources, load dictionaries, metadata, etc. that will be needed for the next steps.

The Feed Handler application does not have any knowledge about the modules’ data schema. Thus, each module is required to supply the metadata that describes the types of its records and their fields. There are two ways to implement this:

1. The application calls mco_fh_mod_fill_metadata() function. If the module knows all names and types of its data fields, it can pass them to the Feed Handler using the mco_fh_metadata_add() function.

2. Alternatively, if the module cannot supply all possible fields for all supported tables at once (e.g. there are too many fields and it is not known beforehand which fields will appear in which tables), the mco_fh_mod_fill_metadata() function should return MCO_FH_E_NOT_IMPLEMENTED. In this case, the Feed Handler application will call the module’s mco_fh_mod_get_field_info() function to retrieve the types of fields configured by the user. The module responds with the field type specification (type, size and precision).

When the metadata is specified, the Feed Handler application creates the tables in the database using the table mappings provided by the user and the modules’ metadata. Then, it tells the module which tables and fields to write via the mco_fh_mod_add_table() function. The module should store the table name and its fields (and the fields’ ordering). The table name will be used to instantiate database writers, and the field information will be used to fill out database records.

When all the tables are passed to the module, the module execution is started. The application calls its mco_fh_mod_start() function, and the module is expected to begin execution in a separate thread and return control to the application immediately.

To write data to the database, the module creates a “writer” object and a “record” object using the mco_fh_writer_create() and mco_fh_record_create() API functions. The module should not create these objects anew for each incoming data record; instead, it is expected to retain them. It is only allowed to create one writer object per table. These objects should only be accessed by the thread that created them.

The module fills out each record with the retrieved market data prior to writing it to the database. It uses the mco_fh_record_set_field_*() family of functions to set a record’s field values. These functions accept the record object’s handle, the index of the field, and the value of the field.

The module continues to retrieve and store market data until the application calls its mco_fh_mod_stop() ModAPI function. When this function is called, the module is expected to stop all feeds, disconnect from the servers, join threads, and return control to the application.

API Usage Samples and Discussion

Some of the API usage scenarios are given below. Please note that the samples are intentionally simplified for clarity and may omit error handling, or have suboptimal performance.

Module Initialization

When the module’s mco_fh_mod_create() ModAPI function is called, it is passed two parameters: custom module configuration (from the “config” section of the module configuration), and a pointer to the module’s handle. The module sets this handle’s value. The Feed Handler application does not interpret it in any way: it is passed unmodified to the ModAPI functions. The module is free to use it to store custom context data.

The module uses the mco_fh_config_*() family of functions to read configuration data. For example:

 
    mco_fh_ret mco_fh_mod_create(mco_fh_config_node_h cfg, mco_fh_module_h *h)
    {
        Module *module = new Module();
        // get the "debug" flag value
        mco_fh_config_node debug_node = mco_fh_config_node_get_child(cfg, "debug");
        if (mco_fh_config_node_get_bool(debug_node, MCO_NO) == MCO_YES) 
        {
            module->enableDebugMode();
        }
 
        *h = module;
        return MCO_FH_OK;
    }
     

Next, after setting data sources and subscriptions, the Feed Handler application calls the module’s mco_fh_mod_init() ModAPI function and passes it the module handle and the database handle. For example:

 
    mco_fh_ret mco_fh_mod_init(mco_fh_module_h h, mco_fh_db_h db)
    {
        Module *module = (Module *)h;
        module->setDatabase(db);
        module->connectToService();
        return MCO_FH_OK;
    }
     

Metadata Definition

As was noted above, the Feed Handler application doesn’t have any knowledge of the module’s data structures. The feed handled by the module may provide Quotes, Trades, Level 2 data, all of the above, or something entirely different. All of these data structures’ contents (number of fields, their types and names) may differ from feed to feed. For example, a Quote record usually contains symbol name, bid and ask values, but may also include auxiliary data, like timestamps, exchange IDs, etc.

In the discussion below we will suppose that the feed provides only quotes, which are described by the following C structure:

 
    struct FeedQuote
    {
        char symbol[4];
        int exchangeCode;
        float bid;
        float ask;
        bool isNBBO;
    };
     

It was mentioned that the module can choose one of two approaches to metadata registration. With the first one, the module registers all records and their fields at once, when its mco_fh_mod_fill_metadata() ModAPI function is called. With the second one, the Feed Handler application reads table mappings from the configuration file and requests the module to provide information for each record and field it encounters.

Below are examples of both approaches.

Providing Metadata with mco_fh_mod_fill_metadata()

If the module knows all of the possible fields of all its records, it can opt to specify them at once inside the mco_fh_mod_fill_metadata() ModAPI function:

 
    mco_fh_ret mco_fh_mod_fill_metadata(mco_fh_module_h h)
    {
        mco_fh_metadata_add(db, “Quote”, “symbol”, McoSql::tpString, 4, 0);
        mco_fh_metadata_add(db, “Quote”, “exchangeCode”, McoSql::tpInt4, 0, 0);
        mco_fh_metadata_add(db, “Quote”, “bid”, McoSql::tpReal4, 0, 0);
        mco_fh_metadata_add(db, “Quote”, “ask”, McoSql::tpReal4, 0, 0);
        mco_fh_metadata_add(db, “Quote”, “isNBBO”, McoSql::tpBool, 0, 0);
        return MCO_FH_OK;
    }
     

Note that the mco_fh_metadata_add() API function should only be called from mco_fh_mod_fill_metadata().

Providing Metadata with mco_fh_mod_get_field_info()

If the module opts not to specify its metadata in mco_fh_mod_fill_metadata() function, it should return MCO_FH_E_NOT_IMPLEMENTED. In this case, the Feed Handler will call its mco_fh_mod_get_field_info() function instead. This method will be called for each record type and field that the application encounters while parsing the configuration file. The module is required to return type information for the requested fields:

 
    mco_fh_ret mco_fh_mod_get_field_info(mco_fh_module_h h, const char *rec_type, 
                            const char *field_name, mco_int4 *type, 
                            mco_size_t *size, mco_size_t *precision)
    {
        // only "Quote" type is supported
        if (std::string(rec_type) != "Quote") 
        {
            return MCO_FH_E_INVALID_RECORD_TYPE;
        }
         
        if (std::string(field_name) == "symbol") 
        {
            *type = McoSql::tpString;
            *size = 4;
            *precision = 0;
        } else if (std::string(field_name) == "exchangeCode") 
        {
            *type = McoSql::tpInt4;
            *size = 0;
            *precision = 0;
        } else if (std::string(field_name) == "bid") 
        {
            *type = McoSql::tpReal4;
            *size = 0;
            *precision = 0;
        } else if (std::string(field_name) == "ask") 
        {
            *type = McoSql::tpReal4;
            *size = 0;
            *precision = 0;
        } else if (std::string(field_name) == "isNBBO") 
        {
            *type = McoSql::tpBool;
            *size = 0;
            *precision = 0;
        } else 
        {
            return MCO_FH_E_INVALID_FIELD_NAME;
        }
         
        return MCO_FH_OK;
    }
     

For example, the user might specify the following table mapping (note that the isNBBO field is omitted):

 
    {
        "type": "Quote",
        "name": "MyQuotes",
        "fields":
        [
            { "source": "symbol", "target": "sym" },
            { "source": "bid", "target": "bid" },
            { "source": "ask", "target": "ask" },
            { "source": "exchangeCode", "target": "exchCode" }
        ],
    }
     

In this case, the Feed Handler application will issue a sequence of calls to the module’s mco_fh_mod_get_field_info() function like this:

 
    mco_int4 type;
    mco_size_t size, precision;
    // ...
    mco_fh_mod_get_field_info(modHandle, "Quote", "symbol", &type, &size, &precision);
    // ...
    mco_fh_mod_get_field_info(modHandle, "Quote", "bid", &type, &size, &precision);
    // ...
    mco_fh_mod_get_field_info(modHandle, "Quote", "ask", &type, &size, &precision);
    // ...
    mco_fh_mod_get_field_info(modHandle, "Quote", "exchangeCode", &type, &size, &precision);
     

mco_fh_mod_fill_metadata() vs mco_fh_mod_get_field_info()

Both approaches yield the same result. However, the first approach has an added benefit of allowing the user to specify empty table mappings. For example:

 
    {
        "type": "Quote",
        "name": "MyQuotes",
        "fields": [],
    }
     

In this case, the “MyQuotes” table will contain “symbol”, “exchangeCode”, “bid”, “ask” and “isNBBO” fields. The fields will appear in the same order as they were specified by the mco_fh_mod_fill_metadata() function.

Error Handling and mco_fh_log()

Currently, the Feed Handler application does not try to interpret the ModAPI return codes, because if the module fails to initialize or start (and most of these errors are returned at the initialization phase), then there is no point in trying to continue execution. The application writes a log message and shuts down. The module is expected to write descriptive log messages, too, using the mco_fh_log() API. The API reference guide pages specify error codes that the ModAPI implementations should return in certain circumstances; however, they are not limited to these codes and may use others if needed.

Handling Database Table Descriptions

After the metadata is successfully registered, the Feed Handler application tells the module which database tables and fields to write; the user may omit some fields in the configuration, so only the requested ones should be written. Furthermore, the field order is configurable as well. The module should store the table names and field positions. The module may assign integer codes to fields, or use any other suitable approach, as required by its logic. However, for the sake of brevity, the example below stores field names as strings, which might not be the optimal approach from the performance standpoint:

 
    struct TableInfo
    {
        std::string name;
        std::vector<std::string> fields;
        // these two fields will be used in further examples:
        mco_fh_writer_h writer;
        mco_fh_record_h record;
    };
    std::vector<TableInfo> tables;
    // ...
    mco_fh_ret mco_fh_mod_add_table(mco_fh_module_h h, const char *rec_type, 
                        const char *name, const char **fields, 
                        mco_size_t nfields)
    {
        if (std::string(rec_type) != "Quote") 
        {
            return MCO_FH_E_INVALID_RECORD_TYPE;
        }
         
        TableInfo table;
        table.name = std::string(name);
        for (mco_size_t i = 0; i < nfields; i++) 
        {
            table.fields.push_back(fields[i]);
        }
        tables.push_back(table);
        return MCO_FH_OK;
    }
     

Writing Data to the Database

When all tables are passed to the module, the Feed Handler application calls the module’s mco_fh_mod_start() ModAPI function. The module should launch its worker threads and return control immediately.

For each database table that the module needs to write, it should create a writer object and one or more record objects. Only one writer per table is allowed. The number of record objects is not limited; however, the module is expected to reuse the records instead of allocating them anew for each incoming feed data entity. Generally, it does not make sense to allocate more than one record object per table. Writer and record objects should only be used from the thread where they were created.

The module knows which tables to write from the previous step. It may now create the writer and record objects:

 
    std::vector<TableInfo>::iterator it;
    for (it = tables.begin(); it != tables.end(); ++it) 
    {
        it->writer = mco_fh_writer_create(db, it->name.c_str());
        it->record = mco_fh_record_create(writer);
    }
     

To store a record in the database, the module first fills its fields with actual data retrieved from the feed. It uses the previously stored field information to decide which fields to write:

 
    // retrieve next quote from the feed
    FeedQuote *quote = feed->nextQuote();
    std::vector<TableInfo>::iterator it;
    for (it = tables.begin(); it != tables.end(); ++it) 
    {
        // get handles of the previously created writer and record
        mco_fh_writer_t writer = it->writer;
        mco_fh_record_t record = it->record;
        // optionally, reset the record's contents
        mco_fh_record_zero_out(record);
        // fill only the required fields of the record
        for (size_t fieldIndex = 0; fieldIndex < it->fields.size(); fieldIndex++) 
        {
            if (it->fields[fieldIndex] == "symbol") 
            {
                mco_fh_record_set_field_string(record, fieldIndex, quote->symbol);
            } else if (it->fields[fieldIndex] == "exchangeCode") {
                mco_fh_record_set_field_int4(record, fieldIndex, quote->exchangeCode);
            } else if (it->fields[fieldIndex] == "bid") {
                mco_fh_record_set_field_float(record, fieldIndex, quote->bid);
            } else if (it->fields[fieldIndex] == "ask") {
                mco_fh_record_set_field_float(record, fieldIndex, quote->ask);
            } else if (it->fields[fieldIndex] == "isNBBO") {
                mco_fh_record_set_field_bool(record, fieldIndex, quote->isNBBO);
            }
        }
        // write the record to the database
        mco_fh_writer_store_record(writer, record);
    }
     

Module Shutdown

The module may request application shutdown in abnormal circumstances (e.g. in case of an unrecoverable error) using the mco_fh_app_stop()API. Note, however, that this function should only be called after the module’s execution has been started (with mco_fh_mod_start()) and before it was required to stop (with mco_fh_mod_stop()).

When the Feed Handler application needs to shut down (either due to the user’s request, or the mco_fh_app_stop() call), it calls the module’s mco_fh_mod_stop() ModAPI function. The module should stop all its threads and wait for them to be joined. Please note that the writer and record objects should only be destroyed from the threads that created them:

 
    mco_fh_record_destroy(record);
    mco_fh_writer_destroy(writer);
     

Finally, the Feed Handler application calls the module’s mco_fh_mod_destroy() ModAPI function. The module should release all allocated resources:

 
    void mco_fh_mod_destroy(mco_fh_module_h h)
    {
        Module *module = (Module *)h;
        delete module;
    }
     

 

Feed Handler API Reference

See page Feed Handler C API for detailed description of the individual functions.