Appendix A: Using the eXtremeDB Feed Handler

This article is deprecated.

Introduction

In a general sense, feed handlers are a broad class of applications and frameworks that parse, process, aggregate, and/or store market data. Different kinds of market data feeds may supply different kinds of data, such as quote or trade data, orders and order books, yield curves, etc. Some feeds originate directly at the exchanges (NYSE, LSE, …), others are aggregated from multiple sources by third-party institutions and redistributed using a normalized representation.

The eXtremeDB Feed Handler receives market data using industry standard frameworks and stores it in an eXtremeDB database.

Market instrument subscriptions are configurable on a feed-by-feed basis. It is possible to subscribe only to the relevant instruments.

The database schema is flexible and configurable, thus allowing users to store only the data relevant to their needs. The data can be accessed and manipulated using ordinary SQL expressions with xSQL. The user is free to choose a persistent or an in-memory database depending on performance and data persistence requirements.

Currently, the eXtremeDB Feed Handler is only available for 64-bit Linux, Mac OS and Windows systems.

The Feed Handler Application Programming Interface allows the users to create their own extension modules that handle market data feeds not currently supported by the eXtremeDB Feed Handler.

Application Structure

The Feed Handler application consists of the primary executable and loadable modules. The executable is responsible for the Feed Handler configuration file and command line parsing, logging, launching and controlling market data handler modules. The loadable modules implement shared functionality and market data handlers.

The market data handler modules are components that use third-party frameworks (SMDS, TR RFA) to subscribe to, receive and handle market data, and store it in the database. The eXtremeDB Feed Handler includes modules for Refinitiv (formerlyThomson Reuters) RFA and Vela SMDS frameworks, as well as a test module provided for demonstration of the Feed Handler Module API.

 

Supported Frameworks

Vela (formerly SRLabs) SMDS

From the vendor’s website: “Vela’s SMDS market data solution is a software-based feed handler that can be used in process or deployed as a stand-alone application.” SMDS supports more than 130 feeds and provides uniform access to market data originating from different vendors.

The full list of supported feeds is available in the SMDS User Guide.

Thomson Reuters Enterprise Platform

Thomson Reuters (TR) supplies a wide range of financial products. Its Enterprise Platform (TREP) facilitates implementation of services compatible with the TR infrastructure. The same Enterprise Platform underlies many of TR’s own products. TREP’s Open Message Model (OMM) is the common object model used by the platform’s applications, which enables interoperability between components potentially supplied by multiple vendors.

Data feed solutions from Refinitiv include:

TREP infrastructure can be accessed using a number of frameworks supplied by Refinitiv . The eXtremeDB Feed Handler uses the Robust Foundation API (RFA) and the Elektron Message API (EMA) C++ frameworks from Refinitiv (formerly Thomson Reuters).

 

Getting Started

The Feed Handler application is distributed as a part of the eXtremeDB Financial Edition package.

Quick Start Guide

The eXtremeDB Financial Edition package contains EMA redistributable libraries, EMA market data generator, and sample configuration files that make it possible to quickly get started with the Feed Handler application and explore its features. Below is a simple step-by-step guide to launching the application. For more detailed discussion of the application, its prerequisites, configuration and execution refer to the chapters below.

First of all, extract the eXtremeDB distribution archive. For brevity, file and directory paths below will be given relative to the eXtremeDB root directory, e.g.:

     
    /home/user/eXtremeDB
     

The Feed Handler demonstration setup includes the following key components:

Launching the Market Data Generator

The generator requires the RDM dictionary files (RDMFieldDictionary and enumtype.def) to run. These files can be obtained from the Elektron SDK distribution. Please place them in the Generator directory prior to running the application.

Change the current directory to the market data generator subdirectory and launch the generator:

     
    cd target/feedhandler/EMA_redistributable/Generator
    ./generator
            
     

On Windows, the generator is launched similarly:

 
    cd target\feedhandler\EMA_redistributable\Generator
    generator.exe
     

Upon a successful launch, the following message appears:

     
    OmmProvider ready
     

Launching the Feed Handler

When the Generator is ready, the Feed Handler application is ready to run. Open a second console window, change directories to the directory containing sample EMA scripts and start the Feed Handler:

     
    cd target/feedhandler/scripts/EMA
    ./feedhandler.sh -c feed_inmem.cfg
     

On Windows, use the provided feedhandler.bat wrapper script:

 
    cd target\feedhandler\scripts\EMA
    feedhandler.bat -c feed_inmem.cfg
     

The files feedhandler.sh and feedhandler.bat are wrapper scripts that set up necessary environment variables, prepare symbolic links for shared libraries and start the Feed Handler application. The script’s command line arguments are passed to the Feed Handler application. The configuration file feed_inmem.cfg contains settings for an in-memory database. There is also feed_disk.cfg, a configuration file in the same directory, which can be used to evaluate the Feed Handler with a persistent database.

When the application is run, you’ll see log output like the following:

     
    2017-01-18 01:55:32 INFO consumer.cpp:115: market data clients ready
     

 

Examining Database Content

The market data produced by the Generator application is collected and stored in the eXtremeDB database. It is recommended to use xSQL to connect to the database and execute SQL queries against it.

Open a console window, change directories to the eXtremeDB root directory and launch xSQL:

     
    ./target/bin/xsql @localhost:5001
     

When xSQL is run and connected to the database, you’ll see the XSQL> command prompt:

     
    xsql started
    Runtime configuration
    Remote node(s)       : localhost:5001
    Runtime              : Debug
    XSQL>
     

Now, you can use SQL queries to explore the database contents. For example:

Get the number of Level 1 entries in table MarketPrice:

     
    XSQL>select count(*) from MarketPrice;
    #1
    ------------------------------------------------
    1500
    Selected records: 1
     

Get the RICs present in table MarketPrice:

     
    XSQL>select distinct RIC from MarketPrice;
    RIC
    ------------------------------------------------
    ACN.N
    IBM.N
    TRI.N
     
    Selected records: 3
     

Read some records:

     
    XSQL>>select RIC, BID, ASK from MarketPrice where RIC='TRI.N' limit 10;
    RIC	BID	ASK
    ----------------------------------------------------------------------
    TRI.N	18	19    
    TRI.N	17	19
    TRI.N	21	22
    TRI.N	20	21
    TRI.N	25	28
    TRI.N	22	24
    TRI.N	20	21
    TRI.N	21	23
    TRI.N	17	19
    TRI.N	19	20
    
    Selected records: 10     
 

Explore the MarketByOrder domain summary data:

     
    XSQL>select RIC, CURRENCY, TRD_UNITS, PR_RNK_RUL, RDN_EXCHD2 from MarketByOrderSummary;
    
    RIC	CURRENCY	TRD_UNITS	PR_RNK_RUL	RDN_EXCHD2
    ------------------------------------------------
    TRI.ARC	USD	INT 	NOR	ASE
     
    Selected records: 1
     

 

Using Web Console to Examine Application and Database Status

You may use a web browser to examine application status, database schema and performance indicators. In order to do that, launch the browser and type the Feed Handler web console’s default address:

     
    http://localhost:8082
     

 

Shutting Down

To leave xSQL, type “exit” (or simply press Control-D). You’ll see the Linux command prompt again:

     
    XSQL>exit
    user@ubuntu:~/eXtremeDB$
     

To stop the Feed Handler, press Control-C in the console window (on Linux you may also send it a SIGTERM signal using the “kill” command). Since the Feed Handler was configured to use an in-memory database, its contents are lost when the application shuts down. To prevent accidental data loss, the Feed Handler application requests the user to repeat the Control-C key combination before shutting down the database. Until the confirmation is received, the database remains intact and can be accessed using xSQL:

     
    ^C
            
    2017-01-18 02:15:48 INFO app.cpp:188: stopped
    2017-01-18 02:15:49 WARNING app.cpp:230: shutting down Feed handler, in-memory database will be destroyed; press Ctrl-C again or issue "kill 20189" command to complete shutdown
     

When the Feed Handler is stopped, the Generator can be stopped as well, using the same Control-C key combination.

 

Prerequisites

The handler modules of the Feed Handler application require the corresponding frameworks: Vela SMDS, Refinitiv RFA and Refinitiv EMA.

The Feed Handler application was designed to be used with the following versions of the frameworks:

Failing to meet the framework version requirements may result in abnormal functioning.

 

Environment variables

The following environment variables need to be set:

Please note that the frameworks may contain multiple sets of shared libraries for different target operating systems and compilers. For example, RFA may contain shared libraries for CentOS (Libs/OL7_64_GCC482) and Red Hat Enterprise Linux (Libs/RHEL6_64_GCC444). Be careful to point the LD_LIBRARY_PATH entries to the correct directories.

 

Building the Feed Handler

If you have an eXtremeDB Financial Edition source code package, you can build the Feed Handler application from source. The build procedure is described in section "Building the Feed Handler from Source Code".

 

Running the Feed Handler

The application executable file is named fh. It is located in the target/bin subdirectory of the eXtremeDB Financial Edition package.

Before launching the application, make sure that the LD_LIBRARY_PATH environment variable contains entries for eXtremeDB and SMDS shared library directories - no LD_LIBRARY_PATH settings are required for the EMA module, since it is linked statically with the Elektron API libraries. On Windows, the PATH environment variable can be used.

To launch the application, prepare the required configuration files and launch the Feed Handler executable file. For information on Feed Handler command line arguments and configuration files, refer to the sections below.

To gracefully shut down the Feed Handler running in the foreground, press Ctrl-C. It is also acceptable to send it the SIGTERM signal on Linux (using the “kill <PID>” command), which is helpful when the application runs in the background, or as a daemon. The application will also stop on conditions making its further functioning impossible, such as running out of memory when using in-memory database.

Unlike a shared-memory database, the conventional in-memory database is destroyed upon application exit. In order to prevent data loss, if the application uses a non-persistent database in conventional memory and its embedded RSQL server is enabled, it will pause and require confirmation before shutting down the database in order to make it possible for the user to export database contents, if needed. Refer to the “Accessing Feed Handler database contents” section for further information.

 

Accessing Feed Handler database contents

The Feed Handler application allows the user to set various parameters of the eXtremeDB database such as: choose a persistent (disk) or in-memory database, set database size, file paths, etc.

The recommended approach to accessing the database is to enable the embedded RSQL server in the Feed Handler application configuration file. In this case the database can be accessed from RSQL-enabled clients, as well as from xSQL.

When the Feed Handler application is running and its RSQL server is enabled, the user can connect to the database and execute SQL queries using xSQL. For instance, if the RSQL server is configured to use port 5001 and the Feed Handler is running on localhost:

     
    $ xsql @localhost:5001
    xsql started
    Runtime configuration
    Remote node(s)       : localhost:5001
    Runtime              : Debug
    XSQL>select count(*) from MyQuotes;
    #1
    -------------------------------------------------
    248361
     
    Selected records: 1
     

For more information on RSQL server configuration, refer to the “database” section of the “Feed Handler Configuration” section. For information on xSQL, refer to the eXtremeSQL Quick Start.

 

Using the Feed Handler Web Console to Examine Application and Database Status

The Feed Handler application has an embedded web console that allows the user to examine the following:

The web console is always enabled. Its address and port are specified in the Feed Handler application configuration file’s “database” section (see the “Feed Handler Configuration” section for details). In order to access the console, launch a web browser and enter the address and port of the console in the address bar, e.g.:

     
    http://localhost:8082
     

 

Vertical Storage Support in Feed Handler

The Feed Handler is able to store incoming data in vertical storage, i.e. using eXtremeDB sequences. Please see the Vertical Storage Table Configuration section below for configuration details.

Sequence data buffering

It is inefficient to append elements to sequences one by one. Hence, the Feed Handler buffers the incoming records in a temporary "flat" table, and then moves them to the target table. The number of the buffered records is defined by the sequenceBufferSize parameter.

Ordered sequences

The common limitations for ordered sequences apply:

The Feed Handler checks the database table configuration upon startup and rejects it unless these requirements are met.

If no sequence order is defined, the records will be stored in the order of their arrival.

To define an ordered sequence, set the sequenceOrder parameter of the respective field to asc or desc, as needed.

Out-of-order record handling

It is possible for some records to arrive in the wrong order, breaking the ordering rule defined by the configuration. The Feed Handler supports optional recovery of the records' order.

If the order cannot be recovered, the records are stored in the "out-of-order" table ("OO table" for brevity). This is a regular (i.e. "flat") table that has the same name as the data table, with an _OO_ prefix.

No recovery

This is the default behavior. The out-of-order records will be moved to the OO table.

Sequence order recovery

Due to the constraints of the out-of-order sequence element insertion, the Feed Handler will attempt to insert out-of-order records one by one. Those that still fail to be inserted will be moved to the OO table.

This approach will obviously carry a performance penalty, but it can be convenient if out-of-order records are relatively rare.

 

Feed Handler Configuration

The Feed Handler application is configured using command line arguments and a configuration file.

Command Line Arguments

Currently, only one command line argument is supported.

Configuration File

The Feed Handler application configuration file is a text file that contains generic application settings and database parameters, specifies frameworks to be used (EMA, SMDS), subscriptions, database schema definitions, etc. It is structured as a JSON file with a few extensions:

Apart from the Feed Handler application configuration file, the supported third-party frameworks (SMDS, EMA) also require separate configuration files in their own formats. The paths to these files are set in the application configuration file. Refer to the following sections for details.

Refer to the examples below that demonstrate usage off application configuration files.

The configuration file has the following structure:

The contents of each of these sections and subsections are described below.

The "application" Section

This section contains settings for the application itself.

The "logging" subsection

This subsection contains a list of loggers. Each logger has the following parameters:

The "handler" section

This section contains settings for market data retrieval, processing and storage.

The “config” subsection

This subsection contains handler module-specific settings.

For the SMDS module:

For the Refinitiv EMA module:

For the Refinitiv RFA module:

The “tables” subsection

This subsection defines the mapping of the market data records’ fields to the database tables. Each table is described by a JSON object that contains the following fields:

Please note that fields’ SQL types are not specified in the “fields” array. For SMDS feeds, these types are fixed and correspond to the types of fields of SMDS entity classes (MDQuote, MDTrade, etc.) For TR OMM-based feeds (RFA and EMA), the field types are retrieved from the provider's service dictionary.

Vertical Storage Table Configuration

Sequences are declared in the table mappings, together with the related settings. For example:

     
    "tables": [
        {
            "name": "MyQuotes",
            "sequenceBufferSize": 300,
            "sequenceOrderRecovery": "true",
            "fields":
            [
                { "source": "symbol", "target": "symbol" },
                { "source": "timestamp", "target": "timestamp", "sequence": "true", "sequenceOrder": "asc" },
                { "source": "bid", "target": "bid", "sequence": "true" },
                { "source": "ask", "target": "ask", "sequence": "true" },
            ],
            "indexes": [{ "name": "pk", "unique": "true", "fields": ["symbol"] }]
        }
    ]
     

The “dataSources” subsection

This subsection contains a list of “data sources” in the given feed. For SMDS feeds, this must contain a list of session managers corresponding to the SMDS configuration file. For TR feeds, the services required by the user are listed here; these services must be available in the RFA Source Directory. Each data source is described by a JSON object that contains the following fields:

For the SMDS module, the “type” field must specify a valid SMDS session manager type name (e.g. “CTASessionMgr”, “ITCH5SessionMgr”). The list of available SMDS session managers is available in the SMDS User Guide. The “name” field must correspond to the session manager name specified in the session manager configuration section of the SMDS configuration file.

For the RFA and EMA modules, the “type” field should be left empty, and the “name” field must contain the name of the service available in the provider's Source Directory (i.e. “DIRECT_FEED”). The list of available service names depends on the market data provider.

The “subscriptions” subsection

This subsection describes subscriptions to be made.

Each subscription is described by a JSON object that contains the following fields:

SMDS feeds support “wildcard” subscriptions: it is possible to subscribe to all symbols provided by a data source by passing an asterisk (‘*’) for the instrument name.

Instrument subscriptions for the RFA and EMA feeds must list symbol names (RICs) explicitly, as wildcard subscriptions are not supported.

*For convenience, it is possible to specify symbol lists for subscriptions in external files using the “symListFile” field, which must contain the path to the symbol list file. This file uses JSON syntax (same as the Feed Handler application configuration files) and should contain a single array called “symbols”. This array must contain strings representing symbol names, e.g.:

     
    "symbols": [
    "SYM1",
    "SYM2"
    ]
     

If both the “sym” and “symListFile” fields are set, their symbol lists are merged. Duplicate symbol names are ignored.

The “database” section

This section contains eXtremeDB database settings for the Feed Handler.

An example of the “database” section specifying an in-memory database in conventional memory:

     
    "database":
    {
        "transactionManager": "mvcc",
        "useSharedMemory": "false",
        "useDiskDatabase": "false",
        "databaseName": "RFA_inmem_db",
        "databaseSize": "1073741824"
    }
     

Please note that a persistent database will require additional parameters:

     
    "database":
    {
        "transactionManager": "mvcc",
        "useSharedMemory": "false",
        "useDiskDatabase": "true",
        "databaseName": "RFA_inmem_db",
        "databaseSize": "1073741824",
        "databaseFileName": "RFA_db.dbs",
        "databaseLogFileName": "RFA_db.log",
        "diskCacheSize": "1073741824",
    }
     

Running SQL statements and scripts upon database startup:

     
    "database":
    {
        "transactionManager": "mvcc",
        "useSharedMemory": "false",
        "useDiskDatabase": "false",
        "databaseName": "RFA_inmem_db",
        "databaseSize": "1073741824"
        "SQLstatements": "create table test (t int);",
            
        "SQLscripts": ["create_tables.sql", "import_assets.sql"]
            
    }
     

The “RSQLserver” subsection

This subsection contains settings for the RSQL server.

If enabled, the Feed Handler application will start an embedded RSQL server. It will then be possible to access the database contents using the eXtremeDB RSQL protocol, e.g. using the xSQL, or a custom RSQL client (see eXtremeSQL and xSQL documentation for more details).

An example of an RSQL server configuration for an in-memory database:

     
    "database":
    {
        "transactionManager": "mvcc",
        "useSharedMemory": "false",
        "useDiskDatabase": "false",
        "databaseName": "RFA_inmem_db",
        "databaseSize": "1073741824",
        "RSQLserver": 
            
        {
            
            "enable": "true",
            
            "port": 5001
            
        }
    }
     

The “WebConsole” subsection

This subsection contains settings for the Feed Handler web console.

The embedded web console is used to examine application status and statistics in real time.

An example of web console configuration subsection in the “database” section:

     
    "database":
    {
        "transactionManager": "mvcc",
        "useSharedMemory": "false",
        "useDiskDatabase": "false",
        "databaseName": "RFA_inmem_db",
        "databaseSize": "1073741824",
        "WebConsole":  
        {
            
            "addr": "0.0.0.0",
            
            "port": 8082
            
        }
            
    }
     

 

Vela SMDS-Specific Database Schema Configuration

SMDS provides multiple kinds of data through its MDListener interface callbacks. Except for “Custom Event” data, SMDS provides all of its records as C++ classes with fixed layout. SMDS module tables store data provided by the MDListener callbacks. The supported table types and their fields (as specified in the “source” field of the “fields” array of a table definition) are:

An example of a table containing Quote data:

     
    {
        "type": "Quote",
        "name": "MyQuotes",
        "fields":
        [
            { "source": "bid", "target": "bid" },
            { "source": "ask", "target": "ask" },
            { "source": "condition", "target": "condition" }
        ]
    }
     

For a more detailed explanation of these classes and their fields, refer to the SMDS API Guide and SMDS User Guide. Refer to the feed specific information in the SMDS User Guide for information on data provided by certain feeds.

 

Fields Supported By Multiple Tables

Besides the table-specific fields listed above, all tables (except for OrderReplace) support the following fields from the InstrumentDef SMDS class: “symbol”, “exchange”.

All of the tables corresponding to SMDS classes derived from MDBaseData (i.e. all but OrderReplace and SecurityStatusChange) also support the following fields: "exchTimeSec", "exchTimeMicro", "exchTimestamp", "recvTimeSec", "recvTimeMicro", "recvTimestamp", "MDRCRecvTimeSec", "MDRCRecvTimeMicro", "MDRCRecvTimestamp".

All of the tables listed above may also include DB write timestamp fields: "DBTimestamp".

Thus, the table definition for Quotes may contain such common fields, e.g.:

     
    {"source": "symbol", "target": "symbol"},
    {"source": "exchTimeSec", "target": "exchTimeSec"},
    {"source": "exchTimeMicro", "target": "exchTimeMicro"},
    {"source": "DBTimestampSec", "target": "DBTimestampSec"},
    {"source": "DBTimestampMicro", "target": "DBTimestampMicro"}
     

 

Custom Event Tables

Apart from well-defined records provided by the MDListener callbacks in the form of C++ classes, SMDS supports “custom” events, i.e. events that cannot be represented as any of these classes. These custom events are represented internally in SMDS as data structures containing event type code and a dynamic array of fields.

SMDS custom events are stored in database tables just like other records, except for this limitation: custom event field data is always stored in textual (string) form.

All table type names for custom events have the following structure: “Custom.” prefix followed by custom event type string, i.e. “Custom.CE_OPN_CLS_HG_LW_STL_PRICE”. Event type strings correspond to the enumeration values defined in MDCustomEvent::EventType enumeration in the SMDS MDCustomEvent.h header.

Custom event field names correspond to the enumeration values defined in the MDCustomEvent::EventFields enumeration in the SMDS MDCustomEvent.h header, i.e. “CF_AUCTION_PRICE”.

The exact number and names of custom event fields for each custom event type depend on the feed. They are specified in the feed-specific information section of the SMDS User Guide.

Below is an example of a “custom event” table containing information on close prices for the CTA feed, corresponding to the description in the SMDS User Guide:

     
    {
        "type": "Custom.CE_OPN_CLS_HG_LW_STL_PRICE",
        "name": "MyClosePrice",
        "fields":
        [
            { "source": "symbol", "target": "symbol" },
            { "source": "exchange", "target": "exchange" },
            { "source": "CF_PREVCLOSE_PRICE", "target": "prevClosePrice" },
            { "source": "CF_PREV_CLOSE_DATE", "target": "prevCloseDate" }
        ]
    }
     

For more information on the structure of custom events, refer to the SMDS API Guide.

 

Refinitiv RFA- and EMA-Specific Database Schema Configuration

Unlike SMDS, the content of TR OMM-based feed records is not predefined. For instance, MarketPrice refresh messages contain a list of fields (RDM FieldList), which may vary depending on the data source. The types of the fields are not known beforehand as well: they must be retrieved from the Refinitiv RFA and EMA dictionary. This makes it impossible to know the database structure beforehand. Thus, for TR OMM-based feeds, the database schema is not created until a connection to the service is established.

The Refinitiv RFA and EMA handler modules currently supports data provided by MarketPrice and MarketByOrder RDM domains. So, the supported table types for TR OMM-based feeds are “MarketPrice”, “MarketByOrder.Summary” and “MarketByOrder.Entries”.

The field mappings in the table definitions for TR OMM-based feeds contain field names as specified in the RDM dictionary of the service (e.g. “RDNDISPLAY”, “RDN_EXCHID”, etc.), e.g.:

     
    "fields":
    [
        { "source": "RDNDISPLAY", "target": "RDNDISPLAY" },
        { "source": "RDN_EXCHID", "target": "RDN_EXCHID" },
        { "source": "DIVPAYDATE", "target": "DIVPAYDATE" }
    ]
     

For the structure of the MarketPrice and MarketByOrder messages, refer to the RDM Usage Guide. For exact contents and data structure of the records, please consult your service provider.

Refinitiv RFA and EMA do not allow its users to perform “wildcard” subscriptions to all symbols provided by an exchange. The list of symbols must be specified explicitly in the “subscriptions” subsection of the configuration file.

Feed Handler-Defined Table Fields

In addition to the fields defined in the RDM dictionary, Feed Handler defines some additional fields that are specific to certain table types and are described in the sections below.

Further, all TR tables may also include DB write timestamp fields: "DBTimestamp".

MarketPrice Tables

Tables of the MarketPrice type contain L1 market data provided by the MarketPrice domain of the RDM. Apart from the fields defined in the RDM dictionary, these tables may also contain the following fields:

An example of a MarketPrice table definition:

     
    {
        "type": "MarketPrice",
        "name": "MyMarketPrice",
        "fields":
        [
            { "source": "RIC", "target": "RIC" },
            { "source": "respTypeNum", "target": "respTypeNum" },
            { "source": "RDNDISPLAY", "target": "RDNDISPLAY" },
            { "source": "RDN_EXCHID", "target": "RDN_EXCHID" },
            { "source": "DIVPAYDATE", "target": "DIVPAYDATE" }
        ]
    }
     

MarketByOrder.* Tables

Tables of the MarketByOrder.* type store L2 order books provided by the MarketByOrder domain of the RDM. Currently, two table types are supported for this domain: “MarketByOrder.Summary” and “MarketByOrder.Entries”. The former stores the summary data provided in the “summary” section of the MarketByOrder refresh message. The latter stores data provided in the MarketByOrder update messages.

MarketByOrder.Summary tables may contain the following fields in addition to the RDM fields:

MarketByOrder.Entries tables may contain the following fields in addition to the RDM fields:

An example of a summary and a data table for MarketByOrder data:

     
    {
        "type": "MarketByOrder.Summary",
        "name": "MyMarketByOrderSummary",
        "fields":
        [
            { "source": "RIC", "target": "RIC" },
            { "source": "PROD_PERM", "target": "PROD_PERM" },
            { "source": "CURRENCY", "target": "CURRENCY" },
            { "source": "TRD_UNITS", "target": "TRD_UNITS" },
        ],
    },
     
    {
        "type": "MarketByOrder.Entries",
        "name": "MarketByOrderEntries",
        "fields":
        [
            { "source": "RIC", "target": "RIC" },
            { "source": "orderId", "target": "orderId" },
            { "source": "entryAction", "target": "entryAction" },
            { "source": "ORDER_PRC", "target": "ORDER_PRC" },
            { "source": "ORDER_SIDE", "target": "ORDER_SIDE" },
            { "source": "ORDER_SIZE", "target": "ORDER_SIZE" },
            { "source": "QUOTIM_MS", "target": "QUOTIM_MS" },
        ],
    }
     

 

Third-Party Frameworks Configuration

Vela SMDS and Refinitiv RFA and EMA frameworks require additional configuration files in their own formats. Paths to these configuration files must be specified in the Feed Handler application configuration file. For a brief description of the structure and contents of these files, please refer to the framework-specific subsections below.

 

Refinitiv RFA Configuration File

The Refinitiv RFA configuration file has a custom format. This file contains session and connection settings, logging parameters, etc. The format and the contents of this file are described in the RFA Configuration Guide (supplied as a part of the RFA distribution). For an example of an RFA configuration file, refer to Appendix D.

The Feed Handler application requires at least one RFA configuration file to be present. Its location should be specified using the “configPath” and “configFile” keys of the “handler.tr” subsection in the Feed Handler configuration. All configuration nodes defined in this file are imported into the Default namespace of the RFA configuration database. However, it is possible to define additional configuration files and optionally import their contents into custom RFA namespaces. For further information, refer to the “Feed Handler Configuration” chapter.

Refinitiv EMA Configuration File

The Refinitiv EMA configuration file is an XML file. This file contains session and connection settings, logging parameters, etc. The format and the contents of this file are described in the EMA Configuration Guide (supplied as a part of the EMA distribution). For an example of an EMA configuration file, refer to Appendix B.

Vela SMDS Configuration File

The SMDS configuration file is an XML file. Its structure is described in the SMDS User Guide. It contains both generic SMDS settings (license file location, logging, etc.) and feed-specific settings (session managers, connections, etc.). For an example of a SMDS configuration file, refer to Appendix D.

 

Appendices

Please note that all configuration files are provided for demonstration purposes only. Configuration settings must be chosen carefully on a case-by-case basis. Discussion of SMDS and Refinitiv RFA and EMA configuration files is beyond the scope of this document.

Appendix A - Sample Configurations

Below are some examples of the Feed Handler configuration sections. Parameters are explained inline.

Application Settings

The “application” section describes generic application settings. Below are settings for terminal log output and release mode:

     
    "application":
    {
        "logging":
        [
        # Print log messages with priorities "INFO" and above to the terminal
            { "type": "stdout", "level": "INFO" }
        ]
    }
     

To write log messages of DEBUG severity and above to a log file, add the corresponding line to the “logging” array:

     
    "application":
    {
        "logging":
        [
            { "type": "stdout", "level": "INFO" },
            { "type": "file", "level": "DEBUG", "filename": "feedhandler.log" },
            
        ]
    }
     

Database Settings

The “database” section contains eXtremeDB database settings. Below is an example of an in-memory database. RSQL server is enabled to allow for connecting to the database using xSQL and RSQL-enabled applications.

     
    "database":
    {
        "transactionManager": "mvcc", # Transaction manager
        "useSharedMemory": "false", # Don't use shared memory
        "useDiskDatabase": "false", # Don't use disk database
        "databaseName": "inmem_db", # Database name
        "databaseSize": "1073741824", # Size of the database, bytes
        # RSQL server parameters
        "RSQLserver": 
        {
            "enable": "true", # Enable RSQL server
            "port": 5001, # Port number for incoming RSQL connections
        },
    }
     

To create a persistent (disk) database, add corresponding settings (RSQL server is still enabled for remote access):

     
    "database":
    {
        "transactionManager": "mvcc",
        "useSharedMemory": "false",
        "useDiskDatabase": "true", # Use disk database
        "databaseName": "disk_db",
        "databaseSize": "1073741824",
        "databaseFileName": "RFA_db.dbs", # Name of the database file
        "databaseLogFileName": "RFA_db.log", # Name of the database log file
        "diskCacheSize": "1073741824", # Disk cache size, bytes
         
        "RSQLserver": 
        {
            "enable": "true",
            "port": 5001,
        },
    }
     

Additionally, custom SQL statements and scripts can be run upon database startup:

     
    "database":
    {
        "transactionManager": "mvcc",
        "useSharedMemory": "false",
        "useDiskDatabase": "false",
        "databaseName": "inmem_db",
        "databaseSize": "1073741824",
        "SQLstatements": "create table test (t int);",
        "SQLscripts": ["create_tables.sql", "import_assets.sql"]
            
    }
     

Handler Module Settings

These settings describe which Feed Handler modules are used to connect to the market data sources, their specific settings (e.g. locations of configuration files, authentication settings), database tables, etc.

An example of a Refinitiv RFA listener that subscribes to Level 1 (MarketPrice domain) data from the DIRECT_FEED service of the provider and stores it in table MarketPrice:

     
    # Handler Module settings
    "handler":
    {
        # Module name: TR RFA module
        "name" : "trfeed",
        # Module-specific settings
        "config":
        {
            "useAppLog": "false", # Don't send RFA log messages to the Feed Handler application log
            "configPath": "./", # Directory containing RFA configuration file
            "configFile": "RFA.cfg", # Name of the RFA configuration file
            "sessionName": "RFA_session", # RFA session name (must correspond to the settings in the 
                                # RFA configuration file)
            "consumerName": "RFA_OMM_Consumer", # RFA consumer name
            "connStatsReportPeriod": 5, # Period of connection statistics reporting, seconds
            # Login settings
            "login": 
            {
                "userName": "user1", # username
                "appId": "123", # application identifier
            },
        },
     
        # Database tables configuration
        "tables":
        [
            {
                "type": "MarketPrice", # Table contents' type: MarketPrice domain data
                "name": "MarketPrice", # Table name
                # Mapping of fields from the incoming data stream to the database table's columns
                "fields":
                [
                    { "source": "RIC", "target": "RIC" },
                    { "source": "BID", "target": "BID" },
                    { "source": "ASK", "target": "ASK" }
                ],
            }
        ],
     
        # Data sources: data provider's services to request data from
        "dataSources":
        [
            {
                "type": "", # Data source type: must be empty for TR feeds
                "name": "DIRECT_FEED" # Service name
            },
        ],
     
        # Level 1 subscriptions (MarketPrice domain)
        "subscriptions":
        [
            {
                "dataSource": "DIRECT_FEED", # Service name to request data from
                "sym": ["TRI.N","IBM.N","ACN.N"], # List of symbols to subscribe to
                "domain": "MarketPrice", # OMM MARKET_PRICE domain
            },
        ],
    }
     

An example of a Vela SMDS listener that subscribes to Quote data from the CQS session manager (which must be defined in the SMDS configuration file):

     
    "handler":
    {
        # Module name: Vela SMDS module
        "name" : "smdsfeed",
     
        # Module-specific settings
        "config":
        {
            "configPath": "./", # location of the SMDS configuration file
            "configFile": "config.xml" # name of the SMDS configuration file
        },
        # Database tables configuration
        "tables":
        [
            {
                "type": "Quote", # Table contents' type: Quotes
                "name": "MyQuotes", # Table name
                 
                # Mapping of fields from the incoming data stream to the database table's columns
                "fields":
                [
                    { "source": "symbol", "target": "symbol" },
                    { "source": "bid", "target": "bid" },
                    { "source": "ask", "target": "ask" }
                ],
            }
        ],
        # Data sources: types and names of the Session Managers defined in the 
        # SMDS configuration file
        "dataSources":
        [
            {
                "type": "CTASessionMgr", # type of the Session Manager
                "name": "cqs" # name of the session manager
            }
        ],
        # Market data subscriptions
        "subscriptions":
        [
            {
                "dataSource": "cqs", # name of the datasource defined above
                "sym": "*", # wildcard subscription
            }
        ],
    }
     

An example of a Refinitiv EMA listener that subscribes to Level 1 (MarketPrice domain) data from the DIRECT_FEED service of the provider and stores it in table MarketPrice:

     
    "handler":
    {
        # Module name: Refinitiv EMA handler
        "name" : "emafeed",
     
        # Module-specific settings
        "config":
        {
             "configFile": "ema.xml" # name of the EMA configuration file
        },
        # Database tables configuration
        "tables":
        [
            {
                "type": "Quote", # Table contents' type: MarketPrice domain data                
		"name": "MarketPrice", # Table name
                 
                # Mapping of fields from the incoming data stream to the database table's columns
                "fields":
                [
                    { "source": "RIC", "target": "RIC" },
                    { "source": "BID", "target": "BID" },
                    { "source": "ASK", "target": "ASK" }
                ],
            }
        ],
        # Data sources: data provider's services to request data from 
        "dataSources":
        [
            {
                "type": "", # Data source type: must be empty for EMA feeds
                "name": "DIRECT_FEED" # Service name
            }
        ],
        # Level 1 subscriptions (MarketPrice domain)
        "subscriptions":
        [
            {
                "dataSource": "DIRECT_FEED", # Service name to request data from
                "sym": ["TRI.N","IBM.N","ACN.N"], # List of symbols to subscribe to
                "domain": "MarketPrice", # OMM MARKET_PRICE domain
            }
        ],
    }
     

Putting It All Together

Below is a sample configuration file for a Refinitiv RFA handler using an in-memory eXtremeDB database. The handler subscribes to both Level 1 and Level 2 market data (MarketPrice and MarketByOrder domains). Please note that the settings must be adjusted according to the usage scenarios. Please also note that the RFA configuration file is not provided in this appendix.

A valid configuration file, along with an RFA configuration file, is also available in the target/feedhandler/scripts/TR_RFA subdirectory of the eXtremeDB for HPC package.

     
    {
        # Application settings
        "application":
        {
            "logging":
            [
                # Print log messages with priorities "INFO" and above to the terminal
                { "type": "stdout", "level": "INFO" }
            ]
        },
         
        # Handler module settings
        "handler":
        {
            # Module name: TR RFA module
            "name" : "trfeed",
             
            # Module-specific settings
            "config":
            {
                "useAppLog": "false", # Don't send RFA log messages to the Feed Handler application log
                "configPath": "./", # Directory containing RFA configuration file
                "configFile": "RFA.cfg", # Name of the RFA configuration file
                "sessionName": "RFA_session", # RFA session name (must correspond to the settings in 
                                # the RFA configuration file)
                "consumerName": "RFA_OMM_Consumer", # RFA consumer name
                "connStatsReportPeriod": 5, # Period of connection statistics reporting, seconds
             
                # Login settings
                "login": 
                {
                    "userName": "user1", # username
                    "appId": "123", # application identifier
                },
            },
         
            # Database tables configuration
            "tables":
            [
                {
                    "type": "MarketPrice", # Table contents' type: MarketPrice domain data
                    "name": "MarketPrice", # Table name
                 
                    # Mapping of fields from the incoming data stream to the database table's columns
                    "fields":
                    [
                        { "source": "RIC", "target": "RIC" },
                        { "source": "RDN_EXCHID", "target": "RDN_EXCHID" },
                        { "source": "BID", "target": "BID" },
                        { "source": "ASK", "target": "ASK" }
                    ],
                },
             
                {
                    "type": "MarketByOrder.Summary", # Table contents' type: 
                                            # MarketByOrder domain summary data
                    "name": "MarketByOrderSummary", # Table name
                 
                    # Mapping of fields from the incoming data stream to the database table's columns
                    "fields":
                    [
                        { "source": "RIC", "target": "RIC" },
                        { "source": "CURRENCY", "target": "CURRENCY" },
                        { "source": "TRD_UNITS", "target": "TRD_UNITS" }
                    ],
                },
     
                {
                    "type": "MarketByOrder.Entries", # Table contents' type: 
                                            # MarketByOrder domain order data
                    "name": "MarketByOrderEntries", # Table name
                 
                    # Mapping of fields from the incoming data stream to the database table's columns
                    "fields":
                    [
                        { "source": "RIC", "target": "RIC" },
                        { "source": "orderId", "target": "orderId" },
                        { "source": "entryAction", "target": "entryAction" },
                        { "source": "ORDER_PRC", "target": "ORDER_PRC" },
                        { "source": "ORDER_SIDE", "target": "ORDER_SIDE" },
                        { "source": "ORDER_SIZE", "target": "ORDER_SIZE" },
                        { "source": "QUOTIM_MS", "target": "QUOTIM_MS" },
                    ],
                },
            ],
         
            # Data sources: data provider's services to request data from
            "dataSources":
            [
                {
                    "type": "", # Data source type: must be empty for TR feeds
                    "name": "DIRECT_FEED" # Service name
                },
            ],
     
            # Level 1 and Level 2 subscriptions (MarketPrice and MarketByOrder domains)
            "subscriptions":
            [
                {
                    "dataSource": "DIRECT_FEED", # Service name to request data from
                    "sym": ["TRI.N","IBM.N","ACN.N"], # List of symbols to subscribe to
                    "domain": "MarketPrice", # OMM MARKET_PRICE domain
                },
                {
                    "dataSource": "DIRECT_FEED", # Service name to request data from
                    "sym": "TRI.ARC", # Symbol to subscribe to
                    "domain": "MarketByOrder", # OMM MARKET_BY_ORDER domain
                },
            ],
         
        },
     
        # Database configuration
        "database":
        {
            "transactionManager": "mvcc", # Transaction manager
            "useSharedMemory": "false", # Don't use shared memory
            "useDiskDatabase": "false", # Don't use disk database
            "databaseName": "RFA_inmem_db", # Database name
            "databaseSize": "1073741824", # Size of the database, bytes
            # RSQL server parameters
            "RSQLserver": 
            {
                "enable": "true", # Enable RSQL server
                "port": 5001, # Port number for incoming RSQL connections
            },
        },
    }
     

 

Appendix B - Testing the Refinitiv EMA Handler Module Using the Feed Handler EMA Generator application

It is possible to use the bundled Generator application instead of a real market data feed for EMA handler evaluation purposes.

The Generator is a C++ application based on the EMA C++ Interactive Provider samples. It implements a simplistic OMM Provider that emits pseudo-random market data. The Feed Handler application can subscribe to this data stream and store it just like “real” data, which is useful for evaluating and learning to use the Feed Handler application.

Running the Generator

The Generator binary is available in the $(MCO_ROOT)/target/feedhandler/EMA_redistributable/Generator directory. It requires the RDM dictionary files (RDMFieldDictionary and enumtype.def) to run. These files can be obtained from the Elektron SDK distribution. Please place them in the Generator directory prior to running the application.

A sample EMA configuration file, EmaConfig.xml, is available in the Generator directory. It will be used by the EMA runtime by default.

Run the Generator application from the Generator directory:

 
    ./Generator
     

If the application launches successfully, the following line will be printed:

 
    OmmProvider ready
     

Configuring the Feed Handler

As described in this document, the Feed Handler application itself needs a configuration file. Furthermore, the EMA framework module needs its own configuration file. Examples of these files are given in the sections below.

Feed Handler Application Configuration File

Create a plain text file named "feed.cfg" in the directory from which you will launch the Feed Handler application. You can use the following configuration for testing:

     
    {
       "application":
        {
            "logging":
            [
               { "type": "stdout", "level": "INFO" }
            ]
        },
         
       "handler":
        {
            "name" : "emafeed",
            "config":
            {
                "configFile": "ema.xml"
            },
         
             "tables":
            [
                {
                    "type": "MarketPrice", 
                    "name": "MarketPrice",
                    "fields":
                    [
                        { "source": "RIC", "target": "RIC" },
                        { "source": "BID", "target": "BID" },
                        { "source": "ASK", "target": "ASK" }
                    ],
                },
            ],
         
            "dataSources":
            [
                {
                    "type": "",
                    "name": "DIRECT_FEED"
                },
            ],
     
            "subscriptions":
            [
                {
                    "dataSource": "DIRECT_FEED", 
                    "sym": "TRI.N",
                    "domain": "MarketPrice"
                }
            ],
         
        },
     
        "database":
        {
            "transactionManager": "mvcc", 
            "useSharedMemory": "false",
            "useDiskDatabase": "true",
            "databaseName": "intradaydb",
            "databaseSize": "1073741824",
            "databaseFileName": "EMA.dbs",
  	    "databaseLogFileName": "EMA.log",
	    "diskCacheSize": "1073741824",
            "RSQLserver": 
            {
                "enable": "true", 
                "port": 5001
            },
        },
    }
     

Note a few important details.

EMA Framework Configuration File

For detailed information on the EMA framework configuration file structure and contents, please refer to the EMA Configuration Guide (supplied with the EMA distribution archive).

As we have defined in the Feed Handler application file, the EMA configuration file must be named ema.xml. Create a plain-text file called “ema.xml” in the same directory as the “feed.cfg” file with the following contents:

     
 <?xml version="1.0" encoding="UTF-8"?>
 <EmaConfig>
 <ConsumerGroup>
       <DefaultConsumer value="Consumer_1"/>
       <ConsumerList>
               <Consumer>
		   <Name value="Consumer_1"/>
		   <Channel value="Channel_1"/>
		   <Logger value="Logger_1"/>
		   <Dictionary value="Dictionary_1"/>
		   <XmlTraceToStdout value="0"/>
		</Consumer>
        <ConsumerList>
</ConsumerGroup>
<ChannelGroup>
	<ChannelList>
		<Channel>
		   <Name value="Channel_1"/>
                   <ChannelType value="ChannelType::RSSL_SOCKET"/>
       	    <CompressionType value="CompressionType::None"/>
      		    <GuaranteedOutputBuffers value="5000"/>
       	    <ConnectionPingTimeout value="30000"/>
            	    <TcpNodelay value="1"/>
                   <Host value="localhost"/>
                   <Port value="14002"/>
                </Channel>
         </ChannelList>
</ChannelGroup>
<LoggerGroup>
         <LoggerList>
                 <Logger>
                    <Name value="Logger_1"/>
                    <LoggerType value="LoggerType::Stdout"/>
                    <LoggerSeverity value="LoggerSeverity::Success"/>
                 </Logger>
  	  </LoggerList>
</LoggerGroup>
<DictionaryGroup> 
         <DictionaryList>
           	 <Dictionary>
                    <Name value="Dictionary_1"/>
                    <DictionaryType value="DictionaryType::ChannelDictionary"/>
          	 </Dictionary>
  	  </DictionaryList>         
</DictionaryGroup>

</EmaConfig>
 

Note that the <Port> value corresponds to the port configured in the <Server> section of the Generator configuration file.

Discussion of all possible configuration keys and values is beyond the scope of this document. For more information please refer to the EMA Configuration Guide.

Running the Feed Handler

Once all configuration files are ready and the Generator is launched successfully, one can start the Feed Handler application. To start it from the command line, change the current directory to the one containing the feed.cfg and ema.xml configuration files and launch the fh binary:

 
    /home/user/eXtremeDB/target/bin/fh
     

If no errors are reported, one can connect to the database using xSQL and examine the data being stored:

 
    /home/user/eXtremeDB/target/bin/xsql @localhost:5001
     

For more information on xSQL, refer to the eXtremeSQL Quick Start documentation.

 

Appendix C - Testing the Refinitiv RFA Handler Module Using the RFA Provider_Interactive Sample

It is possible to use the RFA Provider_Interactive sample (called “Provider” for brevity hereafter in this document) instead of a real market data feed for RFA handler evaluation purposes.

The Provider is a C++ application that is a part of the RFA C++ framework. It implements a simplistic OMM Provider that emits pseudo-random market data. The Feed Handler application can subscribe to this data stream and store it just like “real” data, which is useful for evaluating and learning to use the Feed Handler application.

 

Building the Provider

The Provider application can be found in the Examples/Provider_Interactive subdirectory of the RFA distribution. It must be built and run according to the instructions in its Readme file, as well as the RFA Readme file. In general, it is sufficient to:

 
    ldconfig -n /home/user/rfa8/Libs/RHEL6_64_GCC444
     
 
    cd /home/user/rfa8/Examples/Provider_Interactive
    make
     

Once the build process finishes, the binary can be found in a newly created subdirectory (typically named “RHEL6_64_GCC444”), alongside symbolic links to configuration files and dictionaries.

 

Configuring the Provider

Prior to launching the Provider application, one should edit its configuration files: ExampleRFA.cfg and Provider_Interactive.cfg. The former contains the session and connection parameters for the Provider (host, port, etc.). The latter file contains Provider-specific configuration (list of provided symbols, RDM domains, update frequency, etc.).

The Provider_Interactive.cfg file contains a line with Provider session name which will be needed a bit later, e.g.:

     
    \Session = "SessionOMMProv"
     

The ExampleRFA.cfg file contains settings for this session, alongside other settings:

     
    [...]
    \Connections\Connection_OMMProv\connectionType = "RSSL_PROV"
    \Connections\Connection_OMMProv\rsslPort = "<rsslPort-for listening>"
    [...]
    \Sessions\SessionOMMProv\connectionList = "Connection_OMMProv"
     

Note that the settings for the Provider’s session (called “SessionOMMProv”) are located in the \Sessions\SessionOMMProv branch of the configuration file (other Sessions for other examples are configured in this file as well). The connectionList configuration key contains the single connection for this session: Connection_OMMProv, whose parameters we need to change in the \Connections\Connection_OMMProv branch of this file.

The “rsslPort” parameter of the Connection_OMMProv must be set to an actual value instead of the placeholder. It must be set to any valid port value corresponding to an unoccupied TCP port on the machine (typically larger than 1023 on Linux systems); the Provider application will accept incoming connections on this port. Thus, this is the port the Feed Handler application will connect to.

For demonstration purposes, we set the port number to 30000:

     
    \Connections\Connection_OMMProv\rsslPort = "30000"
     

When the configuration is done, one can start the Provider by simply launching its executable file with the command line:

     
    ./provider_interactive
     

If the Provider launches successfully, one will see a log output ending with text similar to this:

     
    Logger message comes from: RSSL_Adapter
    Log Id: 10114
    Event Type: 1
    Message Text: [Tue Nov 10 17:00:00 2016]: (ComponentName) RSSL_Adapter: (Severity) Information: Connection "Connection_OMMProv" listen socketID 10 success on port 30000
 
    Logger message comes from: RSSL_Prov_Adapter
    Log Id: 7012
    Event Type: 1
    Message Text: [Tue Nov 10 17:00:00 2016]: (ComponentName) RSSL_Prov_Adapter: (Severity) Information: Listen Connection Status Changed, Connection: Default::Connection_OMMProv
    State:	Up
    StatusCode:	None
    StatusText:	Connection up
     
 
    Listen Connection Connection_OMMProv - Up: Connection up
     

Note that the port number that we configured appears in the message above:

     
    “Connection "Connection_OMMProv" listen socketID 10 success on port 30000”.
     

 

Configuring the Feed Handler

As described in this document, the Feed Handler application itself needs a configuration file. Furthermore, the RFA framework module needs its own configuration file. Examples of these files are given in the sections below.

 

Feed Handler Application Configuration File

Create a plain text file called “feed.cfg” in the directory from which you will launch the Feed Handler application. You can use the following configuration for testing:

     
    {
        "application":
        {
            "logging":
            [ { "type": "stdout", "level": "INFO" } ]
        },
         
        "handler":
        {
            "name" : "trfeed",
             
            "config":
            {
                "useAppLog": "false",
                "configPath": "./",
                "configFile": "tr.cfg",
                "sessionName": "TR_app_session",
                "consumerName": "TR_OMM_Consumer",
                "connStatsReportPeriod": 5,
                "login": 
                {
                    "userName": "user1",
                    "appId": "123"
                }
            },
         
            "tables":
            [
                {
                    "type": "MarketPrice",
                    "name": "MyMarketPrice",
                    "fields":
                    [
                        { "source": "DBTimestampSec", "target": "DBTimestampSec" },
                        { "source": "DBTimestampMicro", "target": "DBTimestampMicro" },
                        { "source": "RIC", "target": "RIC" },
                        { "source": "respTypeNum", "target": "respTypeNum" },
                        { "source": "RDNDISPLAY", "target": "RDNDISPLAY" },
                        { "source": "RDN_EXCHID", "target": "RDN_EXCHID" },
                        { "source": "DIVPAYDATE", "target": "DIVPAYDATE" },
                        { "source": "TRDPRC_1", "target": "TRDPRC_1" },
                        { "source": "BID", "target": "BID" },
                        { "source": "ASK", "target": "ASK" },
                        { "source": "ACVOL_1", "target": "ACVOL_1" },
                        { "source": "NETCHNG_1", "target": "NETCHNG_1" },
                        { "source": "ASK_TIME", "target": "ASK_TIME" },
                    ],
                }
            ],
         
            "dataSources":
            [ { "type": "", "name": "DIRECT_FEED" } ],
         
            "subscriptions":
            [
                { 
                    "dataSource": "DIRECT_FEED", 
                    "sym": "TRI.N", 
                    "domain": "MarketPrice" 
                }, 
            ],
        },
     
        "database":
        {
            "transactionManager": "mvcc", 
            "useSharedMemory": "false", 
            "useDiskDatabase": "true",
            "databaseName": "intradaydb",
            "databaseSize": "1073741824",
            "databaseFileName": "TR.dbs",
            "databaseLogFileName": "TR.log",
            "diskCacheSize": "1073741824",
            "RSQLserver": 
            {
                "enable": "true", 
                "port": 5001,
            },
        },
    }
     

Note a few important details.

 

RFA Framework Configuration File

For detailed information on the RFA framework configuration file structure and contents, please refer to the RFA Configuration Guide (supplied with the RFA distribution archive).

As we have defined in the Feed Handler application file, the RFA configuration file must be named tr.cfg. Create a plain-text file called “tr.cfg” in the same directory as the “feed.cfg” file with the following contents:

     
    \Sessions\TR_app_session\connectionList = "Connection_TR_app"
 
    \Connections\Connection_TR_app\connectionType = "RSSL"
    \Connections\Connection_TR_app\rsslPort = "30000"
    \Connections\Connection_TR_app\hostName = "localhost"
 
    \Logger\AppLogger\useInternalLogStrings = True
     

Note that the “rsslPort” value corresponds to the port we have set for the Provider. Also note that the session name (TR_app_session) corresponds to what we set in the Feed Handler application configuration file.

Discussion of all possible configuration keys and values is beyond the scope of this document. For more information please refer to the RFA Configuration Guide.

 

Running the Feed Handler

Once all configuration files are ready and the Provider is launched successfully, one can start the Feed Handler application. To start it from the command line, change the current directory to the one containing the feed.cfg and tr.cfg configuration files and launch the fh binary:

     
    /home/user/eXtremeDB/target/bin/fh
     

If no errors are reported, one can connect to the database using xSQL and examine the data being stored:

     
    /home/user/eXtremeDB/target/bin/xsql @localhost:5001
     

For more information on xSQL, refer to the eXtremeSQL Quick Start documentation.

 

Appendix D - Testing the Vela SMDS Handler Module Using Feed Data Dumps

The SMDS framework is able to “replay” binary dumps of market data traffic. This feature is useful for SMDS handler evaluation.

Market data “playback” is configured in the SMDS configuration file.

 

Configuring the Feed Handler

As described in this document, the Feed Handler application itself needs a configuration file. Furthermore, the SMDS framework needs its own configuration file. Examples of these files are given in the sections below.

 

Feed Handler Application Configuration File

Create a plain text file called “feed.cfg” in the directory from which you will launch the Feed Handler application. You can use the following configuration for testing:

 
    {
        "application":
        {
            "logging":
            [ { "type": "stdout", "level": "INFO" } ]
        },
         
        "handler":
        {
            "name" : "smdsfeed",
            "config":
            { "configPath": "./", "configFile": "config.xml" },
         
            "tables":
            [
                {
                    "type": "Quote",
                    "name": "MyQuotes",
                    "fields":
                    [
                        { "source": "symbol", "target": "symbol" },
                        { "source": "exchange", "target": "exchange" },
                        { "source": "DBTimestampSec", "target": "DBTimestampSec" },
                        { "source": "DBTimestampMicro", "target": "DBTimestampMicro" },
                        { "source": "bid", "target": "bid" },
                        { "source": "ask", "target": "ask" },
                    ],
                },
                {
                    "type": "Custom.CE_OPN_CLS_HG_LW_STL_PRICE",
                    "name": "MyClosePrice",
                    "fields":
                    [
                        { "source": "symbol", "target": "symbol" },
                        { "source": "exchange", "target": "exchange" },
                        { "source": "CF_PREVCLOSE_PRICE", "target": "prevClosePrice" },
                        { "source": "CF_PREV_CLOSE_DATE", "target": "prevCloseDate" },
                    ],
                },
            ],
         
            "dataSources":
            [
                { "type": "CTASessionMgr", "name": "cqs" }
            ],
         
            "subscriptions":
            [
                {
                    "dataSource": "cqs",
                    "sym": "*",
                },
            ],
         
        },
     
        "database":
        {
            "transactionManager": "mvcc", 
            "useSharedMemory": "false", 
            "useDiskDatabase": "true",
            "databaseName": "intradaydb",
            "databaseSize": "1073741824",
            "databaseFileName": "SMDS.dbs",
            "databaseLogFileName": "SMDS.log",
            "diskCacheSize": "1073741824",
            "RSQLserver": 
            {
                "enable": "true", 
                "port": 5001,
            },
        },
    }
     

Note a few important details:

 

SMDS Framework Configuration File

For information on the SMDS framework configuration file structure and contents please refer to the SMDS User Guide.

As we have defined in the Feed Handler application file, the SMDS configuration file must be named “config.xml”. Create a plain-text file called “config.xml” in the same directory as the “feed.cfg” file with the following contents:

     
    <config>
    <License file="PATH_TO_THE_LICENSE_FILE" />
    <MDLibraryList>
    <MDLibrary name="libmd-cta.so" />
    </MDLibraryList>
    <Logger file="smds.log" level="debug" max-size-in-mb="300" />
    <Resources>
    <Resource type="CTASessionMgr" name="cqs" lb-type="exchange-dictated" record="false" conn-type="file" pb-file="cqsE1-2015-11-03-4.1.0-record.1446568260.pb.gz" pb-buffer="1048576">
    <MDConnections>
        <MDConnection max-record-size="1000" max-q-size="200000" name="CTA-GRP-2">
            <MDLineHandler source="CQS" name="CQS-L1" primary-mc-channel="233.200.79.32:61032"  secondary-mc-channel="233.200.79.48:61048"/>
            <MDLineHandler source="CQS" name="CQS-L2" primary-mc-channel="233.200.79.33:61033" secondary-mc-channel="233.200.79.49:61049"/>
            <MDLineHandler source="CQS" name="CQS-L3" primary-mc-channel="233.200.79.34:61034" secondary-mc-channel="233.200.79.50:61050"/>
            <MDLineHandler source="CQS" name="CQS-L4" primary-mc-channel="233.200.79.35:61035" secondary-mc-channel="233.200.79.51:61051"/>
            <MDLineHandler source="CQS" name="CQS-L5" primary-mc-channel="233.200.79.36:61036" secondary-mc-channel="233.200.79.52:61052"/>
            <MDLineHandler source="CQS" name="CQS-L6" primary-mc-channel="233.200.79.37:61037"  secondary-mc-channel="233.200.79.53:61053"/>
            <MDLineHandler source="CQS" name="CQS-L7" primary-mc-channel="233.200.79.38:61038"  secondary-mc-channel="233.200.79.54:61054"/>
            <MDLineHandler source="CQS" name="CQS-L8" primary-mc-channel="233.200.79.39:61039" secondary-mc-channel="233.200.79.55:61055"/>
            <MDLineHandler source="CQS" name="CQS-L9" primary-mc-channel="233.200.79.40:61040" secondary-mc-channel="233.200.79.56:61056"/>
            <MDLineHandler source="CQS" name="CQS-L10" primary-mc-channel="233.200.79.41:61041" secondary-mc-channel="233.200.79.57:61057"/>
            <MDLineHandler source="CQS" name="CQS-L11" primary-mc-channel="233.200.79.42:61042" secondary-mc-channel="233.200.79.58:61058"/>
            <MDLineHandler source="CQS" name="CQS-L12" primary-mc-channel="233.200.79.43:61043" secondary-mc-channel="233.200.79.59:61059"/>
        </MDConnection>
    </MDConnections>
    </Resource>
    </Resources>
    </config>
     

Note that this file should be edited prior to launching the feed handler.

A few important details:

The SMDS configuration file is complex. Different session managers have different sets of supported settings. Discussion of all possible configuration keys and values is beyond the scope of this document. For more information please refer to the SMDS User Guide.

 

Running the Feed Handler

Once all configuration files are ready, one can start the Feed Handler application. To start it from the command line, change the current directory to the one containing the feed.cfg and config.xml configuration files and launch the fh binary:

     
    /home/user/eXtremeDB/target/bin/fh
     

When the Feed Handler application is started, it loads the SMDS framework, which, in turn, loads its configuration file and begins playback of the feed data dump file. The Feed Handler receives the records extracted from the data dump file and stores them in the database.

If no errors are reported, one can connect to the database using xSQL and examine the data being stored:

     
    /home/user/eXtremeDB/target/bin/xsql @localhost:5001
     

For more information on xSQL, refer to the eXtremeDB xSQL documentation.

 

Appendix E - Building the Feed Handler Application from Source Code

The Feed Handler application consists of the primary executable and loadable modules. All of these are built using a single makefile in the application’s root directory.

Prerequisites

In addition to the framework libraries required for the application to run, the corresponding C++ header files are required for compilation as well. These are included in the framework distribution archives supplied by their vendors.

Environment Variables

The environment variables must be set according to the instructions in the “Prerequisites” section of the “Getting Started” chapter.

If the SMDS_ROOT or RFA_ROOT environment variables are not set, the corresponding handler modules will be skipped during the build process.

Building the Application

When the third-party frameworks are prepared and the environment variables are set, one can build the Feed Handler simply by issuing make in the root folder of the application:

     
    cd /home/user/eXtremeDB/target/feedhandler
    make
     

If some Feed Handler modules should not be built, one can remove them from the FEED_MODULES list in the root makefile.

On Windows, Visual Studio project files can be used to build the Feed Handler application and the modules.

 

Appendix F - Testing the Feed Handler Application Using the Test Data Provider

It's possible to use the provided test data provider with the Feed Handler application. It may be useful to evaluate software without any need for external dependencies, for assessing hardware, measuring hardware capabilities, etc.

The test provider can generate test trades and quotes. It will start a thread for each table, and perform inserts at maximum speed.

Generated tables and fields are:

Table Field name Data type
Quote symbol string
exch char
qtime int8
bid float
ask float
bidsz int4
asksz int4
mode char
Trade symbol string
exch char
ttime int8
price float
size int4
stop bool
condition char

To use the test provider, include the following snippet into the feed handler configuration file:

     
    # Market data handler module configuration
    "handler":
    {
        # Use Test generator
        "name" : "testfeed",
         
        # Database tables subsection
        "tables":
        [
            # Table definition subsection for storage of quote data
            {
                # Type of the table. Depends on feed module.
                "type": "Quote",
                # Name of the table in the database
                "name": "MyQuotes",
                # Array of field mappings
                "fields":
                [
                    { "source": "symbol", "target": "symbol" },
                    { "source": "exch", "target": "exchange" },
                    { "source": "bid", "target": "bid" },
                    { "source": "ask", "target": "ask" },
                ],
            },
             
            # Another table definition subsection for storage of
            # trade data
            {
                "type": "Trade",
                "name": "MyTrades",
                "fields":
                [
                    { "source": "price", "target": "lastPrice" },
                    { "source": "size", "target": "lastSize" },
                ],
            },
        ],
    }
     

This will load the test generator, define 2 tables in the output database and define some field mappings.

 

Appendix G - References

Send feedback on this topic to McObject.Download documentation.© 2001-2024 McObject LLC. All rights reserved.