Using eXtremeDB Events and Time-To-Live Features in C

As explained in the introduction page, eXtremeDB provides the capability to automatically delete obsolete database objects through the Time-To-Live feature, and to manage database object new, delete, update and checkpoint events. The C APIs for these features are explained in the sections below.

Time To Live

The Time-To-Live (TTL) mechanism facilitates automatic deletion of objects according to TTL policies. Two TTL policies are supported: TTL_count and TTL_clock_time. The former sets an object count threshold, while the latter sets an object age threshold. Both policies can be set for a single class at the same time.

The time to live can be defined in seconds (sec), milliseconds (ms) and microseconds (us). The syntax is:

 
    TTL_count <max_object_count>;
     
    TTL_clock_time <max_object_age> <sec | ms | us>;
     

For example, the following schema definition sets both TTL policies for class A:

 
    class A
    {
        unsigned<4> id;
        char<256> name;
         
        tree<id> idx_id;
        TTL_count 10;
        TTL_clock_time 5 sec;
    };
     

With this schema, when more than 10 objects of class A are created, extra objects (those created previously) are deleted. Also, objects that had been stored for more than 5 seconds are deleted as well.

The TTL mechanism implementation adds two hidden fields, ttl_count@ and ttl_timestamp@, and two indexes, ttl_count_idx@ and ttl_timestamp_idx@. These fields and indexes are only added when the corresponding TTL policy is set for the given class. These fields’ values are set when a new object of the given class is created.

TTL policies are only checked and enforced when a READ_WRITE transaction is committed, before the commit’s first stage.

(An important side effect of this is that the indexes for the newly created objects (i.e. the ones created within the current transaction) are not updated yet, and thus these objects are not checked against the TTL policies. To mitigate this effect, a transaction checkpoint may be added before the commit.)

Side effects in a distributed environment

For distributed databases in a network (using eXtremeDB Cluster) it is important to note that clocks need to be carefully synchronized between machines participating in a cluster when the TTL_clock_time policy is used. Bear in mind the following:

1. In the cluster environment, the TTL clock is verified on the transaction initiator side only in the beginning of the commit phase 1. On the remote side the clock time is not verified and the transaction is applied regardless of the actual clock on that node. If the transaction is applied successfully on the remote nodes, the notifications are sent back and the initiator commits the transaction. The databases are kept consistent regardless of the clock on each node (i.e. their content is the same on every node as long as the transactions are committed).

2. It is possible that a node will have some data that violates the node’s TTL requirements — in the example above the clock on the remote node could be far ahead of the clock on the local node where the transaction had been initiated. By the remote node clock the record should’ve been removed, but it is going to be kept in the node’s database regardless.

3. It is also possible that the object would be removed from the node’s database even if the TTL on that node is not expired, because it has expired on a different node. For example, suppose that node1 clock is set to 1 pm and node2 clock is set to 2 pm. The transaction is initiated on the node1 and gets successfully committed to both nodes despite the fact that the TTL for the record expired on the node2. Shortly after the transaction is committed, node2 initiates another transaction and after verifying the TTL condition by its own clock, removes the object just inserted, naturally propagating the delete to the entire cluster. The record is thus short-lived (shorter than expected).

Event Interfaces

eXtremeDB applications can respond to data events like creating, updating or deleting database objects. For C applications the DDL event declarations cause applications to receive notification of the following database event types: adding a new object, deleting an object or all objects of a class, checkpoint events and updating an object or a specified field of an object. Events are specific to classes. In other words, a new event handler for class A will not receive a notification when an object of class B is added to the database.

The event declaration in the schema defines what events will trigger application notifications. How the application handles the events is determined at run-time by the event handlers. In C applications, events can be handled synchronously or asynchronously.

Asynchronous Event Handling

In asynchronous event handling, the application spawns a separate thread to handle each type of event. This event thread calls function mco_async_event_wait() to wait for the specified event. When the event occurs, the runtime releases the thread. Upon releasing the thread, the runtime continues normal processing, so the handler thread runs in parallel with other threads, until it completes its processing and again calls mco_async_event_wait().

There is a small window of possibility for another instance of the event to occur before the event handler has completed its task and calls mco_async_event_wait() again to wait on the event (events are not queued). This window can be minimized if the handler delegates the processing of the event to yet another thread, allowing the handler thread to immediately wait on the event again. If this risk of an unhandled event cannot be tolerated, either a synchronous event handler can be used, the application can maintain a separate table of unhandled events, or the mco_translog_play() function of the eXtremeDB Transaction Logging can be employed.

Asynchronous events are activated after the transaction commits. If, within the scope of a single transaction, several objects are added, or deleted, or several fields are updated which have event handlers waiting, all the handlers will be activated simultaneously.

A C application will define asynchronous event handlers in code like the following:

 
    /* Thread function that handles the <new> event. */
    void NewEventHandler( sample_task_t * descriptor )
    {
        mco_db_h db = descriptor->db_connection;
        while ( MCO_S_OK == mco_async_event_wait(db, MCO_EVENT_newEvent) ) 
        {
            /* Process the event. */
            ...
        }
    }
     
    /* Thread function that handles the <update> event. */
    void UpdateEventHandler( sample_task_t * descriptor )
    {
        mco_db_h db = descriptor->db_connection;
        while ( MCO_S_OK == mco_async_event_wait(db, MCO_EVENT_updateEvent) ) 
        {
            /* Process the event. */
            ...
        }
    }
     
    /* Thread function that handles the <delete> event. */
    void DeleteEventHandler( sample_task_t * descriptor )
    {
        mco_db_h db = descriptor->db_connection;
        while ( MCO_S_OK == mco_async_event_wait(db, MCO_EVENT_deleteEvent) ) 
        {
            /* Process the event. */
            ...
        }
    }
 
    /* Thread function that handles the <update> event. */
    void CheckpointEventHandler( sample_task_t * descriptor )
    {
        mco_db_h db = descriptor->db_connection;
 
        while ( MCO_S_OK == mco_async_event_wait(db, MCO_EVENT_checkpointEvent) ) 
        {
            /* Process the event. */
            ...
        }
    }
     

Then the C application will start the event handler threads and cause the main application thread to sleep for some milliseconds (100 is usually adequate) in order for the event handler threads to start listening (waiting). Then the main application thread will proceed with normal database processing. When terminating, the application will call mco_async_event_release_all() to release all events and then stop the event handler threads. (See the SDK sample samples/core/10-events/asynch for implementation details.)

Synchronous Event Handling

C applications can also respond to events synchronously. Synchronous event handlers are called within the context of the same thread that caused the event. Care should be taken in event handlers not to cause extraordinary delays because the handler has control of a transaction that, by definition, is a READ_WRITE transaction and thus could block access to the database. Specifically, the handler should not wait on an indeterminate external event such as user input.

A synchronous handler returns MCO_S_OK to indicate successful completion; any other value (one of MCO_S_* or MCO_E_* constants defined in mco.h, or a user-defined value) indicates success or failure with additional information. The runtime returns this value to the application, which can act on it accordingly (rolling back the transaction if necessary).

Synchronous handlers are registered by calling the generated mco_register_event_handler() function for each event handler. Any number of handlers can be registered for a single event type, but the order in which they are called cannot be predicted. At registration, the application can pass a user-defined parameter that will be, in turn, passed to the event handler when it is invoked. This parameter is a void pointer that can reference a simple scalar value or a complex data structure depending on the application requirements.

For new events, synchronous handlers are called by the classname_new()or classname_oid_new() function immediately after the object is instantiated (so the object handle is guaranteed to be valid). For checkpoint events, synchronous handlers are called by the classname_checkpoint() method immediately before or after the object is inserted into indexes – the application specifies whether the handler will be invoked before or after inserting into indexes through the handler registration interface. Checkpoint events are not fired by mco_trans_commit(), though this function also updates the index(es). For delete events, synchronous handlers are called by the classname_delete() method before the object is deleted (while the object handle is still valid).

Note that delete events are not invoked by the classname_delete_all() method. Only classname_delete() will invoke these events.

Update events can be defined for a class (i.e. all fields of that class) or for a specific field of a class by specifying the field name in the event declaration. As with checkpoint events, the application must specify through the handler registration interface whether the handler will be invoked before or after a field is updated. Update handlers are activated by any interface method that will cause a field’s contents to change, for example, classname_fieldname_put(), classname_fieldname_erase(). If the event handler is called before the update and the handler invokes classname_fieldname_get() on the field, it will retrieve the current value in the database. Conversely, if the event is called after the update, the handler will retrieve the value the application just put in the database. The user-defined parameter can be used to provide additional information to the handler such as the incoming value for a before-event handler, the old value for an after-event handler, or a vector offset for an erase operation.

Note that both synchronous and asynchronous events can be applied to any given event. When using shared memory, synchronous event handlers must belong to the same process that caused the event, or the results will be unpredictable (most likely, a crash, due to an attempt to call a function in another address space). In particular, do not register a synchronous event handler for class Alpha in process A if it is possible that process B will insert, update or delete Alpha objects. Use an asynchronous event handler, instead.

Note that for update events, a class-wide event cannot be combined with field update events for the same class.

The following code fragments illustrate event handling. Consider the following schema DDL definition for a class with event notifications:

 
    class dropped_call
    {
        uint4 trunk_id;
        ...
 
        autoid;
 
        event < trunk_id update > upd_trunk; // any name will do
        event < new > add_trunk;
        event < checkpoint > checkpoint_trunk;
        event < delete > del_trunk;
    };
     

The schema compiler will generate the following definitions in the interface header file:

 
    #define upd_trunk 15
    // 15 is only illustrative; the actual value is not important
    #define add_trunk 16
    #define checkpoint_trunk 17
    #define del_trunk 18
    typedef MCO_RET (*mco_upd_trunk_handler)( /*IN*/ mco_trans_h *t,
                /*IN*/ dropped_call *handle,
                /*IN*/ MCO_EVENT_TYPE et,
                /*INOUT*/ void *param );
 
    typedef MCO_RET (*mco_add_trunk_handler)( /*IN*/ mco_trans_h *t,
                /*IN*/ dropped_call *handle,
                /*IN*/ MCO_EVENT_TYPE et,
                /*INOUT*/ void *param );
 
    typedef MCO_RET (*mco_del_trunk_handler)( /*IN*/ mco_trans_h *t,
                /*IN*/ dropped_call *handle,
                /*IN*/ MCO_EVENT_TYPE et,
                /*INOUT*/ void *param );
 
    MCO_RET mco_register_upd_trunk_handler( /*IN*/ mco_upd_trunk_handler,
                /*IN*/ void *param,
                /*IN*/ MCO_HANDLING_ORDER when);
 
    MCO_RET mco_register_add_trunk_handler( /*IN*/ mco_add_trunk_handler,
                /*IN*/ void *param,
                /*IN*/ MCO_HANDLING_ORDER when);
                 
    MCO_RET mco_register_checkpoint_trunk_handler( /*IN*/ mco_checkpoint_trunk_handler,
                /*IN*/ void *param);
 
    MCO_RET mco_register_del_trunk_handler( /*IN*/ mco_del_trunk_handler,
                /*IN*/ void *param);
                 
    MCO_RET mco_unregister_upd_trunk_handler( /*IN*/ mco_upd_trunk_handler);
 
    MCO_RET mco_unregister_add_trunk_handler( /*IN*/ mco_add_trunk_handler);
 
    MCO_RET mco_unregister_checkpoint_trunk_handler( /*IN*/ mco_checkpoint_trunk_handler);
 
    MCO_RET mco_unregister_del_trunk_handler( /*IN*/ mco_del_trunk_handler);
     

To employ an asynchronous handler for one of the events above, the application would start a thread and, within the thread function, call:

 
    mco_async_event_wait( dbh, upd_trunk );
     

Where dbh is the database handle from the mco_db_connect() method and upd_trunk is the value defined in the generated interface file to reference the event of interest.

As previously mentioned, this thread will block (wait) until released by the runtime. It can be released either by an occurrence of the event, or the application can release it forcibly by calling one of the following:

 
    mco_async_event_release( dbh, upd_trunk );
     
    mco_async_event_release_all_( dbh );
     

A C event handler will know if it was released by an occurrence of the event or by a release() function by the return value of mco_async_event_wait(). MCO_S_OK means the event happened; MCO_S_EVENT_RELEASED means the event was released.

For the preceding class definition and its generated interfaces, the following code fragments illustrate synchronous event handling. First, the application must register its synchronous event handler functions with code like the following:

 
    int register_events(mco_db_h db)
    {
        MCO_RET rc;
        mco_trans_h t;
         
        mco_trans_start(db, MCO_READ_WRITE, MCO_TRANS_FOREGROUND, &t);
 
        mco_register_add_trunk_handler(t, &new_handler, (void*) 0);
        mco_register_checkpoint_trunk_handler(t, &checkpoint_handler, 
                            (void*) 0, MCO_BEFORE_UPDATE );
        mco_register_del_trunk_handler(t, &delete_handler, (void *) 0);
        mco_register_upd_trunk_handler( t, &update_handler1, 
                            (void *) 0, MCO_BEFORE_UPDATE );
        rc = mco_trans_commit(t);
        return rc;
    }
     

The bodies of the handler functions would look like the following:

     
    /* Handler for the "<new>" event. Reads the autoid and prints it out */
    MCO_RET new_handler( /*IN*/ mco_trans_h t, /*IN*/ dropped_call * obj,
                /*IN*/ MCO_EVENT_TYPE et, /*INOUT*/ void *param)
    {
        int8 u8;
        param = (int *)1;
         
        dropped_call_autoid_get( obj, &u8 );
        printf( "Event \"Object New\" : object (%ld,%ld) is created\n", u8.lo, u8.hi );
        return MCO_S_OK;
    }
     
    /* Handler for the "<checkpoint>" event. Reads the autoid and prints it out */
    MCO_RET checkpoint_handler( /*IN*/ mco_trans_h t, /*IN*/ dropped_call * obj,
                    /*IN*/ MCO_EVENT_TYPE et, /*INOUT*/ void *param)
    {
        int8 u8;
        param = (int *)1;
 
        dropped_call_autoid_get( obj, &u8 );
        printf( "Event \"Object Checkpoint\" : object (%ld,%ld) is about to be created\n", u8.lo, u8.hi );
        return MCO_S_OK;
    }
 
    /* Handler for the "<delete>" event. Note that the handler
    * is called before the current transaction is committed.
    * Therefore, the object is still valid; the object handle
    * is passed to the handler and is used to obtain the
    * autoid of the object. The event's handler return value
    * is passed into the "delete" function and is later
    * examined by the mco_trans_commit(). If the value is
    * anything but MCO_S_OK, the transaction is rolled back.
    * In this sample every other delete transaction is
    * committed.
    */
    MCO_RET delete_handler(	/*IN*/ mco_trans_h t, /*IN*/ dropped_call * obj,
                /*IN*/ MCO_EVENT_TYPE et, /*INOUT*/ void *user_param)
    {
        int8 u8;
 
        dropped_call_autoid_get( obj, &u8);
        printf( "Event \"Object Delete\": object (%ld,%ld) is being deleted...", u8.lo, u8.hi );
        return (((u8.lo + u8.hi) %2) ? 1: MCO_S_OK);
    }
 
    /* Handler for the "update" event. This handler is called
    * before the update transaction is committed - hence the
    * value of the field being changed is reported unchanged
    * yet.
    */
    MCO_RET update_handler1( /*IN*/ mco_trans_h t, /*IN*/ dropped_call * obj,
                /*IN*/ MCO_EVENT_TYPE et, /*INOUT*/ void *param)
    {
        uint4 u4;
        int8 u8;
 
        dropped_call_autoid_get( obj, &u8);
        dropped_call_trunk_id_get(obj, &u4);
        printf( "Event \"Object Update\" (before commit): object (%ld,%ld) value %d\n", 
            u8.lo, u8.hi, u4 );
        return MCO_S_OK;
    }
     

When the application is finished handling events, the events are unregistered by code like the following:

     
    int unregister_events(mco_db_h db)
    {
        MCO_RET rc;
        mco_trans_h t;
         
        mco_trans_start(db, MCO_READ_WRITE, MCO_TRANS_FOREGROUND, &t);
        mco_unregister_add_trunk_handler( t, & new_handler);
        mco_unregister_del_trunk_handler( t, & delete_handler);
        mco_unregister_update_handler( t, & update_handler1);
        rc = mco_trans_commit(t);
        return rc;
    }
     

(See the SDK sample samples/core/10-events/synch for further implementation details.)