eXtremeDB Transaction Logging Applications

C/C++ applications using eXtremeDB Transaction Logging must first initialize the TL subsystem by calling mco_translog_init() before the database is opened via mco_db_open_dev(). In java and C# applications, the Database constructor initializes the TL subsystem when the database is instantiated.

After transaction logging is initialized, C/C++ applications connect to the database as in any eXtremeDB application by calling mco_db_connect() or mco_db_connect_ctx(), Java and C# applications simply instantiate a Connection.

At this point a save-point should be created by calling the mco_db_save() C/C++ function, (in Java and C# applications the Connection.SaveSnapshot() method) before logging is started. This establishes the starting point for all logged transactions to follow.

Java and C# applications can recover a database from a saved image and log file as illustrated in the following code fragments extracted from \samples\csharp\tl\TLogBasicCS:

     
        const string TL_LOG_FILE = "tlogbasic_tl_cs.log";
        const String DBIMAGE_FILE = "tlogbasicdb_cs.bak";
        Database.Parameters parameters = new Database.Parameters();
         
        if (File.Exists(DBIMAGE_FILE))
        {
            // set database image filename
            parameters.DatabaseSnapshotFilePath = DBIMAGE_FILE;
            loadedSnapshot = true;
        }
     
        // create Database object
        db = new Database(new ExtremedbWrapper(),
        Database.Mode.TransactionLoggingSupport);
     
        db.Open("tlogbasicdb", parameters, devs); // open database.
        if (loadedSnapshot)
        {
            // Apply stored log file to databas
            LogReader logr = new LogReader(con, TL_LOG_FILE);
            LogReader.LogInfo linfo = new LogReader.LogInfo();
            logr.Apply();
        }
        con.SaveSnapshot(DBIMAGE_FILE);
    }
     

In C/C++ applications, the following additional log file operations can be performed on log files:

A log file may be marked with a label at any point by calling the function mco_translog_label(), which provides the ability to restore the log up to that label. Also, a log file may be truncated and restarted by the function mco_translog_truncate(). Note that it is wise to create a database save-point prior to truncating the log file, otherwise the transactions will be unrecoverable. Further, it is wise to create the save-point to a different (new) file than the previous save-point. If you overwrite the existing save-point and the system crashes prior to completing the new save-point, there can be no recovery. So, for safety, the sequence should be:

1. create a new save-point

2. truncate the log file

3. delete the previous save-point file

 

If transaction logging is not synchronous (see MCO_TRANSLOG_SYNC_INSTANTLY below), a log file may be flushed to disk at any time by calling the function mco_translog_flush() to force immediate flushing of the file system buffers related to a log file. However, automatic flushing strategies (activated by elapsed time or by transaction count) can be specified when starting transaction logging (see the function mco_translog_start()) that eliminate the need for an application to explicitly flush the log file.

 

Information about the current log file (the current log size, the count of stored transactions and other information) may be gathered during the logging process by calling the function mco_translog_get_info(). Or, information can be obtained about a specific log file on persistent media (the ability to apply or append to this log, the log file size and other information) by calling mco_translog_query_info().

A database can be restored from compatible log files by calling the function mco_translog_apply(). Note that a proper save-point, i.e. a previously stored database image, should be restored via mco_db_load(), or a new empty database should be opened before applying a log file(s). Then, mco_translog_apply() is called to replay any transactions committed after that database image was saved. Before applying the log(s) to the database, the TL runtime first checks for dictionary compatibility, runtime modes and data size compatibility, and finally verifies that the current value of the database transaction count is the same as that stored in the log. If all is in order, mco_translog_apply() proceeds to apply all transactions stored in the log file, after which mco_translog_start()is called to begin transaction logging for the new database session.

Database Recovery

The sequence to recover a database from a save-point or an empty database as the starting point, and two or more consecutive log files is shown here:

Transaction Logging:

     
    char buff[64];
    mco_TL_start_data_t log_parms;
    log_parms.flags = 0;
    sprintf(buff, "transactions_%d.log", 0);
    CHECK(mco_translog_start (db, buff, &log_parms));
    log_parms.flags |= MCO_TRANSLOG_RESTART;
     
    for (i = 1; i < NPARTS; i++)
    {
        /* do database activity here */
        use_database();
        /* under some condition, e.g. log file size, restart log with new file */
        if( /* some condition */ )
        {
            sprintf(buff, "transactions_%d.log", i);
            CHECK(mco_translog_start (db, buff, &log_parms));
        }
    }
    CHECK(mco_translog_stop(db));
     

Database Recovery:

     
    for (i = 0; i < NPARTS; i++)
    {
        char buff[64];
        mco_TL_log_info_t info;
         
        /* load next log file */
        sprintf(buff, "transactions_%d.log", i);
        CHECK(mco_translog_query_info(db, buff, 0, 0, &info));
        ASSERT(info.transaction_apply_compat, MCO_YES);
        CHECK(mco_translog_apply(db, buff, MCO_TRANSLOG_ALL_LABELS));
    }
     

This sequence fully recovers the database. In the normal course of events, the database is restored from the last save-point image that was saved during an orderly shutdown, and there is no transaction log(s). In the case of an abnormal termination, transactions from the log file will be recovered but, because the application terminated without calling mco_translog_stop(), there is no proper end of file mark at the end of the (last) log file and mco_translog_apply() will return MCO_S_TL_INVDATA. In this case, you can call mco_translog_query_info() (which will also return MCO_S_TL_INVDATA) and the count of transactions actually restored by mco_translog_apply() will be returned in info.stored_trans_count. (Whether the last restored transaction is the last transaction actually performed before the application crashed will depend on the flushing policy employed, and, if any policy other than synchronous logging, then also luck).

When the application has finished, transaction logging is halted by calling mco_translog_stop(). Then normal application termination is affected by calling mco_db_disconnect() and mco_db_close(). The following pseudo-code snippet illustrates how an application might perform a “restore and proceed” scenario (see samples/tl/tlogbasic for a complete implementation:

     
    mco_runtime_start();
    mco_translog_init();
    ...
    mco_db_load();
    mco_db_connect();
    mco_translog_query_info(&info);
    if (info.stored_transactions > 0)
    {
        mco_translog_apply();
 
        /* Create initial save-point */
        mco_db_save();
    }
     
    mco_translog_start();
    while (run) /* Normal database usage */
    {
        ...
        if (some_condition ) /* periodically create save-point */
        { 
            mco_db_save();
 
            /* Truncate the log file */
            mco_translog_truncate();
        }
    }
    mco_db_save(); /* create the last save-point */
    mco_translog_stop();
     
    /* delete the log file here; it’s no longer needed */
    mco_db_disconnect();
     

Tips

There are three cases when using TL is warranted:

1. When the database consists of only transient (in-memory) objects or when it consists of both transient and persistent objects (a so-called ‘hybrid’ database) and you need durability for both the persistent and the transient data.

2. To export transactions to an external system (e.g. another DBMS), which we refer to as “Data Relay”.

3. To create a persistent queue of events, for more robust event handling than is possible through the ‘event’ DDL notation and associated synchronous and asynchronous event handlers.

Do not use TL to add persistence for persistent-only databases; eXtremeDB runtime has its own robust and efficient database recovery facilities for persistent databases. It is important not to confuse the use of eXtremeDB Transaction Logging APIs with the normal eXtremeDB runtime logging for persistent databases.

Due to the fact that applying a log to a database is slower than loading the database from a save-point (via function mco_db_load()), you should use TL function mco_translog_apply() only in case of an application crash or if a log needs to be applied up to specific label. So create database save-points regularly, then when the application finishes normally, a ‘restore and proceed’ scenario like that shown above can be applied.

 

DDL Requirements

The schema for a database that uses transaction logging must include the auto_oid declaration.

     
    declare auto_oid [ESTIMATED_NUMBER_OF_OBJECTS];
     

Whether the database actually uses transaction logging is determined at run-time. When the data definition is compiled for transaction logging, the eXtremeDB DDL compiler inserts an 8-byte unique identifier (called auto_OID, not to be confused with the field type autoid) into each object.