Introduction to eXtremeDB Cluster

eXtremeDB Cluster is a distributed database implementation. Instances of applications can run on different nodes in a network, and all database modifications will be automatically replicated between all instances. For example, the following diagram illustrates a 5-node cluster.

 

All nodes in a cluster hold the same copy of the database. All nodes in a cluster are equal, i.e. eXtremeDB Cluster has no dedicated master or replica nodes (as with eXtremeDB High Availability). Each node can perform READ_WRITE and READ_ONLY operations. READ_ONLY transactions are local, i.e. they don't require network operations, and run as fast as with a standalone database.

Theory of operation

To update a database, a two-phase commit protocol is used. All operations are local, no network communications are involved, until the transaction is committed. The update scenario is as follows:

1. The thread running on the cluster node (called the initiator) performs a call to the C/C++ function mco_trans_commit(), or the Java/C# ClusterConnection.CommitTransaction() method, for a READ_WRITE transaction.

2. Internally the eXtremeDB Cluster implementation of the transaction commit invokes commit_phase1() for the initiator database copy. If commit_phase1() fails, the transaction is aborted and rolled back locally. (No network communication happens.)

3. If the commit_phase1() succeeds, the transaction is serialized and sent to all other nodes (called participants) in the Cluster.

4. Each participant opens a transaction, applies the serialized data from the initiator and performs commit_phase1().

5. The results of these operations (SUCCESS or FAIL) are sent back to the initiator.

6. The initiator gathers replies from all participants and decides how to complete the global transaction:

• If all participants report SUCCESS, the initiator completes the local transaction with commit_phase2() and sends CONFIRM messages to participants.
• If at least one participants replied with FAIL, the initiator rolls back the transaction and sends REJECT messages to the participants.
• Depending on the type of message (CONFIRM or REJECT), participants perform commit_phase2() or rollback() to complete the transaction.

Accordingly, database changes are either applied to all nodes, or rolled back. Since pessimistic locking is extremely expensive and subject to unpredictable failure-points when implemented in a network environment, eXtremeDB Cluster employs only the MVCC transaction manager.

The eXtremeDB Cluster implementation automatically controls the availability of nodes using timeouts and keep-alive messages. If a number of nodes become unavailable (i.e. there is no quorum), the processing of READ_WRITE transactions may be suspended.

The schema for a C/C++ application using a database that participate in the cluster must include the auto_oid declaration.

 

Terminology

The term Node Address refers to a character string containing the transport-layer-dependent address of a node. For example, a node address for the TCP/IP transport could be <hostname>:<port> or <IP-address>:<port>. (See examples below.)

The term Node Quorum Rank refers to an integer value that is used to determine whether enough interconnected nodes are online to work. If the sum of the qrank values for node A and all nodes connected to A is more than half of the sum of the qrank values for all cluster nodes (online and offline), then node A (and other nodes online) will continue to work.

It is helpful to consider two node quorum rank example scenarios:

1. All nodes have the same qrank value: in this case the cluster will work if more than half of the nodes are active and connected to each other. If the cluster is split exactly in half, the cluster will not work.

2. One node (call it main) has a qrank value of 1, all other nodes have a qrank value of 0: in this case the cluster of nodes connected to main will always work. If the main node dies, the cluster ceases to work.

Note that in any scenario, if the cluster nodes split into several disconnected parts, WRITE operations will be allowed only for one of these parts

 

Overview

eXtremeDB Cluster is built on the eXtremeDB High Availability technology. However, while eXtremeDB High Availability provides the synchronization / replication support in a master-replica scenario, eXtremeDB Cluster provides a simple interface for managing a peer-to-peer network of “equal” nodes.

As a new node is connected to the cluster, it is synchronized in hot mode, i.e. the newly connected node receives synchronization data and transactions from other nodes. Other currently connected cluster nodes can still perform READ_WRITE transactions during the startup synchronization of the new node. (The new node selects the active node with the largest number of completed transactions at the moment of connection to perform the synchronization. If all the cluster nodes are equally loaded with completed transactions, then the synchronizer node is selected randomly from the active cluster nodes.)