Chapter 20. MySQL Cluster

This chapter contains information about MySQL Cluster, which is a high-availability, high-redundancy version of MySQL adapted for the distributed computing environment, using the NDB (also known as NDBCLUSTER) storage engine to enable running several MySQL servers in a cluster.

This storage engine is available in MySQL 5.1 binary releases through MySQL 5.1.23, including RPMs compatible with most modern Linux distributions. Beginning with MySQL 5.1.24, standard MySQL server binaries built by MySQL no longer provide support for the NDBCLUSTER storage engine. Instead, MySQL Cluster users should upgrade to MySQL Cluster NDB 6.2.15 (or a later MySQL Cluster release) which includes binary releases for supported platforms, including RPMs that should work with most Linux distributions. MySQL Cluster users who build from source should be aware that, also beginning with MySQL 5.1.24, NDBCLUSTER sources in the standard MySQL 5.1 tree are no longer maintained; these users should upgrade using the MySQL Cluster NDB 6.2.15 (or later) source release.

Note

MySQL Cluster NDB 6.1, 6.2, and 6.3 were formerly known as “MySQL Cluster Carrier Grade Edition”. This designation was dropped beginning with MySQL Cluster NDB 6.2.15 and MySQL Cluster NDB 6.3.14.

This chapter contains information about MySQL Cluster in MySQL 5.1 mainline releases through MySQL 5.1.23, MySQL Cluster NDB 6.2 releases through 5.1.24-ndb-6.2.16, and MySQL Cluster NDB 6.3 releases through 5.1.24-ndb-6.3.16-beta. It also contains historical information about the MySQL Cluster NDB 6.1 series, although this series is no longer in active development.

Platforms supported. MySQL Cluster is currently available and supported on a number of platforms, including Linux, Solaris, Mac OS X, HP-UX, and other Unix-style operating systems on a variety of hardware. For exact levels of support available for on specific combinations of operating system versions, operating system distributions, and hardware platforms, please refer to the Cluster Supported Platforms list maintained by the MySQL Support Team on the MySQL AB Web site.

Important

MySQL Cluster is not currently supported on Microsoft Windows. We are working to make Cluster available on all operating systems supported by MySQL, including Windows, and will update the information provided here as this work continues.

Availability. MySQL Cluster NDB 6.2 binary and source packages are available for supported platforms from http://dev.mysql.com/downloads/cluster.

Note

Binary releases and RPMs are not available for MySQL Cluster NDB 6.2 prior to MySQL Cluster NDB 6.2.15.

MySQL Cluster NDB 6.3 is currently available only as source. Source tarballs for the latest MySQL Cluster NDB 6.3 release can be obtained from ftp://ftp.mysql.com/pub/mysql/download/cluster_telco/.

MySQL Cluster versioning. Starting with MySQL Cluster NDB 6.1 and 6.2, MySQL Cluster follows a somewhat different release pattern from the mainline MySQL 5.1 Cluster series of releases. Each MySQL Cluster release is identified by a two-part version string, which is displayed by SELECT VERSION() in the mysql client, as shown here:

shell> mysql
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 2
Server version: 5.1.24-ndb-6.2.16-debug Source distribution

Type 'help;' or '\h' for help. Type '\c' to clear the buffer.

mysql> SELECT VERSION();
+-------------------------+
| VERSION()               |
+-------------------------+
| 5.1.24-ndb-6.2.16-debug |
+-------------------------+
1 row in set (0.00 sec)

This version string is also displayed in the output from the ndb_mgm client's SHOW command:

ndb_mgm> SHOW
Connected to Management Server at: localhost:1186
Cluster Configuration
---------------------
[ndbd(NDB)]     2 node(s)
id=1    @10.0.10.6  (mysql-5.1.24 ndb-6.2.15, Nodegroup: 0, Master)
id=2    @10.0.10.8  (mysql-5.1.24 ndb-6.2.15, Nodegroup: 0)

[ndb_mgmd(MGM)] 1 node(s)
id=3    @10.0.10.2  (mysql-5.1.24 ndb-6.2.15)

[mysqld(API)]   2 node(s)
id=4    @10.0.10.10  (mysql-5.1.24 ndb-6.2.15)
id=5 (not connected, accepting connect from any host)

The version string identifies the mainline MySQL version from which the MySQL Cluster release was branched and the version of the NDB storage engine used. For example, the full version string for MySQL Cluster NDB 6.1.15 (the first MySQL Cluster NDB 6.2 binary release, as shown in the preceding examples) is mysql-5.1.24 ndb-6.2.15. From this we can determine the following information:

The portion of the version string preceding ndb is the MySQL Server base version. This means that MySQL Cluster NDB 6.2.15 derives from the MySQL 5.1.24, and contains all feature enhancement and bugfixes from MySQL 5.1 up to and including MySQL 5.1.24.
The portion of the version string following ndb- represents the version number of the NDB (or NDBCLUSTER) storage engine. MySQL Cluster NDB 6.2.15 uses version 6.2.15 of the NDB storage engine.

MySQL Cluster development source trees. MySQL Cluster development trees can also be accessed via https://code.launchpad.net/~mysql/:

The MySQL Cluster development sources maintained at https://code.launchpad.net/~mysql/ are GPL-licensed. For information about obtaining MySQL sources using Bazaar and building them yourself, see Section 2.9.3, “Installing from the Development Source Tree”.

Currently, MySQL Cluster NDB 6.2 and MySQL Cluster NDB 6.3 are both under active development, with the 6.2 series now available in source and binary format and 6.3 as source only. MySQL Cluster NDB 6.4 is beginning to be developed internally and is intended for testing purposes. MySQL Cluster NDB 6.1 is no longer in active development. For an overview of major features added in MySQL Cluster, see Section 20.15, “MySQL Cluster Development Roadmap”.

This chapter represents a work in progress, and its contents are subject to revision as MySQL Cluster continues to evolve. Additional information regarding MySQL Cluster can be found on the MySQL AB Web site at http://www.mysql.com/products/cluster/.

Additional resources. More information may be found in the following places:

Answers to some commonly asked questions about Cluster may be found in the Section A.10, “MySQL 5.1 FAQ — MySQL Cluster”.
The MySQL Cluster mailing list: http://lists.mysql.com/cluster.
The MySQL Cluster Forum: http://forums.mysql.com/list.php?25.
Many MySQL Cluster users and some of the MySQL Cluster developers blog about their experiences with Cluster, and make feeds of these available through PlanetMySQL.
If you are new to MySQL Cluster, you may find our Developer Zone article How to set up a MySQL Cluster for two servers to be helpful.

20.1. MySQL Cluster Overview

20.1.1. MySQL Cluster Core Concepts
20.1.2. MySQL Cluster Nodes, Node Groups, Replicas, and Partitions

MySQL Cluster is a technology that enables clustering of in-memory databases in a shared-nothing system. The shared-nothing architecture allows the system to work with very inexpensive hardware, and with a minimum of specific requirements for hardware or software.

MySQL Cluster is designed not to have any single point of failure. For this reason, each component is expected to have its own memory and disk, and the use of shared storage mechanisms such as network shares, network filesystems, and SANs is not recommended or supported.

MySQL Cluster integrates the standard MySQL server with an in-memory clustered storage engine called NDB. In our documentation, the term NDB refers to the part of the setup that is specific to the storage engine, whereas “MySQL Cluster” refers to the combination of MySQL and the NDB storage engine.

A MySQL Cluster consists of a set of computers, each running a one or more processes which may include a MySQL server, a data node, a management server, and (possibly) a specialized data access programs. The relationship of these components in a cluster is shown here:

All these programs work together to form a MySQL Cluster. When data is stored in the NDBCLUSTER storage engine, the tables are stored in the data nodes. Such tables are directly accessible from all other MySQL servers in the cluster. Thus, in a payroll application storing data in a cluster, if one application updates the salary of an employee, all other MySQL servers that query this data can see this change immediately.

The data stored in the data nodes for MySQL Cluster can be mirrored; the cluster can handle failures of individual data nodes with no other impact than that a small number of transactions are aborted due to losing the transaction state. Because transactional applications are expected to handle transaction failure, this should not be a source of problems.

20.1.1. MySQL Cluster Core Concepts

NDB is an in-memory storage engine offering high-availability and data-persistence features.

The NDB storage engine can be configured with a range of failover and load-balancing options, but it is easiest to start with the storage engine at the cluster level. MySQL Cluster's NDB storage engine contains a complete set of data, dependent only on other data within the cluster itself.

The cluster portion of MySQL Cluster is currently configured independently of the MySQL servers. In a MySQL Cluster, each part of the cluster is considered to be a node.

Note

In many contexts, the term “node” is used to indicate a computer, but when discussing MySQL Cluster it means a process. It is possible to run any number of nodes on a single computer, for which we use the term cluster host.

(However, it should be noted MySQL does not currently support the use of multiple data nodes on a single computer in a production setting. See Section 20.14.10, “Limitations Relating to Multiple Cluster Nodes”.)

There are three types of cluster nodes, and in a minimal MySQL Cluster configuration, there will be at least three nodes, one of each of these types:

Management node (MGM node): The role of this type of node is to manage the other nodes within the MySQL Cluster, performing such functions as providing configuration data, starting and stopping nodes, running backup, and so forth. Because this node type manages the configuration of the other nodes, a node of this type should be started first, before any other node. An MGM node is started with the command ndb_mgmd.
Data node: This type of node stores cluster data. There are as many data nodes as there are replicas, times the number of fragments. For example, with two replicas, each having two fragments, you will need four data nodes. It is not necessary to have more than one replica. A data node is started with the command ndbd.
SQL node: This is a node that accesses the cluster data. In the case of MySQL Cluster, an SQL node is a traditional MySQL server that uses the NDB Cluster storage engine. An SQL node is typically started with the command mysqld --ndbcluster or by using mysqld with the ndbcluster option added to my.cnf.
An SQL node is actually just a specialised type of API node, which designates any application which accesses Cluster data. One example of an API node is the ndb_restore utility that is used to restore a cluster backup. It is possible to write such applications using the NDB API.

Important

It is not realistic to expect to employ a three-node setup in a production environment. Such a configuration provides no redundancy; in order to benefit from MySQL Cluster's high-availability features, you must use multiple data and SQL nodes. The use of multiple management nodes is also highly recommended.

For a brief introduction to the relationships between nodes, node groups, replicas, and partitions in MySQL Cluster, see Section 20.1.2, “MySQL Cluster Nodes, Node Groups, Replicas, and Partitions”.

Configuration of a cluster involves configuring each individual node in the cluster and setting up individual communication links between nodes. MySQL Cluster is currently designed with the intention that data nodes are homogeneous in terms of processor power, memory space, and bandwidth. In addition, to provide a single point of configuration, all configuration data for the cluster as a whole is located in one configuration file.

The management server (MGM node) manages the cluster configuration file and the cluster log. Each node in the cluster retrieves the configuration data from the management server, and so requires a way to determine where the management server resides. When interesting events occur in the data nodes, the nodes transfer information about these events to the management server, which then writes the information to the cluster log.

In addition, there can be any number of cluster client processes or applications. These are of two types:

Standard MySQL clients. These are no different for MySQL Cluster than they are for standard (non-Cluster) MySQL. In other words, MySQL Cluster can be accessed from existing MySQL applications written in PHP, Perl, C, C++, Java, Python, Ruby, and so on.
Management clients. These clients connect to the management server and provide commands for starting and stopping nodes gracefully, starting and stopping message tracing (debug versions only), showing node versions and status, starting and stopping backups, and so on.

20.1.2. MySQL Cluster Nodes, Node Groups, Replicas, and Partitions

This section discusses the manner in which MySQL Cluster divides and duplicates data for storage.

Central to an understanding of this topic are the following concepts, listed here with brief definitions:

(Data) Node. An ndbd process, which stores a replica —that is, a copy of the partition (see below) assigned to the node group of which the node is a member.
Each data node should be located on a separate computer. While it is also possible to host multiple ndbd processes on a single computer, such a configuration is not supported.
It is common for the terms “node” and “data node” to be used interchangeably when referring to an ndbd process; where mentioned, management (MGM) nodes (ndb_mgmd processes) and SQL nodes (mysqld processes) are specified as such in this discussion.
Node Group. A node group consists of one or more nodes, and stores partitions, or sets of replicas (see next item).
Note
All node groups in a cluster must have the same number of nodes.
Partition. This is a portion of the data stored by the cluster. There are as many cluster partitions as nodes participating in the cluster. Each node is responsible for keeping at least one copy of any partitions assigned to it (that is, at least one replica) available to the cluster.
A replica belongs entirely to a single node; a node can (and usually does) store several replicas.
Replica. This is a copy of a cluster partition. Each node in a node group stores a replica. Also sometimes known as a partition replica. The number of replicas is equal to the number of nodes per node group.

The following diagram illustrates a MySQL Cluster with four data nodes, arranged in two node groups of two nodes each; nodes 1 and 2 belong to node group 0, and nodes 3 and 4 belong to node group 1. Note that only data (ndbd) nodes are shown here; although a working cluster requires an ndb_mgm process for cluster management and at least one SQL node to access the data stored by the cluster, these have been omitted in the figure for clarity.

A MySQL Cluster, with 2 node groups having 2
nodes each

The data stored by the cluster is divided into four partitions, numbered 0, 1, 2, and 3. Each partition is stored — in multiple copies — on the same node group. Partitions are stored on alternate node groups:

Partition 0 is stored on node group 0; a primary replica (primary copy) is stored on node 1, and a backup replica (backup copy of the partition) is stored on node 2.
Partition 1 is stored on the other node group (node group 1); this partition's primary replica is on node 3, and its backup replica is on node 4.
Partition 2 is stored on node group 0. However, the placing of its two replicas is reversed from that of Partition 0; for Partition 2, the primary replica is stored on node 2, and the backup on node 1.
Partition 3 is stored on node group 1, and the placement of its two replicas are reversed from those of partition 1. That is, its primary replica is located on node 4, with the backup on node 3.

What this means regarding the continued operation of a MySQL Cluster is this: so long as each node group participating in the cluster has at least one node operating, the cluster has a complete copy of all data and remains viable. This is illustrated in the next diagram.

Nodes required to keep a 2x2 cluster
viable

In this example, where the cluster consists of two node groups of two nodes each, any combination of at least one node in node group 0 and at least one node in node group 1 is sufficient to keep the cluster “alive” (indicated by arrows in the diagram). However, if both nodes from either node group fail, the remaining two nodes are not sufficient (shown by the arrows marked out with an X); in either case, the cluster has lost an entire partition and so can no longer provide access to a complete set of all cluster data.

20.2. Simple Multi-Computer How-To

20.2.1. Hardware, Software, and Networking
20.2.2. Multi-Computer Installation
20.2.3. Multi-Computer Configuration
20.2.4. Initial Startup
20.2.5. Loading Sample Data and Performing Queries
20.2.6. Safe Shutdown and Restart

This section is a “How-To” that describes the basics for how to plan, install, configure, and run a MySQL Cluster. Whereas the examples in Section 20.3, “MySQL Cluster Configuration” provide more in-depth information on a variety of clustering options and configuration, the result of following the guidelines and procedures outlined here should be a usable MySQL Cluster which meets the minimum requirements for availability and safeguarding of data.

This section covers hardware and software requirements; networking issues; installation of MySQL Cluster; configuration issues; starting, stopping, and restarting the cluster; loading of a sample database; and performing queries.

Basic assumptions. This How-To makes the following assumptions:

The cluster is to be set up with four nodes, each on a separate host, and each with a fixed network address on a typical Ethernet network as shown here:
Node IP Address
Management (MGM) node 192.168.0.10
MySQL server (SQL) node 192.168.0.20
Data (NDBD) node "A" 192.168.0.30
Data (NDBD) node "B" 192.168.0.40
This may be made clearer in the following diagram:
In the interest of simplicity (and reliability), this How-To uses only numeric IP addresses. However, if DNS resolution is available on your network, it is possible to use hostnames in lieu of IP addresses in configuring Cluster. Alternatively, you can use the /etc/hosts file or your operating system's equivalent for providing a means to do host lookup if such is available.
Note
A common problem when trying to use hostnames for Cluster nodes arises because of the way in which some operating systems (including some Linux distributions) set up the system's own hostname in the /etc/hosts during installation. Consider two machines with the hostnames ndb1 and ndb2, both in the cluster network domain. Red Hat Linux (including some derivatives such as CentOS and Fedora) places the following entries in these machines' /etc/hosts files:
```
#  ndb1 /etc/hosts:
127.0.0.1   ndb1.cluster ndb1 localhost.localdomain localhost
```
```
#  ndb2 /etc/hosts:
127.0.0.1   ndb2.cluster ndb2 localhost.localdomain localhost
```
SUSE Linux (including OpenSUSE) places these entries in the machines' /etc/hosts files:
```
#  ndb1 /etc/hosts:
127.0.0.1       localhost
127.0.0.2       ndb1.cluster ndb1
```
```
#  ndb2 /etc/hosts:
127.0.0.1       localhost
127.0.0.2       ndb2.cluster ndb2
```
In both instances, ndb1 routes ndb1.cluster to a loopback IP address, but gets a public IP address from DNS for ndb2.cluster, while ndb2 routes ndb2.cluster to a loopback address and obtains a public address for ndb1.cluster. The result is that each data node connects to the management server, but cannot tell when any other data nodes have connected, and so the data nodes appear to hang while starting.
You should also be aware that you cannot mix localhost and other hostnames or IP addresses in config.ini. For these reasons, the solution in such cases (other than to use IP addresses for all config.ini HostName entries) is to remove the fully qualified hostnames from /etc/hosts and use these in config.ini for all cluster hosts.
Each host in our scenario is an Intel-based desktop PC running a common, generic Linux distribution installed to disk in a standard configuration, and running no unnecessary services. The core OS with standard TCP/IP networking capabilities should be sufficient. Also for the sake of simplicity, we also assume that the filesystems on all hosts are set up identically. In the event that they are not, you will need to adapt these instructions accordingly.
Standard 100 Mbps or 1 gigabit Ethernet cards are installed on each machine, along with the proper drivers for the cards, and that all four hosts are connected via a standard-issue Ethernet networking appliance such as a switch. (All machines should use network cards with the same throughout. That is, all four machines in the cluster should have 100 Mbps cards or all four machines should have 1 Gbps cards.) MySQL Cluster will work in a 100 Mbps network; however, gigabit Ethernet will provide better performance.
Note that MySQL Cluster is not intended for use in a network for which throughput is less than 100 Mbps. For this reason (among others), attempting to run a MySQL Cluster over a public network such as the Internet is not likely to be successful, and is not recommended.
For our sample data, we will use the world database which is available for download from the MySQL AB Web site. As this database takes up a relatively small amount of space, we assume that each machine has 256MB RAM, which should be sufficient for running the operating system, host NDB process, and (for the data nodes) for storing the database.

Although we refer to a Linux operating system in this How-To, the instructions and procedures that we provide here should be easily adaptable to other supported operating systems. We also assume that you already know how to perform a minimal installation and configuration of the operating system with networking capability, or that you are able to obtain assistance in this elsewhere if needed.

We discuss MySQL Cluster hardware, software, and networking requirements in somewhat greater detail in the next section. (See Section 20.2.1, “Hardware, Software, and Networking”.)

20.2.1. Hardware, Software, and Networking

One of the strengths of MySQL Cluster is that it can be run on commodity hardware and has no unusual requirements in this regard, other than for large amounts of RAM, due to the fact that all live data storage is done in memory. (It is possible to reduce this requirement using Disk Data tables — see Section 20.12, “MySQL Cluster Disk Data Tables”, for more information about these.) Naturally, multiple and faster CPUs can enhance performance. Memory requirements for other Cluster processes are relatively small.

The software requirements for Cluster are also modest. Host operating systems do not require any unusual modules, services, applications, or configuration to support MySQL Cluster. For supported operating systems, a standard installation should be sufficient. The MySQL software requirements are simple: all that is needed is a production release of MySQL 5.1.24-ndb-6.2.16 or 5.1.24-ndb-6.3.16-beta to have Cluster support. It is not necessary to compile MySQL yourself merely to be able to use Cluster. In this How-To, we assume that you are using the server binary appropriate to your platform, available via the MySQL Cluster software downloads page at http://dev.mysql.com/downloads/cluster.

For communication between nodes, Cluster supports TCP/IP networking in any standard topology, and the minimum expected for each host is a standard 100 Mbps Ethernet card, plus a switch, hub, or router to provide network connectivity for the cluster as a whole. We strongly recommend that a MySQL Cluster be run on its own subnet which is not shared with non-Cluster machines for the following reasons:

Security. Communications between Cluster nodes are not encrypted or shielded in any way. The only means of protecting transmissions within a MySQL Cluster is to run your Cluster on a protected network. If you intend to use MySQL Cluster for Web applications, the cluster should definitely reside behind your firewall and not in your network's De-Militarized Zone (DMZ) or elsewhere.
See Section 20.8.1, “MySQL Cluster Security and Networking Issues”, for more information.
Efficiency. Setting up a MySQL Cluster on a private or protected network allows the cluster to make exclusive use of bandwidth between cluster hosts. Using a separate switch for your MySQL Cluster not only helps protect against unauthorized access to Cluster data, it also ensures that Cluster nodes are shielded from interference caused by transmissions between other computers on the network. For enhanced reliability, you can use dual switches and dual cards to remove the network as a single point of failure; many device drivers support failover for such communication links.

It is also possible to use the high-speed Scalable Coherent Interface (SCI) with MySQL Cluster, but this is not a requirement. See Section 20.13, “Using High-Speed Interconnects with MySQL Cluster”, for more about this protocol and its use with MySQL Cluster.

20.2.2. Multi-Computer Installation

Each MySQL Cluster host computer running an SQL node must have installed on it a MySQL binary. For management nodes and data nodes, it is not necessary to install the MySQL server binary, but management nodes require the management server daemon (ndb_mgmd) and data nodes require the data node daemon (ndbd). It is also a good idea to install the management client (ndb_mgm) on the management server host. This section covers the steps necessary to install the correct binaries for each type of Cluster node.

MySQL AB provides precompiled binaries that support Cluster, and there is generally no need to compile these yourself. However, we also include information relating to installing a MySQL Cluster after building MySQL from source. For setting up a cluster using MySQL's binaries, the first step in the installation process for each cluster host is to download the file mysql-5.1.24-ndb-6.2.16-linux-i686-glibc23.tar.gz from the MySQL downloads area. We assume that you have placed it in each machine's /var/tmp directory. (If you do require a custom binary, see Section 2.9.3, “Installing from the Development Source Tree”.)

Note

After completing the installation, do not yet start any of the binaries. We show you how to do so following the configuration of all nodes.

Data and SQL Node Installation — .tar.gz Binary. On each of the machines designated to host data or SQL nodes, perform the following steps as the system root user:

Check your /etc/passwd and /etc/group files (or use whatever tools are provided by your operating system for managing users and groups) to see whether there is already a mysql group and mysql user on the system. Some OS distributions create these as part of the operating system installation process. If they are not already present, create a new mysql user group, and then add a mysql user to this group:
```
shell> groupadd mysql
shell> useradd -g mysql mysql
```
The syntax for useradd and groupadd may differ slightly on different versions of Unix, or they may have different names such as adduser and addgroup.
Change location to the directory containing the downloaded file, unpack the archive, and create a symlink to the mysql directory named mysql. Note that the actual file and directory names will vary according to the MySQL version number.
```
shell> cd /var/tmp
shell> tar -C /usr/local -xzvf mysql-5.1.24-ndb-6.2.16-linux-i686-glibc23.tar.gz
shell> ln -s /usr/local/mysql-5.1.24-ndb-6.2.16-linux-i686-glibc23.tar.gz /usr/local/mysql
```
Change location to the mysql directory and run the supplied script for creating the system databases:
```
shell> cd mysql
shell> scripts/mysql_install_db --user=mysql
```
Set the necessary permissions for the MySQL server and data directories:
```
shell> chown -R root .
shell> chown -R mysql data
shell> chgrp -R mysql .
```
Note that the data directory on each machine hosting a data node is /usr/local/mysql/data. This piece of information is essential when configuring the management node. (See Section 20.2.3, “Multi-Computer Configuration”.)
Copy the MySQL startup script to the appropriate directory, make it executable, and set it to start when the operating system is booted up:
```
shell> cp support-files/mysql.server /etc/rc.d/init.d/
shell> chmod +x /etc/rc.d/init.d/mysql.server
shell> chkconfig --add mysql.server
```
(The startup scripts directory may vary depending on your operating system and version — for example, in some Linux distributions, it is /etc/init.d.)
Here we use Red Hat's chkconfig for creating links to the startup scripts; use whatever means is appropriate for this purpose on your operating system and distribution, such as update-rc.d on Debian.

Remember that the preceding steps must be performed separately on each machine where an SQL node is to reside.

SQL node installation — building from source. If you compile MySQL with clustering support (for example, by using the BUILD/compile-platform_name-max script appropriate to your platform), and perform the default installation (using make install as the root user), mysqld is placed in /usr/local/mysql/bin. Follow the steps given in Section 2.9, “MySQL Installation Using a Source Distribution” to make mysqld ready for use. If you want to run multiple SQL nodes, you can use a copy of the same mysqld executable and its associated support files on several machines. The easiest way to do this is to copy the entire /usr/local/mysql directory and all directories and files contained within it to the other SQL node host or hosts, then repeat the steps from Section 2.9, “MySQL Installation Using a Source Distribution” on each machine. If you configure the build with a non-default --prefix, you need to adjust the directory accordingly.

Data node installation — building from source. The only executable required on a data node host is ndbd (mysqld, for example, does not have to be present on the host machine). By default when doing a source build, this file is placed in the directory /usr/local/mysql/libexec. For installing on multiple data node hosts, only ndbd need be copied to the other host machine or machines. (This assumes that all data node hosts use the same architecture and operating system; otherwise you may need to compile separately for each different platform.) ndbd need not be in any particular location on the host's filesystem, as long as the location is known.

Management node installation — .tar.gz binary. Installation of the management node does not require the mysqld binary. Only the binary for the management server is required, which can be found in the downloaded archive. You most likely want to install the management client as well; this can also be found in the .tar.gz archive. Again, we assume that you have placed this archive in /var/tmp.

As system root (that is, after using sudo, su root, or your system's equivalent for temporarily assuming the system administrator account's privileges), perform the following steps to install ndb_mgmd and ndb_mgm on the Cluster management node host:

Change location to the /var/tmp directory, and extract the ndb_mgm and ndb_mgmd from the archive into a suitable directory such as /usr/local/bin:
```
shell> cd /var/tmp
shell> tar -zxvf mysql-5.1.24-ndb-6.2.16-linux-i686-glibc23.tar.gz
shell> cd mysql-5.1.24-ndb-6.2.16-linux-i686-glibc23
shell> cp /bin/ndb_mgm* /usr/local/bin
```
(You can safely delete the directory created by unpacking the downloaded archive, and the files it contains, from /var/tmp once ndb_mgm and ndb_mgmd have been copied to the executables directory.)
Change location to the directory into which you copied the files, and then make both of them executable:
```
shell> cd /usr/local/bin
shell> chmod +x ndb_mgm*
```

Management node installation — building from source. When building from source and running the default make install, the management server binary (ndb_mgmd) is placed in /usr/local/mysql/libexec, while the management client binary (ndb_mgm) can be found in /usr/local/mysql/bin. Only ndb_mgmd is required to be present on a management node host; however, it is also a good idea to have ndb_mgm present on the same host machine. Neither of these executables requires a specific location on the host machine's filesystem.

In Section 20.2.3, “Multi-Computer Configuration”, we create configuration files for all of the nodes in our example Cluster.

20.2.3. Multi-Computer Configuration

For our four-node, four-host MySQL Cluster, it is necessary to write four configuration files, one per node host.

Each data node or SQL node requires a my.cnf file that provides two pieces of information: a connectstring that tells the node where to find the MGM node, and a line telling the MySQL server on this host (the machine hosting the data node) to run in NDB mode.
For more information on connectstrings, see Section 20.3.4.2, “The Cluster Connectstring”.
The management node needs a config.ini file telling it how many replicas to maintain, how much memory to allocate for data and indexes on each data node, where to find the data nodes, where to save data to disk on each data node, and where to find any SQL nodes.

Configuring the Storage and SQL Nodes

The my.cnf file needed for the data nodes is fairly simple. The configuration file should be located in the /etc directory and can be edited using any text editor. (Create the file if it does not exist.) For example:

shell> vi /etc/my.cnf

We show vi being used here to create the file, but any text editor should work just as well.

For each data node and SQL node in our example setup, my.cnf should look like this:

# Options for mysqld process:
[mysqld]                        
ndbcluster                      # run NDB storage engine
ndb-connectstring=192.168.0.10  # location of management server

# Options for ndbd process:
[mysql_cluster]                 
ndb-connectstring=192.168.0.10  # location of management server

After entering the preceding information, save this file and exit the text editor. Do this for the machines hosting data node “A”, data node “B”, and the SQL node.

Important

Once you have started a mysqld process with the ndbcluster and ndb-connectstring parameters in the [mysqld] in the my.cnf file as shown previously, you cannot execute any CREATE TABLE or ALTER TABLE statements without having actually started the cluster. Otherwise, these statements will fail with an error. This is by design.

Configuring the management node. The first step in configuring the MGM node is to create the directory in which the configuration file can be found and then to create the file itself. For example (running as root):

shell> mkdir /var/lib/mysql-cluster
shell> cd /var/lib/mysql-cluster
shell> vi config.ini

For our representative setup, the config.ini file should read as follows:

# Options affecting ndbd processes on all data nodes:
[ndbd default]    
NoOfReplicas=2    # Number of replicas
DataMemory=80M    # How much memory to allocate for data storage
IndexMemory=18M   # How much memory to allocate for index storage
                  # For DataMemory and IndexMemory, we have used the
                  # default values. Since the "world" database takes up
                  # only about 500KB, this should be more than enough for
                  # this example Cluster setup.

# TCP/IP options:
[tcp default]     
portnumber=2202   # This the default; however, you can use any port that is free 
                  # for all the hosts in the cluster
                  # Note: It is recommended that you do not specify the port 
                  # number at all and allow the default value to be used instead

# Management process options:
[ndb_mgmd]                      
hostname=192.168.0.10           # Hostname or IP address of MGM node
datadir=/var/lib/mysql-cluster  # Directory for MGM node log files

# Options for data node "A":
[ndbd]                          
                                # (one [ndbd] section per data node)
hostname=192.168.0.30           # Hostname or IP address
datadir=/usr/local/mysql/data   # Directory for this data node's data files

# Options for data node "B":
[ndbd]                          
hostname=192.168.0.40           # Hostname or IP address
datadir=/usr/local/mysql/data   # Directory for this data node's data files

# SQL node options:
[mysqld]                        
hostname=192.168.0.20           # Hostname or IP address
                                # (additional mysqld connections can be
                                # specified for this node for various
                                # purposes such as running ndb_restore)

Note

The world database can be downloaded from http://dev.mysql.com/doc/, where it can be found listed under “Examples”.

After all the configuration files have been created and these minimal options have been specified, you are ready to proceed with starting the cluster and verifying that all processes are running. We discuss how this is done in Section 20.2.4, “Initial Startup”.

For more detailed information about the available MySQL Cluster configuration parameters and their uses, see Section 20.3.4, “Configuration File”, and Section 20.3, “MySQL Cluster Configuration”. For configuration of MySQL Cluster as relates to making backups, see Section 20.9.4, “Configuration for Cluster Backup”.

Note

The default port for Cluster management nodes is 1186; the default port for data nodes is 2202. However, the cluster can automatically allocate ports for data nodes from those that are already free.

20.2.4. Initial Startup

Starting the cluster is not very difficult after it has been configured. Each cluster node process must be started separately, and on the host where it resides. The management node should be started first, followed by the data nodes, and then finally by any SQL nodes:

On the management host, issue the following command from the system shell to start the management node process:
```
shell> ndb_mgmd -f /var/lib/mysql-cluster/config.ini
```
Note
ndb_mgmd must be told where to find its configuration file, using the -f or --config-file option. (See Section 20.6.3, “ndb_mgmd — The Management Server Process”, for details.)
For additional options which can be used with ndb_mgmd, see Section 20.6.5, “Command Options for MySQL Cluster Processes”.
On each of the data node hosts, run this command to start the ndbd process:
```
shell> ndbd
```
If you used RPM files to install MySQL on the cluster host where the SQL node is to reside, you can (and should) use the supplied startup script to start the MySQL server process on the SQL node.

If all has gone well, and the cluster has been set up correctly, the cluster should now be operational. You can test this by invoking the ndb_mgm management node client. The output should look like that shown here, although you might see some slight differences in the output depending upon the exact version of MySQL that you are using:

shell> ndb_mgm
-- NDB Cluster -- Management Client --
ndb_mgm> SHOW
Connected to Management Server at: localhost:1186
Cluster Configuration
---------------------
[ndbd(NDB)]     2 node(s)
id=2    @192.168.0.30  (Version: 5.1.24-ndb-6.2.16, Nodegroup: 0, Master)
id=3    @192.168.0.40  (Version: 5.1.24-ndb-6.2.16, Nodegroup: 0)

[ndb_mgmd(MGM)] 1 node(s)
id=1    @192.168.0.10  (Version: 5.1.24-ndb-6.2.16)

[mysqld(SQL)]   1 node(s)
id=4   (Version: 5.1.24-ndb-6.2.16)

Note

The SQL node is referenced here as [mysqld(API)]. This is perfectly normal, and reflects the fact that the mysqld process is acting as a cluster API node.

You should now be ready to work with databases, tables, and data in MySQL Cluster. See Section 20.2.5, “Loading Sample Data and Performing Queries”, for a brief discussion.

20.2.5. Loading Sample Data and Performing Queries

Working with data in MySQL Cluster is not much different from doing so in MySQL without Cluster. There are two points to keep in mind:

For a table to be replicated in the cluster, it must use the NDBCLUSTER storage engine. To specify this, use the ENGINE=NDB or ENGINE=NDBCLUSTER table option. You can add this option when creating the table:
```
CREATE TABLE tbl_name (col_name column_definitions) ENGINE=NDBCLUSTER;
```
Alternatively, for an existing table that uses a different storage engine, use ALTER TABLE to change the table to use NDBCLUSTER:
```
ALTER TABLE tbl_name ENGINE=NDBCLUSTER;
```
Each NDB table must have a primary key. If no primary key is defined by the user when a table is created, the NDBCLUSTER storage engine automatically generates a hidden one.
Note
This hidden key takes up space just as does any other table index. It is not uncommon to encounter problems due to insufficient memory for accommodating these automatically created indexes.)

If you are importing tables from an existing database using the output of mysqldump, you can open the SQL script in a text editor and add the ENGINE option to any table creation statements, or replace any existing ENGINE (or TYPE) options. Suppose that you have the world sample database on another MySQL server that does not support MySQL Cluster, and you want to export the City table:

shell> mysqldump --add-drop-table world City > city_table.sql

The resulting city_table.sql file will contain this table creation statement (and the INSERT statements necessary to import the table data):

DROP TABLE IF EXISTS `City`;
CREATE TABLE `City` (
  `ID` int(11) NOT NULL auto_increment,
  `Name` char(35) NOT NULL default '',
  `CountryCode` char(3) NOT NULL default '',
  `District` char(20) NOT NULL default '',
  `Population` int(11) NOT NULL default '0',
  PRIMARY KEY  (`ID`)
) ENGINE=MyISAM DEFAULT CHARSET=latin1;

INSERT INTO `City` VALUES (1,'Kabul','AFG','Kabol',1780000);
INSERT INTO `City` VALUES (2,'Qandahar','AFG','Qandahar',237500);
INSERT INTO `City` VALUES (3,'Herat','AFG','Herat',186800);
(remaining INSERT statements omitted)

You need to make sure that MySQL uses the NDB storage engine for this table. There are two ways that this can be accomplished. One of these is to modify the table definition before importing it into the Cluster database. Using the City table as an example, modify the ENGINE option of the definition as follows:

DROP TABLE IF EXISTS `City`;
CREATE TABLE `City` (
  `ID` int(11) NOT NULL auto_increment,
  `Name` char(35) NOT NULL default '',
  `CountryCode` char(3) NOT NULL default '',
  `District` char(20) NOT NULL default '',
  `Population` int(11) NOT NULL default '0',
  PRIMARY KEY  (`ID`)
) ENGINE=NDBCLUSTER DEFAULT CHARSET=latin1;

INSERT INTO `City` VALUES (1,'Kabul','AFG','Kabol',1780000);
INSERT INTO `City` VALUES (2,'Qandahar','AFG','Qandahar',237500);
INSERT INTO `City` VALUES (3,'Herat','AFG','Herat',186800);
(remaining INSERT statements omitted)

This must be done for the definition of each table that is to be part of the clustered database. The easiest way to accomplish this is to do a search-and-replace on the file that contains the definitions and replace all instances of TYPE=engine_name or ENGINE=engine_name with ENGINE=NDBCLUSTER. If you do not want to modify the file, you can use the unmodified file to create the tables, and then use ALTER TABLE to change their storage engine. The particulars are given later in this section.

Assuming that you have already created a database named world on the SQL node of the cluster, you can then use the mysql command-line client to read city_table.sql, and create and populate the corresponding table in the usual manner:

shell> mysql world < city_table.sql

It is very important to keep in mind that the preceding command must be executed on the host where the SQL node is running (in this case, on the machine with the IP address 192.168.0.20).

To create a copy of the entire world database on the SQL node, use mysqldump on the non-cluster server to export the database to a file named world.sql; for example, in the /tmp directory. Then modify the table definitions as just described and import the file into the SQL node of the cluster like this:

shell> mysql world < /tmp/world.sql

If you save the file to a different location, adjust the preceding instructions accordingly.

Running SELECT queries on the SQL node is no different from running them on any other instance of a MySQL server. To run queries from the command line, you first need to log in to the MySQL Monitor in the usual way (specify the root password at the Enter password: prompt):

shell> mysql -u root -p
Enter password:
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 1 to server version: 5.1.24-ndb-6.2.16

Type 'help;' or '\h' for help. Type '\c' to clear the buffer.

mysql>

We simply use the MySQL server's root account and assume that you have followed the standard security precautions for installing a MySQL server, including setting a strong root password. For more information, see Section 2.10.3, “Securing the Initial MySQL Accounts”.

It is worth taking into account that Cluster nodes do not make use of the MySQL privilege system when accessing one another. Setting or changing MySQL user accounts (including the root account) effects only applications that access the SQL node, not interaction between nodes. See Section 20.8.2, “MySQL Cluster and MySQL Privileges”, for more information.

If you did not modify the ENGINE clauses in the table definitions prior to importing the SQL script, you should run the following statements at this point:

mysql> USE world;
mysql> ALTER TABLE City ENGINE=NDBCLUSTER;
mysql> ALTER TABLE Country ENGINE=NDBCLUSTER;
mysql> ALTER TABLE CountryLanguage ENGINE=NDBCLUSTER;

Selecting a database and running a SELECT query against a table in that database is also accomplished in the usual manner, as is exiting the MySQL Monitor:

mysql> USE world;
mysql> SELECT Name, Population FROM City ORDER BY Population DESC LIMIT 5;
+-----------+------------+
| Name      | Population |
+-----------+------------+
| Bombay    |   10500000 |
| Seoul     |    9981619 |
| São Paulo |    9968485 |
| Shanghai  |    9696300 |
| Jakarta   |    9604900 |
+-----------+------------+
5 rows in set (0.34 sec)

mysql> \q
Bye

shell>

Applications that use MySQL can employ standard APIs to access NDB tables. It is important to remember that your application must access the SQL node, and not the management or data nodes. This brief example shows how we might execute the SELECT statement just shown by using the PHP 5.X mysqli extension running on a Web server elsewhere on the network:

<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN"
  "http://www.w3.org/TR/html4/loose.dtd">
<html>
<head>
  <meta http-equiv="Content-Type"
        content="text/html; charset=iso-8859-1">
  <title>SIMPLE mysqli SELECT</title>
</head>
<body>
<?php
  # connect to SQL node:
  $link = new mysqli('192.168.0.20', 'root', 'root_password', 'world');
  # parameters for mysqli constructor are:
  #   host, user, password, database

  if( mysqli_connect_errno() )
    die("Connect failed: " . mysqli_connect_error());

  $query = "SELECT Name, Population
            FROM City
            ORDER BY Population DESC
            LIMIT 5";

  # if no errors...
  if( $result = $link->query($query) )
  {
?>
<table border="1" width="40%" cellpadding="4" cellspacing ="1">
  <tbody>
  <tr>
    <th width="10%">City</th>
    <th>Population</th>
  </tr>
<?
    # then display the results...
    while($row = $result->fetch_object())
      printf(<tr>\n  <td align=\"center\">%s</td><td>%d</td>\n</tr>\n",
              $row->Name, $row->Population);
?>
  </tbody
</table>
<?
  # ...and verify the number of rows that were retrieved
    printf("<p>Affected rows: %d</p>\n", $link->affected_rows);
  }
  else
    # otherwise, tell us what went wrong
    echo mysqli_error();

  # free the result set and the mysqli connection object
  $result->close();
  $link->close();
?>
</body>
</html>

We assume that the process running on the Web server can reach the IP address of the SQL node.

In a similar fashion, you can use the MySQL C API, Perl-DBI, Python-mysql, or MySQL AB's own Connectors to perform the tasks of data definition and manipulation just as you would normally with MySQL.

20.2.6. Safe Shutdown and Restart

To shut down the cluster, enter the following command in a shell on the machine hosting the management node:

shell> ndb_mgm -e shutdown

The -e option here is used to pass a command to the ndb_mgm client from the shell. (See Section 20.6.5, “Command Options for MySQL Cluster Processes”, for more information about this option.) The command causes the ndb_mgm, ndb_mgmd, and any ndbd processes to terminate gracefully. Any SQL nodes can be terminated using mysqladmin shutdown and other means.

To restart the cluster, run these commands:

On the management host (192.168.0.10 in our example setup):
```
shell> ndb_mgmd -f /var/lib/mysql-cluster/config.ini
```
On each of the data node hosts (192.168.0.30 and 192.168.0.40):
```
shell> ndbd
```
On the SQL host (192.168.0.20):
```
shell> mysqld_safe &
```

In a production setting, it is usually not desirable to shut down the cluster completely. In many cases, even when making configuration changes, or performing upgrades to the cluster hardware or software (or both), which require shutting down individual host machines, it is possible to do so without shutting down the cluster as a whole by performing a rolling restart of the cluster. For more information about doing this, see Section 20.5.1, “Performing a Rolling Restart of the Cluster”.

20.3. MySQL Cluster Configuration

20.3.1. Building MySQL Cluster from Source Code
20.3.2. Installing the Cluster Software
20.3.3. Quick Test Setup of MySQL Cluster
20.3.4. Configuration File
20.3.5. Overview of Cluster Configuration Parameters
20.3.6. Configuring Parameters for Local Checkpoints

A MySQL server that is part of a MySQL Cluster differs in only one respect from a normal (non-clustered) MySQL server, in that it employs the NDBCLUSTER storage engine. This engine is also referred to simply as NDB, and the two forms of the name are synonymous.

To avoid unnecessary allocation of resources, the server is configured by default with the NDB storage engine disabled. To enable NDB, you must modify the server's my.cnf configuration file, or start the server with the --ndbcluster option.

For more information about --ndbcluster and other MySQL server options specific to MySQL Cluster, see Section 20.4.2, “MySQL Cluster-Related Command Options for mysqld”.

The MySQL server is a part of the cluster, so it also must know how to access an MGM node to obtain the cluster configuration data. The default behavior is to look for the MGM node on localhost. However, should you need to specify that its location is elsewhere, this can be done in my.cnf or on the MySQL server command line. Before the NDB storage engine can be used, at least one MGM node must be operational, as well as any desired data nodes.

20.3.1. Building MySQL Cluster from Source Code

NDB, the Cluster storage engine, is available in binary distributions for Linux, Mac OS X, and Solaris. We are working to make Cluster run on all operating systems supported by MySQL, including Windows.

If you choose to build from a source tarball or one of the MySQL Cluster public development trees, be sure to use the --with-ndbcluster option when running configure. You can also use the BUILD/compile-pentium-max build script. Note that this script includes OpenSSL, so you must either have or obtain OpenSSL to build successfully, or else modify compile-pentium-max to exclude this requirement. Of course, you can also just follow the standard instructions for compiling your own binaries, and then perform the usual tests and installation procedure. See Section 2.9.3, “Installing from the Development Source Tree”.

BUILD/compile-pentium-max also includes OpenSSL, so you must either have or obtain OpenSSL to build successfully, or else modify compile-pentium-max to exclude this requirement. Of course, you can also just follow the standard instructions for compiling your own binaries, and then perform the usual tests and installation procedure. See Section 2.9.3, “Installing from the Development Source Tree”.

You should also note that compile-pentium-max installs MySQL to the directory /usr/local/mysql, placing all MySQL Cluster executables, scripts, databases, and support files in subdirectories under this directory. If this is not what you desire, be sure to modify the script accordingly.

20.3.2. Installing the Cluster Software

In the next few sections, we assume that you are already familiar with installing MySQL, and here we cover only the differences between configuring MySQL Cluster and configuring MySQL without clustering. (See Chapter 2, Installing and Upgrading MySQL, if you require more information about the latter.)

You will find Cluster configuration easiest if you have already have all management and data nodes running first; this is likely to be the most time-consuming part of the configuration. Editing the my.cnf file is fairly straightforward, and this section will cover only any differences from configuring MySQL without clustering.

20.3.3. Quick Test Setup of MySQL Cluster

To familiarize you with the basics, we will describe the simplest possible configuration for a functional MySQL Cluster. After this, you should be able to design your desired setup from the information provided in the other relevant sections of this chapter.

First, you need to create a configuration directory such as /var/lib/mysql-cluster, by executing the following command as the system root user:

shell> mkdir /var/lib/mysql-cluster

In this directory, create a file named config.ini that contains the following information. Substitute appropriate values for HostName and DataDir as necessary for your system.

# file "config.ini" - showing minimal setup consisting of 1 data node,
# 1 management server, and 3 MySQL servers.
# The empty default sections are not required, and are shown only for
# the sake of completeness.
# Data nodes must provide a hostname but MySQL Servers are not required
# to do so.
# If you don't know the hostname for your machine, use localhost.
# The DataDir parameter also has a default value, but it is recommended to
# set it explicitly.
# Note: [db], [api], and [mgm] are aliases for [ndbd], [mysqld], and [ndb_mgmd],
# respectively. [db] is deprecated and should not be used in new installations.

[ndbd default]
NoOfReplicas= 1

[mysqld  default]
[ndb_mgmd default]
[tcp default]

[ndb_mgmd]
HostName= myhost.example.com

[ndbd]
HostName= myhost.example.com
DataDir= /var/lib/mysql-cluster

[mysqld]
[mysqld]
[mysqld]

You can now start the ndb_mgmd management server. By default, it attempts to read the config.ini file in its current working directory, so change location into the directory where the file is located and then invoke ndb_mgmd:

shell> cd /var/lib/mysql-cluster
shell> ndb_mgmd

Then start a single data node by running ndbd:

shell> ndbd

For command-line options which can be used when starting ndbd, see Section 20.6.5, “Command Options for MySQL Cluster Processes”.

By default, ndbd looks for the management server at localhost on port 1186.

Note

If you have installed MySQL from a binary tarball, you will need to specify the path of the ndb_mgmd and ndbd servers explicitly. (Normally, these will be found in /usr/local/mysql/bin.)

Finally, change location to the MySQL data directory (usually /var/lib/mysql or /usr/local/mysql/data), and make sure that the my.cnf file contains the option necessary to enable the NDB storage engine:

[mysqld]
ndbcluster

You can now start the MySQL server as usual:

shell> mysqld_safe --user=mysql &

Wait a moment to make sure the MySQL server is running properly. If you see the notice mysql ended, check the server's .err file to find out what went wrong.

If all has gone well so far, you now can start using the cluster. Connect to the server and verify that the NDBCLUSTER storage engine is enabled:

shell> mysql
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 1 to server version: 5.1.28

Type 'help;' or '\h' for help. Type '\c' to clear the buffer.

mysql> SHOW ENGINES\G
...
*************************** 12. row ***************************
Engine: NDBCLUSTER
Support: YES
Comment: Clustered, fault-tolerant, memory-based tables
*************************** 13. row ***************************
Engine: NDB
Support: YES
Comment: Alias for NDBCLUSTER
...

The row numbers shown in the preceding example output may be different from those shown on your system, depending upon how your server is configured.

Try to create an NDBCLUSTER table:

shell> mysql
mysql> USE test;
Database changed

mysql> CREATE TABLE ctest (i INT) ENGINE=NDBCLUSTER;
Query OK, 0 rows affected (0.09 sec)

mysql> SHOW CREATE TABLE ctest \G
*************************** 1. row ***************************
       Table: ctest
Create Table: CREATE TABLE `ctest` (
  `i` int(11) default NULL
) ENGINE=ndbcluster DEFAULT CHARSET=latin1
1 row in set (0.00 sec)

To check that your nodes were set up properly, start the management client:

shell> ndb_mgm

Use the SHOW command from within the management client to obtain a report on the cluster's status:

ndb_mgm> SHOW
Cluster Configuration
---------------------
[ndbd(NDB)]     1 node(s)
id=2    @127.0.0.1  (Version: 3.5.3, Nodegroup: 0, Master)

[ndb_mgmd(MGM)] 1 node(s)
id=1    @127.0.0.1  (Version: 3.5.3)

[mysqld(API)]   3 node(s)
id=3    @127.0.0.1  (Version: 3.5.3)
id=4 (not connected, accepting connect from any host)
id=5 (not connected, accepting connect from any host)

At this point, you have successfully set up a working MySQL Cluster. You can now store data in the cluster by using any table created with ENGINE=NDBCLUSTER or its alias ENGINE=NDB.

20.3.4. Configuration File

20.3.4.1. Basic Example Configuration
20.3.4.2. The Cluster Connectstring
20.3.4.3. Defining Cluster Computers
20.3.4.4. Defining the Management Server
20.3.4.5. Defining Data Nodes
20.3.4.6. Defining SQL and Other API Nodes
20.3.4.7. Cluster TCP/IP Connections
20.3.4.8. TCP/IP Connections Using Direct Connections
20.3.4.9. Shared-Memory Connections
20.3.4.10. SCI Transport Connections

Configuring MySQL Cluster requires working with two files:

my.cnf: Specifies options for all MySQL Cluster executables. This file, with which you should be familiar with from previous work with MySQL, must be accessible by each executable running in the cluster.
config.ini: This file is read only by the MySQL Cluster management server, which then distributes the information contained therein to all processes participating in the cluster. config.ini contains a description of each node involved in the cluster. This includes configuration parameters for data nodes and configuration parameters for connections between all nodes in the cluster. For a quick reference to the sections that can appear in this file, and what sorts of configuration parameters may be placed in each section, see Sections of the config.ini File.

We are continuously making improvements in Cluster configuration and attempting to simplify this process. Although we strive to maintain backward compatibility, there may be times when introduce an incompatible change. In such cases we will try to let Cluster users know in advance if a change is not backward compatible. If you find such a change and we have not documented it, please report it in the MySQL bugs database using the instructions given in Section 1.7, “How to Report Bugs or Problems”.

20.3.4.1. Basic Example Configuration

To support MySQL Cluster, you will need to update my.cnf as shown in the following example. Note that the options shown here should not be confused with those that are used in config.ini files. You may also specify these parameters on the command line when invoking the executables.

# my.cnf
# example additions to my.cnf for MySQL Cluster
# (valid in MySQL 5.1)

# enable ndbcluster storage engine, and provide connectstring for
# management server host (default port is 1186)
[mysqld]
ndbcluster
ndb-connectstring=ndb_mgmd.mysql.com


# provide connectstring for management server host (default port: 1186)
[ndbd]
connect-string=ndb_mgmd.mysql.com

# provide connectstring for management server host (default port: 1186)
[ndb_mgm]
connect-string=ndb_mgmd.mysql.com

# provide location of cluster configuration file
[ndb_mgmd]
config-file=/etc/config.ini

(For more information on connectstrings, see Section 20.3.4.2, “The Cluster Connectstring”.)

# my.cnf
# example additions to my.cnf for MySQL Cluster
# (will work on all versions)

# enable ndbcluster storage engine, and provide connectstring for management
# server host to the default port 1186
[mysqld]
ndbcluster
ndb-connectstring=ndb_mgmd.mysql.com:1186

Important

You may also use a separate [mysql_cluster] section in the cluster my.cnf file for settings to be read and used by all executables:

# cluster-specific settings
[mysql_cluster]
ndb-connectstring=ndb_mgmd.mysql.com:1186

For additional NDB variables that can be set in the my.cnf file, see Section 5.1.3, “System Variables”.

The configuration file is named config.ini by default. It is read by ndb_mgmd at startup and can be placed anywhere. Its location and name are specified by using --config-file=path_name on the ndb_mgmd command line. If the configuration file is not specified, ndb_mgmd by default tries to read a file named config.ini located in the current working directory.

Currently, the configuration file is in INI format, which consists of sections preceded by section headings (surrounded by square brackets), followed by the appropriate parameter names and values. One deviation from the standard INI format is that the parameter name and value can be separated by a colon (“:”) as well as the equals sign (“=”); however, the equals sign is preferred. Another deviation is that sections are not uniquely identified by section name. Instead, unique sections (such as two different nodes of the same type) are identified by a unique ID specified as a parameter within the section.

Default values are defined for most parameters, and can also be specified in config.ini. To create a default value section, simply add the word default to the section name. For example, an [ndbd] section contains parameters that apply to a particular data node, whereas an [ndbd default] section contains parameters that apply to all data nodes. Suppose that all data nodes should use the same data memory size. To configure them all, create an [ndbd default] section that contains a DataMemory line to specify the data memory size.

At a minimum, the configuration file must define the computers and nodes involved in the cluster and on which computers these nodes are located. An example of a simple configuration file for a cluster consisting of one management server, two data nodes and two MySQL servers is shown here:

# file "config.ini" - 2 data nodes and 2 SQL nodes
# This file is placed in the startup directory of ndb_mgmd (the
# management server)
# The first MySQL Server can be started from any host. The second
# can be started only on the host mysqld_5.mysql.com

[ndbd default]
NoOfReplicas= 2
DataDir= /var/lib/mysql-cluster

[ndb_mgmd]
Hostname= ndb_mgmd.mysql.com
DataDir= /var/lib/mysql-cluster

[ndbd]
HostName= ndbd_2.mysql.com

[ndbd]
HostName= ndbd_3.mysql.com

[mysqld]
[mysqld]
HostName= mysqld_5.mysql.com

Each node has its own section in the config.ini file. For example, this cluster has two data nodes, so the preceding configuration file contains two [ndbd] sections defining these nodes.

Note

Do not place comments on the same line as a section heading in the config.ini file; this causes the management server not to start because it cannot parse the configuration file in such cases.

Sections of the config.ini File

There are six different sections that you can use in the config.ini configuration file, as described in the following list:

[computer]: Defines cluster hosts. This is not required to configure a viable MySQL Cluster, but be may used as a convenience when setting up a large cluster. See Section 20.3.4.3, “Defining Cluster Computers”, for more information.
[ndbd]: Defines a cluster data node (ndbd process). See Section 20.3.4.5, “Defining Data Nodes”, for details.
[mysqld]: Defines the cluster's MySQL server nodes (also called SQL or API nodes). For a discussion of SQL node configuration, see Section 20.3.4.6, “Defining SQL and Other API Nodes”.
[mgm] or [ndb_mgmd]: Defines a cluster management server (MGM) node. For information concerning the configuration of MGM nodes, see Section 20.3.4.4, “Defining the Management Server”.
[tcp]: Defines a TCP/IP connection between cluster nodes, with TCP/IP being the default connection protocol. Normally, [tcp] or [tcp default] sections are not required to set up a MySQL Cluster, as the cluster handles this automatically; however, it may be necessary in some situations to override the defaults provided by the cluster. See Section 20.3.4.7, “Cluster TCP/IP Connections”, for information about available TCP/IP configuration parameters and how to use them. (You may also find Section 20.3.4.8, “TCP/IP Connections Using Direct Connections” to be of interest in some cases.)
[shm]: Defines shared-memory connections between nodes. In MySQL 5.1, it is enabled by default, but should still be considered experimental. For a discussion of SHM interconnects, see Section 20.3.4.9, “Shared-Memory Connections”.
[sci]:Defines Scalable Coherent Interface connections between cluster data nodes. Such connections require software which, while freely available, is not part of the MySQL Cluster distribution, as well as specialised hardware. See Section 20.3.4.10, “SCI Transport Connections” for detailed information about SCI interconnects.

You can define default values for each section. All Cluster parameter names are case-insensitive, which differs from parameters specified in my.cnf or my.ini files.

20.3.4.2. The Cluster Connectstring

With the exception of the MySQL Cluster management server (ndb_mgmd), each node that is part of a MySQL Cluster requires a connectstring that points to the management server's location. This connectstring is used in establishing a connection to the management server as well as in performing other tasks depending on the node's role in the cluster. The syntax for a connectstring is as follows:

<connectstring> :=
    [<nodeid-specification>,]<host-specification>[,<host-specification>]

<nodeid-specification> := node_id

<host-specification> := host_name[:port_num]

node_id is an integer larger than 1 which identifies a node in config.ini. host_name is a string representing a valid Internet host name or IP address. port_num is an integer referring to a TCP/IP port number.

example 1 (long):    "nodeid=2,myhost1:1100,myhost2:1100,192.168.0.3:1200"
example 2 (short):   "myhost1"

All nodes will use localhost:1186 as the default connectstring value if none is provided. If port_num is omitted from the connectstring, the default port is 1186. This port should always be available on the network because it has been assigned by IANA for this purpose (see http://www.iana.org/assignments/port-numbers for details).

By listing multiple <host-specification> values, it is possible to designate several redundant management servers. A cluster node will attempt to contact successive management servers on each host in the order specified, until a successful connection has been established.

There are a number of different ways to specify the connectstring:

Each executable has its own command-line option which enables specifying the management server at startup. (See the documentation for the respective executable.)
It is also possible to set the connectstring for all nodes in the cluster at once by placing it in a [mysql_cluster] section in the management server's my.cnf file.
For backward compatibility, two other options are available, using the same syntax:
1. Set the NDB_CONNECTSTRING environment variable to contain the connectstring.
2. Write the connectstring for each executable into a text file named Ndb.cfg and place this file in the executable's startup directory.
However, these are now deprecated and should not be used for new installations.

The recommended method for specifying the connectstring is to set it on the command line or in the my.cnf file for each executable.

The maximum length of a connectstring is 1024 characters.

20.3.4.3. Defining Cluster Computers

The [computer] section has no real significance other than serving as a way to avoid the need of defining host names for each node in the system. All parameters mentioned here are required.

Id
This is an integer value, used to refer to the host computer elsewhere in the configuration file. This is not the same as the node ID.
HostName
This is the computer's hostname or IP address.

20.3.4.4. Defining the Management Server

The [ndb_mgmd] section is used to configure the behavior of the management server. [mgm] can be used as an alias; the two section names are equivalent. All parameters in the following list are optional and assume their default values if omitted.

Note

If neither the ExecuteOnComputer nor the HostName parameter is present, the default value localhost will be assumed for both.

Id
Each node in the cluster has a unique identity, which is represented by an integer value in the range 1 to 63 inclusive. This ID is used by all internal cluster messages for addressing the node.
ExecuteOnComputer
This refers to the Id set for one of the computers defined in a [computer] section of the config.ini file.
PortNumber
This is the port number on which the management server listens for configuration requests and management commands.
HostName
Specifying this parameter defines the hostname of the computer on which the management node is to reside. To specify a hostname other than localhost, either this parameter or ExecuteOnComputer is required.
LogDestination
This parameter specifies where to send cluster logging information. There are three options in this regard — CONSOLE, SYSLOG, and FILE — with FILE being the default:
- CONSOLE outputs the log to stdout:
```
CONSOLE
```
- SYSLOG sends the log to a syslog facility, possible values being one of auth, authpriv, cron, daemon, ftp, kern, lpr, mail, news, syslog, user, uucp, local0, local1, local2, local3, local4, local5, local6, or local7.
  Note
  Not every facility is necessarily supported by every operating system.
```
SYSLOG:facility=syslog
```
- FILE pipes the cluster log output to a regular file on the same machine. The following values can be specified:
  - filename: The name of the log file.
  - maxsize: The maximum size (in bytes) to which the file can grow before logging rolls over to a new file. When this occurs, the old log file is renamed by appending .N to the filename, where N is the next number not yet used with this name.
  - maxfiles: The maximum number of log files.
```
FILE:filename=cluster.log,maxsize=1000000,maxfiles=6
```
  The default value for the FILE parameter is FILE:filename=ndb_node_id_cluster.log,maxsize=1000000,maxfiles=6, where node_id is the ID of the node.
It is possible to specify multiple log destinations separated by semicolons as shown here:
```
CONSOLE;SYSLOG:facility=local0;FILE:filename=/var/log/mgmd
```
ArbitrationRank
This parameter is used to define which nodes can act as arbitrators. Only management nodes and SQL nodes can be arbitrators. ArbitrationRank can take one of the following values:
- 0: The node will never be used as an arbitrator.
- 1: The node has high priority; that is, it will be preferred as an arbitrator over low-priority nodes.
- 2: Indicates a low-priority node which be used as an arbitrator only if a node with a higher priority is not available for that purpose.
Normally, the management server should be configured as an arbitrator by setting its ArbitrationRank to 1 (the default value) and that of all SQL nodes to 0.
Beginning with MySQL 5.1.16 and MySQL Cluster NDB 6.1.3, it is possible to disable arbitration completely by setting ArbitrationRank to 0 on all management and SQL nodes.
ArbitrationDelay
An integer value which causes the management server's responses to arbitration requests to be delayed by that number of milliseconds. By default, this value is 0; it is normally not necessary to change it.
DataDir
This specifies the directory where output files from the management server will be placed. These files include cluster log files, process output files, and the daemon's process ID (PID) file. (For log files, this location can be overridden by setting the FILE parameter for LogDestination as discussed previously in this section.)
The default value for this parameter is the directory in which ndb_mgmd is located.

20.3.4.5. Defining Data Nodes

The [ndbd] and[ndbd default]sections are used to configure the behavior of the cluster's data nodes. There are many parameters which control buffer sizes, pool sizes, timeouts, and so forth. The only mandatory parameters are:

Either ExecuteOnComputer or HostName, which must be defined in the local [ndbd] section.
The parameter NoOfReplicas, which must be defined in the[ndbd default]section, as it is common to all Cluster data nodes.

Most data node parameters are set in the [ndbd default] section. Only those parameters explicitly stated as being able to set local values are allowed to be changed in the [ndbd] section. Where present, HostName, Id and ExecuteOnComputer must be defined in the local [ndbd] section, and not in any other section of config.ini. In other words, settings for these parameters are specific to one data node.

For those parameters affecting memory usage or buffer sizes, it is possible to use K, M, or G as a suffix to indicate units of 1024, 1024×1024, or 1024×1024×1024. (For example, 100K means 100 × 1024 = 102400.) Parameter names and values are currently case-sensitive.

Identifying data nodes. The Id value (that is, the data node identifier) can be allocated on the command line when the node is started or in the configuration file.

Id
This is the node ID used as the address of the node for all cluster internal messages. For data nodes, this is an integer in the range 1 to 49 inclusive. Each node in the cluster must have a unique identity.
ExecuteOnComputer
This refers to the Id set for one of the computers defined in a [computer] section.
HostName
Specifying this parameter defines the hostname of the computer on which the data node is to reside. To specify a hostname other than localhost, either this parameter or ExecuteOnComputer is required.
ServerPort (OBSOLETE)
Each node in the cluster uses a port to connect to other nodes. This port is used also for non-TCP transporters in the connection setup phase. The default port is allocated dynamically in such a way as to ensure that no two nodes on the same computer receive the same port number, so it should not normally be necessary to specify a value for this parameter.
TcpBind_INADDR_ANY
Setting this parameter to TRUE or 1 binds IP_ADDR_ANY so that connections can be made from anywhere (for autogenerated connections). The default is FALSE (0).
This parameter was added in MySQL Cluster NDB 6.2.0.
NoOfReplicas
This global parameter can be set only in the [ndbd default] section, and defines the number of replicas for each table stored in the cluster. This parameter also specifies the size of node groups. A node group is a set of nodes all storing the same information.
Node groups are formed implicitly. The first node group is formed by the set of data nodes with the lowest node IDs, the next node group by the set of the next lowest node identities, and so on. By way of example, assume that we have 4 data nodes and that NoOfReplicas is set to 2. The four data nodes have node IDs 2, 3, 4 and 5. Then the first node group is formed from nodes 2 and 3, and the second node group by nodes 4 and 5. It is important to configure the cluster in such a manner that nodes in the same node groups are not placed on the same computer because a single hardware failure would cause the entire cluster to fail.
If no node IDs are provided, the order of the data nodes will be the determining factor for the node group. Whether or not explicit assignments are made, they can be viewed in the output of the management client's SHOW statement.
There is no default value for NoOfReplicas; the maximum possible value is 4. Currently, only the values 1 and 2 are actually supported (see Bug#18621).
Important
Setting NoOfReplicas to 1 means that there is only a single copy of all Cluster data; in this case, the loss of a single data node causes the cluster to fail because there are no additional copies of the data stored by that node.
The value for this parameter must divide evenly into the number of data nodes in the cluster. For example, if there are two data nodes, then NoOfReplicas must be equal to either 1 or 2, since 2/3 and 2/4 both yield fractional values; if there are four data nodes, then NoOfReplicas must be equal to 1, 2, or 4.
DataDir
This parameter specifies the directory where trace files, log files, pid files and error logs are placed.
FileSystemPath
This parameter specifies the directory where all files created for metadata, REDO logs, UNDO logs (for Disk Data tables) and data files are placed. The default is the directory specified by DataDir.
Note
This directory must exist before the ndbd process is initiated.
The recommended directory hierarchy for MySQL Cluster includes /var/lib/mysql-cluster, under which a directory for the node's filesystem is created. The name of this subdirectory contains the node ID. For example, if the node ID is 2, this subdirectory is named ndb_2_fs.
BackupDataDir
This parameter specifies the directory in which backups are placed. If omitted, the default backup location is the directory named BACKUP under the location specified by the FileSystemPath parameter. (See above.)

Data Memory, Index Memory, and String Memory

DataMemory and IndexMemory are [ndbd] parameters specifying the size of memory segments used to store the actual records and their indexes. In setting values for these, it is important to understand how DataMemory and IndexMemory are used, as they usually need to be updated to reflect actual usage by the cluster:

DataMemory
This parameter defines the amount of space (in bytes) available for storing database records. The entire amount specified by this value is allocated in memory, so it is extremely important that the machine has sufficient physical memory to accommodate it.
The memory allocated by DataMemory is used to store both the actual records and indexes. There is a 16-byte overhead on each record; an additional amount for each record is incurred because it is stored in a 32KB page with 128 byte page overhead (see below). There is also a small amount wasted per page due to the fact that each record is stored in only one page.
For variable-size table attributes in MySQL 5.1, the data is stored on separate datapages, allocated from DataMemory. Variable-length records use a fixed-size part with an extra overhead of 4 bytes to reference the variable-size part. The variable-size part has 2 bytes overhead plus 2 bytes per attribute.
The maximum record size is currently 8052 bytes.
The memory space defined by DataMemory is also used to store ordered indexes, which use about 10 bytes per record. Each table row is represented in the ordered index. A common error among users is to assume that all indexes are stored in the memory allocated by IndexMemory, but this is not the case: Only primary key and unique hash indexes use this memory; ordered indexes use the memory allocated by DataMemory. However, creating a primary key or unique hash index also creates an ordered index on the same keys, unless you specify USING HASH in the index creation statement. This can be verified by running ndb_desc -d db_name table_name in the management client.
The memory space allocated by DataMemory consists of 32KB pages, which are allocated to table fragments. Each table is normally partitioned into the same number of fragments as there are data nodes in the cluster. Thus, for each node, there are the same number of fragments as are set in NoOfReplicas.
In addition, due to the way in which new pages are allocated when the capacity of the current page is exhausted, there is an additional overhead of approximately 18.75%. When more DataMemory is required, more than one new page is allocated, according to the following formula:
```
number of new pages = FLOOR(number of current pages × 0.1875) + 1
```
For example, if 15 pages are currently allocated to a given table and an insert to this table requires additional storage space, the number of new pages allocated to the table is FLOOR(15 × 0.1875) + 1 = FLOOR(2.8125) + 1 = 2 + 1 = 3. Now 15 + 3 = 18 memory pages are allocated to the table. When the last of these 18 pages becomes full, FLOOR(18 × 0.1875) + 1 = FLOOR(3.3750) + 1 = 3 + 1 = 4 new pages are allocated, so the total number of pages allocated to the table is now 22.
Note
The “18.75% + 1” overhead is no longer required beginning with MySQL Cluster NDB 6.2.3 and MySQL Cluster NDB 6.3.0.
Once a page has been allocated, it is currently not possible to return it to the pool of free pages, except by deleting the table. (This also means that DataMemory pages, once allocated to a given table, cannot be used by other tables.) Performing a node recovery also compresses the partition because all records are inserted into empty partitions from other live nodes.
The DataMemory memory space also contains UNDO information: For each update, a copy of the unaltered record is allocated in the DataMemory. There is also a reference to each copy in the ordered table indexes. Unique hash indexes are updated only when the unique index columns are updated, in which case a new entry in the index table is inserted and the old entry is deleted upon commit. For this reason, it is also necessary to allocate enough memory to handle the largest transactions performed by applications using the cluster. In any case, performing a few large transactions holds no advantage over using many smaller ones, for the following reasons:
- Large transactions are not any faster than smaller ones
- Large transactions increase the number of operations that are lost and must be repeated in event of transaction failure
- Large transactions use more memory
The default value for DataMemory is 80MB; the minimum is 1MB. There is no maximum size, but in reality the maximum size has to be adapted so that the process does not start swapping when the limit is reached. This limit is determined by the amount of physical RAM available on the machine and by the amount of memory that the operating system may commit to any one process. 32-bit operating systems are generally limited to 2–4GB per process; 64-bit operating systems can use more. For large databases, it may be preferable to use a 64-bit operating system for this reason.
IndexMemory
This parameter controls the amount of storage used for hash indexes in MySQL Cluster. Hash indexes are always used for primary key indexes, unique indexes, and unique constraints. Note that when defining a primary key and a unique index, two indexes will be created, one of which is a hash index used for all tuple accesses as well as lock handling. It is also used to enforce unique constraints.
The size of the hash index is 25 bytes per record, plus the size of the primary key. For primary keys larger than 32 bytes another 8 bytes is added.
The default value for IndexMemory is 18MB. The minimum is 1MB.
StringMemory
This parameter determines how much memory is allocated for strings such as table names, and is specified in an [ndbd] or [ndbd default] section of the config.ini file. A value between 0 and 100 inclusive is interpreted as a percent of the maximum default value, which is calculated based on a number of factors including the number of tables, maximum table name size, maximum size of .FRM files, MaxNoOfTriggers, maximum column name size, and maximum default column value. In general it is safe to assume that the maximum default value is approximately 5 MB for a MySQL Cluster having 1000 tables.
A value greater than 100 is interpreted as a number of bytes.
The default value is 5 — that is, 5 percent of the default maximum, or roughly 5 KB. (Note that this is a change from previous versions of MySQL Cluster.)
Under most circumstances, the default value should be sufficient, but when you have a great many Cluster tables (1000 or more), it is possible to get Error 773 Out of string memory, please modify StringMemory config parameter: Permanent error: Schema error, in which case you should increase this value. 25 (25 percent) is not excessive, and should prevent this error from recurring in all but the most extreme conditions.

The following example illustrates how memory is used for a table. Consider this table definition:

CREATE TABLE example (
  a INT NOT NULL,
  b INT NOT NULL,
  c INT NOT NULL,
  PRIMARY KEY(a),
  UNIQUE(b)
) ENGINE=NDBCLUSTER;

For each record, there are 12 bytes of data plus 12 bytes overhead. Having no nullable columns saves 4 bytes of overhead. In addition, we have two ordered indexes on columns a and b consuming roughly 10 bytes each per record. There is a primary key hash index on the base table using roughly 29 bytes per record. The unique constraint is implemented by a separate table with b as primary key and a as a column. This other table consumes an additional 29 bytes of index memory per record in the example table as well 8 bytes of record data plus 12 bytes of overhead.

Thus, for one million records, we need 58MB for index memory to handle the hash indexes for the primary key and the unique constraint. We also need 64MB for the records of the base table and the unique index table, plus the two ordered index tables.

You can see that hash indexes takes up a fair amount of memory space; however, they provide very fast access to the data in return. They are also used in MySQL Cluster to handle uniqueness constraints.

Currently, the only partitioning algorithm is hashing and ordered indexes are local to each node. Thus, ordered indexes cannot be used to handle uniqueness constraints in the general case.

An important point for both IndexMemory and DataMemory is that the total database size is the sum of all data memory and all index memory for each node group. Each node group is used to store replicated information, so if there are four nodes with two replicas, there will be two node groups. Thus, the total data memory available is 2 × DataMemory for each data node.

It is highly recommended that DataMemory and IndexMemory be set to the same values for all nodes. Data distribution is even over all nodes in the cluster, so the maximum amount of space available for any node can be no greater than that of the smallest node in the cluster.

DataMemory and IndexMemory can be changed, but decreasing either of these can be risky; doing so can easily lead to a node or even an entire MySQL Cluster that is unable to restart due to there being insufficient memory space. Increasing these values should be acceptable, but it is recommended that such upgrades are performed in the same manner as a software upgrade, beginning with an update of the configuration file, and then restarting the management server followed by restarting each data node in turn.

Updates do not increase the amount of index memory used. Inserts take effect immediately; however, rows are not actually deleted until the transaction is committed.

Transaction parameters. The next three [ndbd] parameters that we discuss are important because they affect the number of parallel transactions and the sizes of transactions that can be handled by the system. MaxNoOfConcurrentTransactions sets the number of parallel transactions possible in a node. MaxNoOfConcurrentOperations sets the number of records that can be in update phase or locked simultaneously.

Both of these parameters (especially MaxNoOfConcurrentOperations) are likely targets for users setting specific values and not using the default value. The default value is set for systems using small transactions, to ensure that these do not use excessive memory.

MaxNoOfConcurrentTransactions
Each cluster data node requires a transaction record for each active transaction in the cluster. The task of coordinating transactions is distributed among all of the data nodes. The total number of transaction records in the cluster is the number of transactions in any given node times the number of nodes in the cluster.
Transaction records are allocated to individual MySQL servers. Each connection to a MySQL server requires at least one transaction record, plus an additional transaction object per table accessed by that connection. This means that a reasonable minimum for this parameter is
```
MaxNoOfConcurrentTransactions = 
    (maximum number of tables accessed in any single transaction + 1)
    * number of cluster SQL nodes
```
For example, suppose that there are 4 SQL nodes using the cluster. A single join involving 5 tables requires 6 transaction records; if there are 5 such joins in a transaction, then 5 * 6 = 30 transaction records are required for this transaction, per MySQL server, or 30 * 4 = 120 transaction records total.
This parameter must be set to the same value for all cluster data nodes. This is due to the fact that, when a data node fails, the oldest surviving node re-creates the transaction state of all transactions that were ongoing in the failed node.
Changing the value of MaxNoOfConcurrentTransactions requires a complete shutdown and restart of the cluster.
The default value is 4096.
MaxNoOfConcurrentOperations
It is a good idea to adjust the value of this parameter according to the size and number of transactions. When performing transactions of only a few operations each and not involving a great many records, there is no need to set this parameter very high. When performing large transactions involving many records need to set this parameter higher.
Records are kept for each transaction updating cluster data, both in the transaction coordinator and in the nodes where the actual updates are performed. These records contain state information needed to find UNDO records for rollback, lock queues, and other purposes.
This parameter should be set to the number of records to be updated simultaneously in transactions, divided by the number of cluster data nodes. For example, in a cluster which has four data nodes and which is expected to handle 1,000,000 concurrent updates using transactions, you should set this value to 1000000 / 4 = 250000.
Read queries which set locks also cause operation records to be created. Some extra space is allocated within individual nodes to accommodate cases where the distribution is not perfect over the nodes.
When queries make use of the unique hash index, there are actually two operation records used per record in the transaction. The first record represents the read in the index table and the second handles the operation on the base table.
The default value is 32768.
This parameter actually handles two values that can be configured separately. The first of these specifies how many operation records are to be placed with the transaction coordinator. The second part specifies how many operation records are to be local to the database.
A very large transaction performed on an eight-node cluster requires as many operation records in the transaction coordinator as there are reads, updates, and deletes involved in the transaction. However, the operation records of the are spread over all eight nodes. Thus, if it is necessary to configure the system for one very large transaction, it is a good idea to configure the two parts separately. MaxNoOfConcurrentOperations will always be used to calculate the number of operation records in the transaction coordinator portion of the node.
It is also important to have an idea of the memory requirements for operation records. These consume about 1KB per record.
MaxNoOfLocalOperations
By default, this parameter is calculated as 1.1 × MaxNoOfConcurrentOperations. This fits systems with many simultaneous transactions, none of them being very large. If there is a need to handle one very large transaction at a time and there are many nodes, it is a good idea to override the default value by explicitly specifying this parameter.

Transaction temporary storage. The next set of [ndbd] parameters is used to determine temporary storage when executing a statement that is part of a Cluster transaction. All records are released when the statement is completed and the cluster is waiting for the commit or rollback.

The default values for these parameters are adequate for most situations. However, users with a need to support transactions involving large numbers of rows or operations may need to increase these values to enable better parallelism in the system, whereas users whose applications require relatively small transactions can decrease the values to save memory.

MaxNoOfConcurrentIndexOperations
For queries using a unique hash index, another temporary set of operation records is used during a query's execution phase. This parameter sets the size of that pool of records. Thus, this record is allocated only while executing a part of a query. As soon as this part has been executed, the record is released. The state needed to handle aborts and commits is handled by the normal operation records, where the pool size is set by the parameter MaxNoOfConcurrentOperations.
The default value of this parameter is 8192. Only in rare cases of extremely high parallelism using unique hash indexes should it be necessary to increase this value. Using a smaller value is possible and can save memory if the DBA is certain that a high degree of parallelism is not required for the cluster.
MaxNoOfFiredTriggers
The default value of MaxNoOfFiredTriggers is 4000, which is sufficient for most situations. In some cases it can even be decreased if the DBA feels certain the need for parallelism in the cluster is not high.
A record is created when an operation is performed that affects a unique hash index. Inserting or deleting a record in a table with unique hash indexes or updating a column that is part of a unique hash index fires an insert or a delete in the index table. The resulting record is used to represent this index table operation while waiting for the original operation that fired it to complete. This operation is short-lived but can still require a large number of records in its pool for situations with many parallel write operations on a base table containing a set of unique hash indexes.
TransactionBufferMemory
The memory affected by this parameter is used for tracking operations fired when updating index tables and reading unique indexes. This memory is used to store the key and column information for these operations. It is only very rarely that the value for this parameter needs to be altered from the default.
The default value for TransactionBufferMemory is 1MB.
Normal read and write operations use a similar buffer, whose usage is even more short-lived. The compile-time parameter ZATTRBUF_FILESIZE (found in ndb/src/kernel/blocks/Dbtc/Dbtc.hpp) set to 4000 × 128 bytes (500KB). A similar buffer for key information, ZDATABUF_FILESIZE (also in Dbtc.hpp) contains 4000 × 16 = 62.5KB of buffer space. Dbtc is the module that handles transaction coordination.

Scans and buffering. There are additional [ndbd] parameters in the Dblqh module (in ndb/src/kernel/blocks/Dblqh/Dblqh.hpp) that affect reads and updates. These include ZATTRINBUF_FILESIZE, set by default to 10000 × 128 bytes (1250KB) and ZDATABUF_FILE_SIZE, set by default to 10000*16 bytes (roughly 156KB) of buffer space. To date, there have been neither any reports from users nor any results from our own extensive tests suggesting that either of these compile-time limits should be increased.

MaxNoOfConcurrentScans
This parameter is used to control the number of parallel scans that can be performed in the cluster. Each transaction coordinator can handle the number of parallel scans defined for this parameter. Each scan query is performed by scanning all partitions in parallel. Each partition scan uses a scan record in the node where the partition is located, the number of records being the value of this parameter times the number of nodes. The cluster should be able to sustain MaxNoOfConcurrentScans scans concurrently from all nodes in the cluster.
Scans are actually performed in two cases. The first of these cases occurs when no hash or ordered indexes exists to handle the query, in which case the query is executed by performing a full table scan. The second case is encountered when there is no hash index to support the query but there is an ordered index. Using the ordered index means executing a parallel range scan. The order is kept on the local partitions only, so it is necessary to perform the index scan on all partitions.
The default value of MaxNoOfConcurrentScans is 256. The maximum value is 500.
MaxNoOfLocalScans
Specifies the number of local scan records if many scans are not fully parallelized. If the number of local scan records is not provided, it is calculated as the product of MaxNoOfConcurrentScans and the number of data nodes in the system. The minimum value is 32.
BatchSizePerLocalScan
This parameter is used to calculate the number of lock records used to handle concurrent scan operations.
The default value is 64; this value has a strong connection to the ScanBatchSize defined in the SQL nodes.
LongMessageBuffer
This is an internal buffer used for passing messages within individual nodes and between nodes. Although it is highly unlikely that this would need to be changed, it is configurable. By default, it is set to 1MB.

Memory Allocation

MaxAllocate

This is the maximum size of the memory unit to use when allocating memory for tables. In cases where NDB gives Out of memory errors, but it is evident by examining the cluster logs or the output of DUMP 1000 (see DUMP 1000) that all available memory has not yet been used, you can increase the value of this parameter (or MaxNoOfTables, or both) in order to cause NDB to make sufficient memory available.

This parameter was introduced in MySQL 5.1.20, MySQL Cluster NDB 6.1.12 and MySQL MySQL Cluster NDB 6.2.3.

Logging and checkpointing. These [ndbd] parameters control log and checkpoint behavior.

NoOfFragmentLogFiles
This parameter sets the number of REDO log files for the node, and thus the amount of space allocated to REDO logging. Because the REDO log files are organized in a ring, it is extremely important that the first and last log files in the set (sometimes referred to as the “head” and “tail” log files, respectively) do not meet. When these approach one another too closely, the node begins aborting all transactions encompassing updates due to a lack of room for new log records.
A REDO log record is not removed until three local checkpoints have been completed since that log record was inserted. Checkpointing frequency is determined by its own set of configuration parameters discussed elsewhere in this chapter.
How these parameters interact and proposals for how to configure them are discussed in Section 20.3.6, “Configuring Parameters for Local Checkpoints”.
The default parameter value is 16, which by default means 16 sets of 4 16MB files for a total of 1024MB. Beginning with MySQL Cluster NDB 6.1.1, the size of the individual log files is configurable using the FragmentLogFileSize parameter; more information about this parameter can be found here. In scenarios requiring a great many updates, the value for NoOfFragmentLogFiles may need to be set as high as 300 or even higher to provide sufficient space for REDO logs.
If the checkpointing is slow and there are so many writes to the database that the log files are full and the log tail cannot be cut without jeopardizing recovery, all updating transactions are aborted with internal error code 410 (Out of log file space temporarily). This condition prevails until a checkpoint has completed and the log tail can be moved forward.
Important
This parameter cannot be changed “on the fly”; you must restart the node using --initial. If you wish to change this value for all data nodes in a running cluster, you can do so via a rolling node restart (using --initial when starting each data node).
FragmentLogFileSize
Setting this parameter allows you to control directly the size of redo log files. This can be useful in situations when MySQL Cluster is operating under a high load and it is unable to close fragment log files quickly enough before attemtping to open new ones (only 2 fragment log files can be open at one time); increasing the size of the fragment log files gives the cluster more time before having to open each new fragment log file. The default value for this parameter is 16M. FragmentLogFileSize was added in MySQL MySQL Cluster NDB 6.1.11.
For more information about fragment log files, see the description of the NoOfFragmentLogFiles parameter.
MaxNoOfOpenFiles
This parameter sets a ceiling on how many internal threads to allocate for open files. Any situation requiring a change in this parameter should be reported as a bug.
The default value is 40.
InitialNoOfOpenFiles
This parameter sets the initial number of internal threads to allocate for open files.
The default value is 27.
MaxNoOfSavedMessages
This parameter sets the maximum number of trace files that are kept before overwriting old ones. Trace files are generated when, for whatever reason, the node crashes.
The default is 25 trace files.

Metadata objects. The next set of [ndbd] parameters defines pool sizes for metadata objects, used to define the maximum number of attributes, tables, indexes, and trigger objects used by indexes, events, and replication between clusters. Note that these act merely as “suggestions” to the cluster, and any that are not specified revert to the default values shown.

MaxNoOfAttributes
Defines the number of attributes that can be defined in the cluster.
The default value is 1000, with the minimum possible value being 32. The maximum is 4294967039. Each attribute consumes around 200 bytes of storage per node due to the fact that all metadata is fully replicated on the servers.
When setting MaxNoOfAttributes, it is important to prepare in advance for any ALTER TABLE statements that you might want to perform in the future. This is due to the fact, during the execution of ALTER TABLE on a Cluster table, 3 times the number of attributes as in the original table are used. For example, if a table requires 100 attributes, and you want to be able to alter it later, you need to set the value of MaxNoOfAttributes to 300. Assuming that you can create all desired tables without any problems, a good rule of thumb is to add two times the number of attributes in the largest table to MaxNoOfAttributes to be sure. You should also verify that this number is sufficient by trying an actual ALTER TABLE after configuring the parameter. If this is not successful, increase MaxNoOfAttributes by another multiple of the original value and test it again.
MaxNoOfTables
A table object is allocated for each table, unique hash index, and ordered index. This parameter sets the maximum number of table objects for the cluster as a whole.
For each attribute that has a BLOB data type an extra table is used to store most of the BLOB data. These tables also must be taken into account when defining the total number of tables.
The default value of this parameter is 128. The minimum is 8 and the maximum is 1600. Each table object consumes approximately 20KB per node.
MaxNoOfOrderedIndexes
For each ordered index in the cluster, an object is allocated describing what is being indexed and its storage segments. By default, each index so defined also defines an ordered index. Each unique index and primary key has both an ordered index and a hash index.
The default value of this parameter is 128. Each object consumes approximately 10KB of data per node.
MaxNoOfUniqueHashIndexes
For each unique index that is not a primary key, a special table is allocated that maps the unique key to the primary key of the indexed table. By default, an ordered index is also defined for each unique index. To prevent this, you must specify the USING HASH option when defining the unique index.
The default value is 64. Each index consumes approximately 15KB per node.
MaxNoOfTriggers
Internal update, insert, and delete triggers are allocated for each unique hash index. (This means that three triggers are created for each unique hash index.) However, an ordered index requires only a single trigger object. Backups also use three trigger objects for each normal table in the cluster.
Replication between clusters also makes use of internal triggers.
This parameter sets the maximum number of trigger objects in the cluster.
The default value is 768.
MaxNoOfIndexes
This parameter is deprecated in MySQL 5.1; you should use MaxNoOfOrderedIndexes and MaxNoOfUniqueHashIndexes instead.
This parameter is used only by unique hash indexes. There needs to be one record in this pool for each unique hash index defined in the cluster.
The default value of this parameter is 128.

Boolean parameters. The behavior of data nodes is also affected by a set of [ndbd] parameters taking on boolean values. These parameters can each be specified as TRUE by setting them equal to 1 or Y, and as FALSE by setting them equal to 0 or N.

LockPagesInMainMemory
For a number of operating systems, including Solaris and Linux, it is possible to lock a process into memory and so avoid any swapping to disk. This can be used to help guarantee the cluster's real-time characteristics.
Beginning with MySQL 5.1.15 and MySQL Cluster NDB 6.1.1, this parameter takes one of the integer values 0, 1, or 2, which act as follows:
- 0: Disables locking. This is the default value.
- 1: Performs the lock after allocating memory for the process.
- 2: Performs the lock before memory for the process is allocated.
Previously, this parameter was a Boolean. 0 or false was the default setting, and disabled locking. 1 or true enabled locking of the process after its memory was allocated.
Important
Beginning with MySQL 5.1.15 and MySQL Cluster NDB 6.1.1, it is no longer possible to use true or false for the value of this parameter; when upgrading from a previous version, you must change the value to 0, 1, or 2.
StopOnError
This parameter specifies whether an ndbd process should exit or perform an automatic restart when an error condition is encountered.
This feature is enabled by default.
Diskless
It is possible to specify MySQL Cluster tables as diskless, meaning that tables are not checkpointed to disk and that no logging occurs. Such tables exist only in main memory. A consequence of using diskless tables is that neither the tables nor the records in those tables survive a crash. However, when operating in diskless mode, it is possible to run ndbd on a diskless computer.
Important
This feature causes the entire cluster to operate in diskless mode.
When this feature is enabled, Cluster online backup is disabled. In addition, a partial start of the cluster is not possible.
Diskless is disabled by default.
ODirect
Enabling this parameter causes NDB to attempt using O_DIRECT writes for LCP, backups, and redo logs, often lowering kswapd and CPU usage.
Warning
When using MySQL Cluster Linux, ODirect should be enabled for 2.6 or newer kernels.
This parameter was added in the following releases:
- MySQL 5.1.20
- MySQL Cluster NDB 6.1.11
- MySQL Cluster NDB 6.2.3
- MySQL Cluster NDB 6.3.0
ODirect is disabled by default.
RestartOnErrorInsert
This feature is accessible only when building the debug version where it is possible to insert errors in the execution of individual blocks of code as part of testing.
This feature is disabled by default.
CompressedBackup
Setting this parameter to 1 causes backup files to be compressed. The compression used is equivalent to gzip --fast, and can save 50% or more of the space required on the data node to store uncompressed backup files. Compressed backups can be enabled for individual data nodes, or for all data nodes (by setting this parameter in the [ndbd default] section of the config.ini file).
Important
You cannot restore a compressed backup to a cluster running a MySQL version that does not support this feature.
The default value is 0 (disabled).
This parameter was introduced in MySQL Cluster NDB 6.3.7.
CompressedLCP
Setting this parameter to 1 causes local checkpoint files to be compressed. The compression used is equivalent to gzip --fast, and can save 50% or more of the space required on the data node to store uncompressed checkpoint files. Compressed LCPs can be enabled for individual data nodes, or for all data nodes (by setting this parameter in the [ndbd default] section of the config.ini file).
Important
You cannot restore a compressed local checkpoint to a cluster running a MySQL version that does not support this feature.
The default value is 0 (disabled).
This parameter was introduced in MySQL Cluster NDB 6.3.7.

Controlling Timeouts, Intervals, and Disk Paging

There are a number of [ndbd] parameters specifying timeouts and intervals between various actions in Cluster data nodes. Most of the timeout values are specified in milliseconds. Any exceptions to this are mentioned where applicable.

TimeBetweenWatchDogCheck
To prevent the main thread from getting stuck in an endless loop at some point, a “watchdog” thread checks the main thread. This parameter specifies the number of milliseconds between checks. If the process remains in the same state after three checks, the watchdog thread terminates it.
This parameter can easily be changed for purposes of experimentation or to adapt to local conditions. It can be specified on a per-node basis although there seems to be little reason for doing so.
The default timeout is 6000 milliseconds (6 seconds).
TimeBetweenWatchDogCheckInitial
This is similar to the TimeBetweenWatchDogCheck parameter, except that TimeBetweenWatchDogCheckInitial controls the amount of time that passes between execution checks inside a database node in the early start phases during which memory is allocated.
The default timeout is 6000 milliseconds (6 seconds).
This parameter was added in MySQL 5.1.20.
StartPartialTimeout
This parameter specifies how long the Cluster waits for all data nodes to come up before the cluster initialization routine is invoked. This timeout is used to avoid a partial Cluster startup whenever possible.
The default value is 30000 milliseconds (30 seconds). 0 disables the timeout, in which case the cluster may start only if all nodes are available.
StartPartitionedTimeout
If the cluster is ready to start after waiting for StartPartialTimeout milliseconds but is still possibly in a partitioned state, the cluster waits until this timeout has also passed.
The default timeout is 60000 milliseconds (60 seconds).
StartFailureTimeout
If a data node has not completed its startup sequence within the time specified by this parameter, the node startup fails. Setting this parameter to 0 (the default value) means that no data node timeout is applied.
For nonzero values, this parameter is measured in milliseconds. For data nodes containing extremely large amounts of data, this parameter should be increased. For example, in the case of a data node containing several gigabytes of data, a period as long as 10–15 minutes (that is, 600000 to 1000000 milliseconds) might be required to perform a node restart.
HeartbeatIntervalDbDb
One of the primary methods of discovering failed nodes is by the use of heartbeats. This parameter states how often heartbeat signals are sent and how often to expect to receive them. After missing three heartbeat intervals in a row, the node is declared dead. Thus, the maximum time for discovering a failure through the heartbeat mechanism is four times the heartbeat interval.
The default heartbeat interval is 1500 milliseconds (1.5 seconds). This parameter must not be changed drastically and should not vary widely between nodes. If one node uses 5000 milliseconds and the node watching it uses 1000 milliseconds, obviously the node will be declared dead very quickly. This parameter can be changed during an online software upgrade, but only in small increments.
HeartbeatIntervalDbApi
Each data node sends heartbeat signals to each MySQL server (SQL node) to ensure that it remains in contact. If a MySQL server fails to send a heartbeat in time it is declared “dead,” in which case all ongoing transactions are completed and all resources released. The SQL node cannot reconnect until all activities initiated by the previous MySQL instance have been completed. The three-heartbeat criteria for this determination are the same as described for HeartbeatIntervalDbDb.
The default interval is 1500 milliseconds (1.5 seconds). This interval can vary between individual data nodes because each data node watches the MySQL servers connected to it, independently of all other data nodes.
TimeBetweenLocalCheckpoints
This parameter is an exception in that it does not specify a time to wait before starting a new local checkpoint; rather, it is used to ensure that local checkpoints are not performed in a cluster where relatively few updates are taking place. In most clusters with high update rates, it is likely that a new local checkpoint is started immediately after the previous one has been completed.
The size of all write operations executed since the start of the previous local checkpoints is added. This parameter is also exceptional in that it is specified as the base-2 logarithm of the number of 4-byte words, so that the default value 20 means 4MB (4 × 2²⁰) of write operations, 21 would mean 8MB, and so on up to a maximum value of 31, which equates to 8GB of write operations.
All the write operations in the cluster are added together. Setting TimeBetweenLocalCheckpoints to 6 or less means that local checkpoints will be executed continuously without pause, independent of the cluster's workload.
TimeBetweenGlobalCheckpoints
When a transaction is committed, it is committed in main memory in all nodes on which the data is mirrored. However, transaction log records are not flushed to disk as part of the commit. The reasoning behind this behavior is that having the transaction safely committed on at least two autonomous host machines should meet reasonable standards for durability.
It is also important to ensure that even the worst of cases — a complete crash of the cluster — is handled properly. To guarantee that this happens, all transactions taking place within a given interval are put into a global checkpoint, which can be thought of as a set of committed transactions that has been flushed to disk. In other words, as part of the commit process, a transaction is placed in a global checkpoint group. Later, this group's log records are flushed to disk, and then the entire group of transactions is safely committed to disk on all computers in the cluster.
This parameter defines the interval between global checkpoints. The default is 2000 milliseconds.
TimeBetweenEpochs
This parameter defines the interval between synchronisation epochs for MySQL Cluster Replication. The default value is 100 milliseconds.
TimeBetweenEpochs is part of the implementation of “micro-GCPs”, which can be used to improve the performance of MySQL Cluster Replication. This parameter was introduced in MySQL Cluster NDB 6.2.5 and MySQL Cluster NDB 6.3.2.
TimeBetweenEpochsTimeout
This parameter defines a timeout for synchronisation epochs for MySQL Cluster Replication. If a node fails to participate in a global checkpoint within the time determined by this parameter, the node is shut down. The default value is 4000 milliseconds.
TimeBetweenEpochsTimeout is part of the implementation of “micro-GCPs”, which can be used to improve the performance of MySQL Cluster Replication. This parameter was introduced in MySQL-MySQL Cluster NDB 6.2.7 and MySQL Cluster NDB 6.3.4.
MaxBufferedEpochs
The number of unprocessed epochs by which a subscribing node can lag behind. Exceeding this number causes a lagging subscriber to be disconnected.
The default value of 100 is sufficient for most normal operations. If a subscribing node does lag enough to cause disconnections, it is usually due to network or scheduling issues with regard to processes or threads. (In rare circumstances, the problem may be due to a bug in the NDB client.) It may be desirable to set the value lower than the default when epochs are longer.
Disconnection prevents client issues from affecting the data node service, running out of memory to buffer data, and eventually shutting down. Instead, only the client is affected as a result of the disconnect (by, for example gap events in the binlog), forcing the client to reconnect or restart the process.
TimeBetweenInactiveTransactionAbortCheck
Timeout handling is performed by checking a timer on each transaction once for every interval specified by this parameter. Thus, if this parameter is set to 1000 milliseconds, every transaction will be checked for timing out once per second.
The default value is 1000 milliseconds (1 second).
TransactionInactiveTimeout
This parameter states the maximum time that is permitted to lapse between operations in the same transaction before the transaction is aborted.
The default for this parameter is zero (no timeout). For a real-time database that needs to ensure that no transaction keeps locks for too long, this parameter should be set to a relatively small value. The unit is milliseconds.
TransactionDeadlockDetectionTimeout
When a node executes a query involving a transaction, the node waits for the other nodes in the cluster to respond before continuing. A failure to respond can occur for any of the following reasons:
- The node is “dead”
- The operation has entered a lock queue
- The node requested to perform the action could be heavily overloaded.
This timeout parameter states how long the transaction coordinator waits for query execution by another node before aborting the transaction, and is important for both node failure handling and deadlock detection. In MySQL 5.1.10 and earlier versions, setting it too high could cause undesirable behavior in situations involving deadlocks and node failure. Beginning with MySQL 5.1.11, active transactions occurring during node failures are actively aborted by the Cluster Transaction Coordinator, and so high settings are no longer an issue with this parameter.
The default timeout value is 1200 milliseconds (1.2 seconds).
DiskSyncSize
This is the maximum number of bytes to store before flushing data to a local checkpoint file. This is done in order to prevent write buffering, which can impede performance significantly. This parameter is not intended to take the place of TimeBetweenLocalCheckpoints.
Note
When ODirect is enabled, it is not necessary to set DiskSyncSize; in fact, in such cases its value is simply ignored.
The default value is 4M (4 megabytes).
This parameter was added in MySQL 5.1.12.
DiskCheckpointSpeed
The amount of data,in bytes per second, that is sent to disk during a local checkpoint.
The default value is 10M (10 megabytes per second).
This parameter was added in MySQL 5.1.12.
DiskCheckpointSpeedInRestart
The amount of data,in bytes per second, that is sent to disk during a local checkpoint as part of a restart operation.
The default value is 100M (100 megabytes per second).
This parameter was added in MySQL 5.1.12.
NoOfDiskPagesToDiskAfterRestartTUP
When executing a local checkpoint, the algorithm flushes all data pages to disk. Merely doing so as quickly as possible without any moderation is likely to impose excessive loads on processors, networks, and disks. To control the write speed, this parameter specifies how many pages per 100 milliseconds are to be written. In this context, a “page” is defined as 8KB. This parameter is specified in units of 80KB per second, so setting NoOfDiskPagesToDiskAfterRestartTUP to a value of 20 entails writing 1.6MB in data pages to disk each second during a local checkpoint. This value includes the writing of UNDO log records for data pages. That is, this parameter handles the limitation of writes from data memory. (See the entry for IndexMemory for information about index pages.)
In short, this parameter specifies how quickly to execute local checkpoints. It operates in conjunction with NoOfFragmentLogFiles, DataMemory, and IndexMemory.
For more information about the interaction between these parameters and possible strategies for choosing appropriate values for them, see Section 20.3.6, “Configuring Parameters for Local Checkpoints”.
The default value is 40 (3.2MB of data pages per second).
Note
This parameter is deprecated as of MySQL 5.1.6. For MySQL 5.1.12 and later versions, use DiskCheckpointSpeed and DiskSyncSize instead.
NoOfDiskPagesToDiskAfterRestartACC
This parameter uses the same units as NoOfDiskPagesToDiskAfterRestartTUP and acts in a similar fashion, but limits the speed of writing index pages from index memory.
The default value of this parameter is 20 (1.6MB of index memory pages per second).
Note
This parameter is deprecated as of MySQL 5.1.6. For MySQL 5.1.12 and later versions, use DiskCheckpointSpeed and DiskSyncSize.
NoOfDiskPagesToDiskDuringRestartTUP
This parameter is used in a fashion similar to NoOfDiskPagesToDiskAfterRestartTUP and NoOfDiskPagesToDiskAfterRestartACC, only it does so with regard to local checkpoints executed in the node when a node is restarting. A local checkpoint is always performed as part of all node restarts. During a node restart it is possible to write to disk at a higher speed than at other times, because fewer activities are being performed in the node.
This parameter covers pages written from data memory.
The default value is 40 (3.2MB per second).
Note
This parameter is deprecated as of MySQL 5.1.6. For MySQL 5.1.12 and later versions, use DiskCheckpointSpeedInRestart and DiskSyncSize.
NoOfDiskPagesToDiskDuringRestartACC
Controls the number of index memory pages that can be written to disk during the local checkpoint phase of a node restart.
As with NoOfDiskPagesToDiskAfterRestartTUP and NoOfDiskPagesToDiskAfterRestartACC, values for this parameter are expressed in terms of 8KB pages written per 100 milliseconds (80KB/second).
The default value is 20 (1.6MB per second).
Note
This parameter is deprecated as of MySQL 5.1.6. For MySQL 5.1.12 and later versions, use DiskCheckpointSpeedInRestart and DiskSyncSize.
ArbitrationTimeout
This parameter specifies how long data nodes wait for a response from the arbitrator to an arbitration message. If this is exceeded, the network is assumed to have split.
The default value is 1000 milliseconds (1 second).

Buffering and logging. Several [ndbd] configuration parameters enable the advanced user to have more control over the resources used by node processes and to adjust various buffer sizes at need.

These buffers are used as front ends to the file system when writing log records to disk. If the node is running in diskless mode, these parameters can be set to their minimum values without penalty due to the fact that disk writes are “faked” by the NDB storage engine's filesystem abstraction layer.

UndoIndexBuffer
The UNDO index buffer, whose size is set by this parameter, is used during local checkpoints. The NDB storage engine uses a recovery scheme based on checkpoint consistency in conjunction with an operational REDO log. To produce a consistent checkpoint without blocking the entire system for writes, UNDO logging is done while performing the local checkpoint. UNDO logging is activated on a single table fragment at a time. This optimization is possible because tables are stored entirely in main memory.
The UNDO index buffer is used for the updates on the primary key hash index. Inserts and deletes rearrange the hash index; the NDB storage engine writes UNDO log records that map all physical changes to an index page so that they can be undone at system restart. It also logs all active insert operations for each fragment at the start of a local checkpoint.
Reads and updates set lock bits and update a header in the hash index entry. These changes are handled by the page-writing algorithm to ensure that these operations need no UNDO logging.
This buffer is 2MB by default. The minimum value is 1MB, which is sufficient for most applications. For applications doing extremely large or numerous inserts and deletes together with large transactions and large primary keys, it may be necessary to increase the size of this buffer. If this buffer is too small, the NDB storage engine issues internal error code 677 (Index UNDO buffers overloaded).
Important
It is not safe to decrease the value of this parameter during a rolling restart.
UndoDataBuffer
This parameter sets the size of the UNDO data buffer, which performs a function similar to that of the UNDO index buffer, except the UNDO data buffer is used with regard to data memory rather than index memory. This buffer is used during the local checkpoint phase of a fragment for inserts, deletes, and updates.
Because UNDO log entries tend to grow larger as more operations are logged, this buffer is also larger than its index memory counterpart, with a default value of 16MB.
This amount of memory may be unnecessarily large for some applications. In such cases, it is possible to decrease this size to a minimum of 1MB.
It is rarely necessary to increase the size of this buffer. If there is such a need, it is a good idea to check whether the disks can actually handle the load caused by database update activity. A lack of sufficient disk space cannot be overcome by increasing the size of this buffer.
If this buffer is too small and gets congested, the NDB storage engine issues internal error code 891 (Data UNDO buffers overloaded).
Important
It is not safe to decrease the value of this parameter during a rolling restart.
RedoBuffer
All update activities also need to be logged. The REDO log makes it possible to replay these updates whenever the system is restarted. The NDB recovery algorithm uses a “fuzzy” checkpoint of the data together with the UNDO log, and then applies the REDO log to play back all changes up to the restoration point.
RedoBuffer sets the size of the buffer inwhich the REDO log is written, and is 8MB by default. The minimum value is 1MB.
If this buffer is too small, the NDB storage engine issues error code 1221 (REDO log buffers overloaded).
Important
It is not safe to decrease the value of this parameter during a rolling restart.

Controlling log messages. In managing the cluster, it is very important to be able to control the number of log messages sent for various event types to stdout. For each event category, there are 16 possible event levels (numbered 0 through 15). Setting event reporting for a given event category to level 15 means all event reports in that category are sent to stdout; setting it to 0 means that there will be no event reports made in that category.

By default, only the startup message is sent to stdout, with the remaining event reporting level defaults being set to 0. The reason for this is that these messages are also sent to the management server's cluster log.

An analogous set of levels can be set for the management client to determine which event levels to record in the cluster log.

LogLevelStartup
The reporting level for events generated during startup of the process.
The default level is 1.
LogLevelShutdown
The reporting level for events generated as part of graceful shutdown of a node.
The default level is 0.
LogLevelStatistic
The reporting level for statistical events such as number of primary key reads, number of updates, number of inserts, information relating to buffer usage, and so on.
The default level is 0.
LogLevelCheckpoint
The reporting level for events generated by local and global checkpoints.
The default level is 0.
LogLevelNodeRestart
The reporting level for events generated during node restart.
The default level is 0.
LogLevelConnection
The reporting level for events generated by connections between cluster nodes.
The default level is 0.
LogLevelError
The reporting level for events generated by errors and warnings by the cluster as a whole. These errors do not cause any node failure but are still considered worth reporting.
The default level is 0.
LogLevelCongestion
The reporting level for events generated by congestion. These errors do not cause node failure but are still considered worth reporting.
The default level is 0.
LogLevelInfo
The reporting level for events generated for information about the general state of the cluster.
The default level is 0.
MemReportFrequency
This parameter controls how often data node memory usage reports are recorded in the cluster log; it is an integer value representing the number of seconds between reports.
Each data node's data memory and index memory usage is logged as both a percentage and a number of 32 KB pages of the DataMemory and IndexMemory, respectively, set in the config.ini file. For example, if DataMemory is equal to 100 MB, and a given data node is using 50 MB for data memory storage, the corresponding line in the cluster log might look like this:
```
2006-12-24 01:18:16 [MgmSrvr] INFO -- Node 2: Data usage is 50%(1280 32K pages of total 2560)
```
MemReportFrequency is not a required parameter. If used, it can be set for all cluster data nodes in the [ndbd default] section of config.ini, and can also be set or overridden for individual data nodes in the corresponding [ndbd] sections of the configuration file. The minimum value — which is also the default value — is 0, in which case memory reports are logged only when memory usage reaches certain percentages (80%, 90%, and 100%), as mentioned in the discussion of statistics events in Section 20.7.3.2, “Log Events”.
This parameter was added in MySQL Cluster 5.1.16 and MySQL Cluster NDB 6.1.0.

Backup parameters. The [ndbd] parameters discussed in this section define memory buffers set aside for execution of online backups.

BackupDataBufferSize
In creating a backup, there are two buffers used for sending data to the disk. The backup data buffer is used to fill in data recorded by scanning a node's tables. Once this buffer has been filled to the level specified as BackupWriteSize (see below), the pages are sent to disk. While flushing data to disk, the backup process can continue filling this buffer until it runs out of space. When this happens, the backup process pauses the scan and waits until some disk writes have completed freed up memory so that scanning may continue.
The default value is 2MB.
BackupLogBufferSize
The backup log buffer fulfills a role similar to that played by the backup data buffer, except that it is used for generating a log of all table writes made during execution of the backup. The same principles apply for writing these pages as with the backup data buffer, except that when there is no more space in the backup log buffer, the backup fails. For that reason, the size of the backup log buffer must be large enough to handle the load caused by write activities while the backup is being made. See Section 20.9.4, “Configuration for Cluster Backup”.
The default value for this parameter should be sufficient for most applications. In fact, it is more likely for a backup failure to be caused by insufficient disk write speed than it is for the backup log buffer to become full. If the disk subsystem is not configured for the write load caused by applications, the cluster is unlikely to be able to perform the desired operations.
It is preferable to configure cluster nodes in such a manner that the processor becomes the bottleneck rather than the disks or the network connections.
The default value is 2MB.
BackupMemory
This parameter is simply the sum of BackupDataBufferSize and BackupLogBufferSize.
The default value is 2MB + 2MB = 4MB.
Important
If BackupDataBufferSize and BackupLogBufferSize taken together exceed 4MB, then this parameter must be set explicitly in the config.ini file to their sum.
This parameter controls how often backup status reports are issued in the management client during a backup, as well as how often such reports are written to the cluster log (provided cluster event logging is configured to allow it — see Section 20.3.4.5, “Defining Data Nodes”). BackupReportFrequency represents the time in seconds between backup status reports.
The default value is 0.
This parameter was added in MySQL Cluster NDB 6.2.3.
BackupWriteSize
This parameter specifies the default size of messages written to disk by the backup log and backup data buffers.
The default value is 32KB.
BackupMaxWriteSize
This parameter specifies the maximum size of messages written to disk by the backup log and backup data buffers.
The default value is 256KB.

Important

When specifying these parameters, the following relationships must hold true. Otherwise, the data node will be unable to start:

BackupDataBufferSize >= BackupWriteSize + 188KB
BackupLogBufferSize >= BackupWriteSize + 16KB
BackupMaxWriteSize >= BackupWriteSize

Realtime Performance Parameters

The [ndbd] parameters discussed in this section are used in scheduling and locking of threads to specific CPUs on multiprocessor data node hosts. They were introduced in MySQL Cluster NDB 6.3.4.

LockExecuteThreadToCPU
This parameter specifies the ID of the CPU assigned to handle the NDBCLUSTER execution thread.
The value of this parameter is an integer in the range 0 to 65535 (inclusive). The default is 65535.
LockMaintThreadsToCPU
This parameter specifies the ID of the CPU assigned to handle NDBCLUSTER maintenance threads.
The value of this parameter is an integer in the range 0 to 65535 (inclusive). The default is 65535.
RealtimeScheduler
Setting this parameter to 1 enables real-time scheduling of NDBCLUSTER threads.
The default is 0 (scheduling disabled).
SchedulerExecutionTimer
This parameter specifies the time in microseconds for threads to be executed in the scheduler before being sent.
The default is 50 μsec.
SchedulerSpinTimer
This parameter specifies the time in microseconds for threads to be executed in the scheduler before sleeping.
The default value is 0.

20.3.4.6. Defining SQL and Other API Nodes

The [mysqld] and [api] sections in the config.ini file define the behavior of the MySQL servers (SQL nodes) and other applications (API nodes) used to access cluster data. None of the parameters shown is required. If no computer or host name is provided, any host can use this SQL or API node.

Generally speaking, a [mysqld] section is used to indicate a MySQL server providing an SQL interface to the cluster, and an [api] section is used for applications other than mysqld processes accessing cluster data, but the two designations are actually synonomous; you can, for instance, list parameters for a MySQL server acting as an SQL node in an [api] section.

Note

For a discussion of MySQL server options for MySQL Cluster, see Section 20.4.2, “MySQL Cluster-Related Command Options for mysqld”; for information about MySQL server system variables relating to MySQL Cluster, see Section 20.4.3, “MySQL Cluster System Variables”.

Id
The Id value is used to identify the node in all cluster internal messages. It must be an integer in the range 1 to 63 inclusive, and must be unique among all node IDs within the cluster.
ExecuteOnComputer
This refers to the Id set for one of the computers (hosts) defined in a [computer] section of the configuration file.
HostName
Specifying this parameter defines the hostname of the computer on which the SQL node (API node) is to reside. To specify a hostname, either this parameter or ExecuteOnComputer is required.
If no HostName or ExecuteOnComputer is specified in a given [mysql] or [api] section of the config.ini file, then an SQL or API node may connect using the corresponding “slot” from any host which can establish a network connection to the management server host machine. This differs from the default behavior for data nodes, where localhost is assumed for HostName unless otherwise specified.
ArbitrationRank
This parameter defines which nodes can act as arbitrators. Both MGM nodes and SQL nodes can be arbitrators. A value of 0 means that the given node is never used as an arbitrator, a value of 1 gives the node high priority as an arbitrator, and a value of 2 gives it low priority. A normal configuration uses the management server as arbitrator, setting its ArbitrationRank to 1 (the default) and those for all SQL nodes to 0.
Beginning with MySQL 5.1.16 and MySQL Cluster NDB 6.1.3, it is possible to disable arbitration completely by setting ArbitrationRank to 0 on all management and SQL nodes.
ArbitrationDelay
Setting this parameter to any other value than 0 (the default) means that responses by the arbitrator to arbitration requests will be delayed by the stated number of milliseconds. It is usually not necessary to change this value.
BatchByteSize
For queries that are translated into full table scans or range scans on indexes, it is important for best performance to fetch records in properly sized batches. It is possible to set the proper size both in terms of number of records (BatchSize) and in terms of bytes (BatchByteSize). The actual batch size is limited by both parameters.
The speed at which queries are performed can vary by more than 40% depending upon how this parameter is set. In future releases, MySQL Server will make educated guesses on how to set parameters relating to batch size, based on the query type.
This parameter is measured in bytes and by default is equal to 32KB.
BatchSize
This parameter is measured in number of records and is by default set to 64. The maximum size is 992.
MaxScanBatchSize
The batch size is the size of each batch sent from each data node. Most scans are performed in parallel to protect the MySQL Server from receiving too much data from many nodes in parallel; this parameter sets a limit to the total batch size over all nodes.
The default value of this parameter is set to 256KB. Its maximum size is 16MB.

You can obtain some information from a MySQL server running as a Cluster SQL node using SHOW STATUS in the mysql client, as shown here:

mysql> SHOW STATUS LIKE 'ndb%';
+-----------------------------+---------------+
| Variable_name               | Value         |
+-----------------------------+---------------+
| Ndb_cluster_node_id         | 5             | 
| Ndb_config_from_host        | 192.168.0.112 | 
| Ndb_config_from_port        | 1186          | 
| Ndb_number_of_storage_nodes | 4             | 
+-----------------------------+---------------+
4 rows in set (0.02 sec)

For information about these Cluster system status variables, see Section 5.1.6, “Status Variables”.

20.3.4.7. Cluster TCP/IP Connections

TCP/IP is the default transport mechanism for establishing connections in MySQL Cluster. It is normally not necessary to define connections because Cluster automatically set ups a connection between each of the data nodes, between each data node and all MySQL server nodes, and between each data node and the management server. (For one exception to this rule, see Section 20.3.4.8, “TCP/IP Connections Using Direct Connections”.) [tcp] sections in the config.ini file explicitly define TCP/IP connections between nodes in the cluster.

It is necessary to define a connection only to override the default connection parameters. In that case, it is necessary to define at least NodeId1, NodeId2, and the parameters to change.

Important

Any [tcp] sections in the config.ini file should be listed last, following any other sections in the file. This is not required for a [tcp default] section. This is a known issue with the way in which the config.ini file is read by the cluster management server.

It is also possible to change the default values for these parameters by setting them in the [tcp default] section.

NodeId1, NodeId2
To identify a connection between two nodes it is necessary to provide their node IDs in the [tcp] section of the configuration file. These are the same unique Id values for each of these nodes as described in Section 20.3.4.6, “Defining SQL and Other API Nodes”.
SendBufferMemory
TCP transporters use a buffer to store all messages before performing the send call to the operating system. When this buffer reaches 64KB its contents are sent; these are also sent when a round of messages have been executed. To handle temporary overload situations it is also possible to define a bigger send buffer.
The default size of the send buffer is 256 KB; 2MB is recommended in most situations in which it is necessary to set this parameter. The minimum size is 64 KB; the theoretical maximum is 4 GB.
SendSignalId
To be able to retrace a distributed message datagram, it is necessary to identify each message. When this parameter is set to Y, message IDs are transported over the network. This feature is disabled by default in production builds, and enabled in -debug builds.
Checksum
This parameter is a boolean parameter (enabled by setting it to Y or 1, disabled by setting it to N or 0). It is disabled by default. When it is enabled, checksums for all messages are calculated before they placed in the send buffer. This feature ensures that messages are not corrupted while waiting in the send buffer, or by the transport mechanism.
PortNumber (OBSOLETE)
This formerly specified the port number to be used for listening for connections from other nodes. This parameter should no longer be used.
ReceiveBufferMemory
Specifies the size of the buffer used when receiving data from the TCP/IP socket.
The default value of this parameter from its of 64 KB; 1M is recommended in most situations where the size of the receive buffer needs to be set. The minimum possible value is 16K; the theoretical maximum is 4G.

20.3.4.8. TCP/IP Connections Using Direct Connections

Setting up a cluster using direct connections between data nodes requires specifying explicitly the crossover IP addresses of the data nodes so connected in the [tcp] section of the cluster config.ini file.

In the following example, we envision a cluster with at least four hosts, one each for a management server, an SQL node, and two data nodes. The cluster as a whole resides on the 172.23.72.* subnet of a LAN. In addition to the usual network connections, the two data nodes are connected directly using a standard crossover cable, and communicate with one another directly using IP addresses in the 1.1.0.* address range as shown:

# Management Server
[ndb_mgmd]
Id=1
HostName=172.23.72.20

# SQL Node
[mysqld]
Id=2
HostName=172.23.72.21

# Data Nodes
[ndbd]
Id=3
HostName=172.23.72.22

[ndbd]
Id=4
HostName=172.23.72.23

# TCP/IP Connections
[tcp]
NodeId1=3
NodeId2=4
HostName1=1.1.0.1
HostName2=1.1.0.2

The HostNameN parameter, where N is an integer, is used only when specifying direct TCP/IP connections.

The use of direct connections between data nodes can improve the cluster's overall efficiency by allowing the data nodes to bypass an Ethernet device such as a switch, hub, or router, thus cutting down on the cluster's latency. It is important to note that to take the best advantage of direct connections in this fashion with more than two data nodes, you must have a direct connection between each data node and every other data node in the same node group.

20.3.4.9. Shared-Memory Connections

MySQL Cluster attempts to use the shared memory transporter and configure it automatically where possible. [shm] sections in the config.ini file explicitly define shared-memory connections between nodes in the cluster. When explicitly defining shared memory as the connection method, it is necessary to define at least NodeId1, NodeId2 and ShmKey. All other parameters have default values that should work well in most cases.

Important

SHM functionality is considered experimental only. It is not officially supported in any MySQL release series up to and including 5.1. This means that you must determine for yourself or by using our free resources (forums, mailing lists) whether it can be made to work correctly in your specific case.

NodeId1, NodeId2
To identify a connection between two nodes it is necessary to provide node identifiers for each of them, as NodeId1 and NodeId2.
ShmKey
When setting up shared memory segments, a node ID, expressed as an integer, is used to identify uniquely the shared memory segment to use for the communication. There is no default value.
ShmSize
Each SHM connection has a shared memory segment where messages between nodes are placed by the sender and read by the reader. The size of this segment is defined by ShmSize. The default value is 1MB.
SendSignalId
To retrace the path of a distributed message, it is necessary to provide each message with a unique identifier. Setting this parameter to Y causes these message IDs to be transported over the network as well. This feature is disabled by default in production builds, and enabled in -debug builds.
Checksum
This parameter is a boolean (Y/N) parameter which is disabled by default. When it is enabled, checksums for all messages are calculated before being placed in the send buffer.
This feature prevents messages from being corrupted while waiting in the send buffer. It also serves as a check against data being corrupted during transport.

20.3.4.10. SCI Transport Connections

[sci] sections in the config.ini file explicitly define SCI (Scalable Coherent Interface) connections between cluster nodes. Using SCI transporters in MySQL Cluster is supported only when the MySQL binaries are built using --with-ndb-sci=/your/path/to/SCI. The path should point to a directory that contains at a minimum lib and include directories containing SISCI libraries and header files. (See Section 20.13, “Using High-Speed Interconnects with MySQL Cluster” for more information about SCI.)

In addition, SCI requires specialized hardware.

It is strongly recommended to use SCI Transporters only for communication between ndbd processes. Note also that using SCI Transporters means that the ndbd processes never sleep. For this reason, SCI Transporters should be used only on machines having at least two CPUs dedicated for use by ndbd processes. There should be at least one CPU per ndbd process, with at least one CPU left in reserve to handle operating system activities.

NodeId1, NodeId2
To identify a connection between two nodes it is necessary to provide node identifiers for each of them, as NodeId1 and NodeId2.
Host1SciId0
This identifies the SCI node ID on the first Cluster node (identified by NodeId1).
Host1SciId1
It is possible to set up SCI Transporters for failover between two SCI cards which then should use separate networks between the nodes. This identifies the node ID and the second SCI card to be used on the first node.
Host2SciId0
This identifies the SCI node ID on the second Cluster node (identified by NodeId2).
Host2SciId1
When using two SCI cards to provide failover, this parameter identifies the second SCI card to be used on the second node.
SharedBufferSize
Each SCI transporter has a shared memory segment used for communication between the two nodes. Setting the size of this segment to the default value of 1MB should be sufficient for most applications. Using a smaller value can lead to problems when performing many parallel inserts; if the shared buffer is too small, this can also result in a crash of the ndbd process.
SendLimit
A small buffer in front of the SCI media stores messages before transmitting them over the SCI network. By default, this is set to 8KB. Our benchmarks show that performance is best at 64KB but 16KB reaches within a few percent of this, and there was little if any advantage to increasing it beyond 8KB.
SendSignalId
To trace a distributed message it is necessary to identify each message uniquely. When this parameter is set to Y, message IDs are transported over the network. This feature is disabled by default in production builds, and enabled in -debug builds.
Checksum
This parameter is a boolean value, and is disabled by default. When Checksum is enabled, checksums are calculated for all messages before they are placed in the send buffer. This feature prevents messages from being corrupted while waiting in the send buffer. It also serves as a check against data being corrupted during transport.

20.3.5. Overview of Cluster Configuration Parameters

20.3.5.1. Data Node Configuration Parameters
20.3.5.2. Management Node Configuration Parameters
20.3.5.3. SQL Node and API Node Configuration Parameters

The next three sections provide summary tables of MySQL Cluster configuration parameters used in the config.ini file to govern the cluster's functioning. Each table lists the parameters for one of the Cluster node process types (ndbd, ndb_mgmd, and mysqld), and includes the parameter's type as well as its default, minimum, and maximum values as applicable.

It is also stated what type of restart is required (node restart or system restart) — and whether the restart must be done with --initial — to change the value of a given configuration parameter. This information is provided in each table's Restart Type column, which contains one of the values shown in this list:

N: Node Restart
IN: Initial Node Restart
S: System Restart
IS: Initial System Restart

When performing a node restart or an initial node restart, all of the cluster's data nodes must be restarted in turn (also referred to as a rolling restart). It is possible to update cluster configuration parameters marked N or IN online — that is, without shutting down the cluster — in this fashion. An initial node restart requires restarting each ndbd process with the --initial option.

A system restart requires a complete shutdown and restart of the entire cluster. An initial system restart requires taking a backup of the cluster, wiping the cluster filesystem after shutdown, and then restoring from the backup following the restart.

In any cluster restart, all of the cluster's management servers must be restarted in order for them to read the updated configuration parameter values.

Important

Values for numeric cluster parameters can generally be increased without any problems, although it is advisable to do so progressively, making such adjustments in relatively small increments. However, decreasing the values of such parameters — particularly those relating to memory usage and disk space — is not to be undertaken lightly, and it is recommended that you do so only following careful planning and testing. In addition, it is the generally the case that parameters relating to memory and disk usage which can be raised using a simple node restart require an initial node restart to be lowered.

Because some of these parameters can be used for configuring more than one type of cluster node, they may appear in more than one of the tables.

(Note that 4294967039 — which often appears as a maximum value in these tables — is equal to 2³² – 2⁸ – 1.)

20.3.5.1. Data Node Configuration Parameters

The following table provides information about parameters used in the [ndbd] or [ndbd default] sections of a config.ini file for configuring MySQL Cluster data nodes. For detailed descriptions and other additional information about each of these parameters, see Section 20.3.4.5, “Defining Data Nodes”.

Restart Type Column Values

N: Node Restart
IN: Initial Node Restart
S: System Restart
IS: Initial System Restart

See Section 20.3.5, “Overview of Cluster Configuration Parameters”, for additional explanations of these abbreviations.

Parameter Name	Type/Units	Default Value	Minimum Value	Maximum Value	Restart Type
`ArbitrationTimeout`	milliseconds	3000	10	4294967039	N
`BackupDataBufferSize`	bytes	2M	0	4294967039	N
`BackupDataDir`	string	`FileSystemPath/BACKUP`	N/A	N/A	IN
`BackupLogBufferSize`	bytes	2M	0	4294967039	N
`BackupMemory`	bytes	4M	0	4294967039	N
`BackupReportFrequency` (Added in MySQL Cluster NDB 6.2.3)	seconds	0	0	4294967039	N
`BackupWriteSize`	bytes	32K	2K	4294967039	N
`BackupMaxWriteSize`	bytes	256K	2K	4294967039	N
`BatchSizePerLocalScan`	integer	64	1	992	N
`CompressedBackup` (Added in MySQL Cluster NDB 6.3.7)	boolean	0	0	1	N
`CompressedLCP` (Added in MySQL Cluster NDB 6.3.7)	boolean	0	0	1	N
`DataDir`	string	`/var/lib/mysql-cluster`	N/A	N/A	IN
`DataMemory`	bytes	80M	1M	1024G (subject to available system RAM and size of `IndexMemory`)	N
`DiskCheckpointSpeed` (added in MySQL 5.1.12)	integer (number of bytes per second)	10M	1M	4294967039	N
`DiskCheckpointSpeedInRestart` (added in MySQL 5.1.12)	integer (number of bytes per second)	100M	1M	4294967039	N
`Diskless`	true\|false (`1`\|`0`)	0	0	1	IS
`DiskPageBufferMemory` (added in MySQL 5.1.6)	bytes	64M	4M	1024G	N
`DiskSyncSize` (added in MySQL 5.1.12)	integer (number of bytes)	4M	32K	4294967039	N
`ExecuteOnComputer`	integer
`FileSystemPath` (Added in MySQL-MySQL Cluster NDB 6.1.11)	string	value specified for `DataDir`	N/A	N/A	IN
`FragmentLogFileSize`	integer	16M	4M	1G	IN
`HeartbeatIntervalDbApi`	milliseconds	1500	100	4294967039	N
`HeartbeatIntervalDbDb`	milliseconds	1500	10	4294967039	N
`HostName`	string	`localhost`	N/A	N/A	S
`Id`	integer	None	1	49	N
`IndexMemory`	bytes	18M	1M	1024G (subject to available system RAM and size of `DataMemory`)	N
`InitialNoOfOpenFiles`	integer	27	20	4294967039	N
`LockExecuteThreadToCPU` (Added in MySQL-MySQL Cluster NDB 6.3.4)	integer	65535	0	65535	N
`LockMaintThreadsToCPU` (Added in MySQL-MySQL Cluster NDB 6.3.4)	integer	65535	0	65535	N
`LockPagesInMainMemory`	As of MySQL 5.1.15 and MySQL Cluster NDB 6.1.1: integer; previously: true\|false (`1`\|`0`)	0	0	1	N
`LogLevelCheckpoint`	integer	0	0	15	IN
`LogLevelCongestion`	integer	0	0	15	N
`LogLevelConnection`	integer	0	0	15	N
`LogLevelError`	integer	0	0	15	N
`LogLevelInfo`	integer	0	0	15	N
`LogLevelNodeRestart`	integer	0	0	15	N
`LogLevelShutdown`	integer	0	0	15	N
`LogLevelStartup`	integer	1	0	15	N
`LogLevelStatistic`	integer	0	0	15	N
`LongMessageBuffer`	bytes	1M	512K	4294967039	N
`MaxAllocate` (Added in MySQL Cluster NDB 6.1.12 and MySQL MySQL Cluster NDB 6.2.3)	integer	32M	1M	1G	N
`MaxBufferedEpochs` (Added in MySQL Cluster NDB 6.2.14)	integer	100	0	100000	N
`MaxNoOfAttributes`	integer	1000	32	4294967039	N
`MaxNoOfConcurrentIndexOperations`	integer	8K	0	4294967039	N
`MaxNoOfConcurrentOperations`	integer	32768	32	4294967039	N
`MaxNoOfConcurrentScans`	integer	256	2	500	N
`MaxNoOfConcurrentTransactions`	integer	4096	32	4294967039	N
`MaxNoOfFiredTriggers`	integer	4000	0	4294967039	N
`MaxNoOfIndexes` (DEPRECATED — use `MaxNoOfOrderedIndexes` or `MaxNoOfUniqueHashIndexes` instead)	integer	128	0	4294967039	N
`MaxNoOfLocalOperations`	integer	`UNDEFINED`	32	4294967039	N
`MaxNoOfLocalScans`	integer	`UNDEFINED` (see description)	32	4294967039	N
`MaxNoOfOpenFiles`	integer	40	20	4294967039	N
`MaxNoOfOrderedIndexes`	integer	128	0	4294967039	N
`MaxNoOfSavedMessages`	integer	25	0	4294967039	N
`MaxNoOfTables`	integer	128	8	4294967039	N
`MaxNoOfTriggers`	integer	768	0	4294967039	N
`MaxNoOfUniqueHashIndexes`	integer	64	0	4294967039	N
`MemReportFrequency` (Added in MySQL 5.1.16 and MySQL Cluster NDB 6.1.0)	integer (seconds)	0	0	4294967039	N
`NoOfDiskPagesToDiskAfterRestartACC` (DEPRECATED as of MySQL 5.1.6)	integer (number of 8KB pages per 100 milliseconds)	20 (= 20 * 80KB = 1.6MB/second)	1	4294967039	N
`NoOfDiskPagesToDiskAfterRestartTUP` (DEPRECATED as of MySQL 5.1.6)	integer (number of 8KB pages per 100 milliseconds)	40 (= 40 * 80KB = 3.2MB/second)	1	4294967039	N
`NoOfDiskPagesToDiskDuringRestartACC` (DEPRECATED as of MySQL 5.1.6)	integer (number of 8KB pages per 100 milliseconds)	20 (= 20 * 80KB = 1.6MB/second)	1	4294967039	N
`NoOfDiskPagesToDiskDuringRestartTUP` (DEPRECATED as of MySQL 5.1.6)	integer (number of 8KB pages per 100 milliseconds)	40 (= 40 * 80KB = 3.2MB/second)	1	4294967039	N
`NoOfFragmentLogFiles`	integer	16	3	4294967039	IN
`NoOfReplicas`	integer	None	1	4 (theoretical); 2 (supported)	IS
`ODirect`	boolean	0	0	1	N
`RealTimeScheduler` (Added in MySQL Cluster NDB 6.3.4)	boolean	0	0	1	N
`RedoBuffer`	bytes	8M	1M	4294967039	N
`RestartOnErrorInsert` (DEBUG BUILDS ONLY)	true\|false (`1`\|`0`)	0	0	1	N
`SchedulerExecutionTimer` (added in MySQL Cluster NDB 6.3.4)	μseconds (integer)	50	0	11000	N
`SchedulerSpinTimer` (added in MySQL Cluster NDB 6.3.4)	μseconds (integer)	0	0	500	N
`ServerPort` (OBSOLETE)	integer	1186	0	4294967039	N
`SharedGlobalmemory` (added in MySQL 5.1.6)	bytes	20M	0	65536G	N
`StartFailureTimeout`	milliseconds	0	0	4294967039	N
`StartPartialTimeout`	milliseconds	30000	0	4294967039	N
`StartPartitionedTimeout`	milliseconds	60000	0	4294967039	N
`StopOnError`	true\|false (`1`\|`0`)	1	0	1	N
`StringMemory`	integer or percentage (see description for details)	0	0	4294967039	S
`TcpBind_INADDR_ANY` (Added in MySQL Cluster NDB 6.2.0)	true\|false (`1`\|`0`)	1	0	0	N
`TimeBetweenEpochs` (Added in MySQL Cluster NDB 6.2.5 and MySQL Cluster NDB 6.3.2)	milliseconds	100	0	32000	N
`TimeBetweenEpochsTimeout` (Added in MySQL Cluster NDB 6.2.7 and MySQL Cluster NDB 6.3.4)	milliseconds	4000	0	32000	N
`TimeBetweenGlobalCheckpoints`	milliseconds	2000	10	32000	N
`TimeBetweenInactiveTransactionAbortCheck`	milliseconds	1000	1000	4294967039	N
`TimeBetweenLocalCheckpoints`	integer (number of 4-byte words as a base-2 logarithm)	20 (= `4 * 2²⁰` = 4MB write operations)	0	31	N
`TimeBetweenWatchDogCheck`	milliseconds	6000	70	4294967039	N
`TimeBetweenWatchDogCheckInitial` (added in MySQL 5.1.20)	milliseconds	6000	70	4294967039	N
`TransactionBufferMemory`	bytes	1M	1K	4294967039	N
`TransactionDeadlockDetectionTimeout`	milliseconds	1200	50	4294967039	N
`TransactionInactiveTimeout`	milliseconds	0	0	4294967039	N
`UndoDataBuffer` (OBSOLETE)	bytes	16M	1M	4294967039	N
`UndoIndexBuffer` (OBSOLETE)	bytes	2M	1M	4294967039	N

20.3.5.2. Management Node Configuration Parameters

The following table provides information about parameters used in the [ndb_mgmd] or [mgm] sections of a config.ini file for configuring MySQL Cluster management nodes. For detailed descriptions and other additional information about each of these parameters, see Section 20.3.4.4, “Defining the Management Server”.

Restart Type Column Values

N: Node Restart
IN: Initial Node Restart
S: System Restart
IS: Initial System Restart

See Section 20.3.5, “Overview of Cluster Configuration Parameters”, for additional explanations of these abbreviations.

Parameter Name	Type/Units	Default Value	Minimum Value	Maximum Value	Restart Type
`ArbitrationDelay`	milliseconds	0	0	4294967039	N
`ArbitrationRank`	integer	1	0	2	N
`DataDir`	string	`./` (ndb_mgmd directory)	N/A	N/A	IN
`ExecuteOnComputer`	integer
`HostName`	string	`localhost`	N/A	N/A	IN
`Id`	integer	None	1	63	IN
`LogDestination`	`CONSOLE`, `SYSLOG`, or `FILE`	`FILE` (see Section 20.3.4.4, “Defining the Management Server”)	N/A	N/A	N
`PortNumber`	integer	1186	1	65535	S

20.3.5.3. SQL Node and API Node Configuration Parameters

The following table provides information about parameters used in the [SQL] and [api] sections of a config.ini file for configuring MySQL Cluster SQL nodes and API nodes. For detailed descriptions and other additional information about each of these parameters, see Section 20.3.4.6, “Defining SQL and Other API Nodes”.

Note

Restart Type Column Values

N: Node Restart
IN: Initial Node Restart
S: System Restart
IS: Initial System Restart

See Section 20.3.5, “Overview of Cluster Configuration Parameters”, for additional explanations of these abbreviations.

Parameter Name	Type/Units	Default Value	Minimum Value	Maximum Value	Restart Type
`ArbitrationDelay`	milliseconds	0	0	4294967039	N
`ArbitrationRank`	integer	0	0	2	N
`BatchByteSize`	bytes	32K	1K	1M	N
`BatchSize`	integer	64	1	992	N
`ExecuteOnComputer`	integer
`HostName`	string	none	N/A	N/A	IN
`Id`	integer	None	1	63	IN
`MaxScanBatchSize`	bytes	256K	32K	16M	N

20.3.6. Configuring Parameters for Local Checkpoints

The parameters discussed in Logging and Checkpointing and in Data Memory, Index Memory, and String Memory that are used to configure local checkpoints for a MySQL Cluster do not exist in isolation, but rather are very much interdepedent on each other. In this section, we illustrate how these parameters — including DataMemory, IndexMemory, NoOfDiskPagesToDiskAfterRestartTUP, NoOfDiskPagesToDiskAfterRestartACC, and NoOfFragmentLogFiles — relate to one another in a working Cluster.

Important

The parameters NoOfDiskPagesToDiskAfterRestartTUP and NoOfDiskPagesToDiskAfterRestartACC were deprecated in MySQL 5.1.6. From MySQL 5.1.6 through 5.1.11, disk writes during LCPs took place at the maximum speed possible. Beginning with MySQL 5.1.12, the speed and throughput for LCPs are controlled using the parameters DiskSyncSize, DiskCheckpointSpeed, and DiskCheckpointSpeedInRestart. See Section 20.3.4.5, “Defining Data Nodes”.

In this example, we assume that our application performs the following numbers of types of operations per hour:

50000 selects
15000 inserts
15000 updates
15000 deletes

We also make the following assumptions about the data used in the application:

We are working with a single table having 40 columns.
Each column can hold up to 32 bytes of data.
A typical UPDATE run by the application affects the values of 5 columns.
No NULL values are inserted by the application.

A good starting point is to determine the amount of time that should elapse between local checkpoints (LCPs). It worth noting that, in the event of a system restart, it takes 40-60 percent of this interval to execute the REDO log — for example, if the time between LCPs is 5 minutes (300 seconds), then it should take 2 to 3 minutes (120 to 180 seconds) for the REDO log to be read.

The maximum amount of data per node can be assumed to be the size of the DataMemory parameter. In this example, we assume that this is 2 GB. The NoOfDiskPagesToDiskAfterRestartTUP parameter represents the amount of data to be checkpointed per unit time — however, this parameter is actually expressed as the number of 8K memory pages to be checkpointed per 100 milliseconds. 2 GB per 300 seconds is approximately 6.8 MB per second, or 700 KB per 100 milliseconds, which works out to roughly 85 pages per 100 milliseconds.

Similarly, we can calculate NoOfDiskPagesToDiskAfterRestartACC in terms of the time for local checkpoints and the amount of memory required for indexes — that is, the IndexMemory. Assuming that we allow 512 MB for indexes, this works out to approximately 20 8-KB pages per 100 milliseconds for this parameter.

Next, we need to determine the number of REDO log files required — that is, fragment log files — the corresponding parameter being NoOfFragmentLogFiles. We need to make sure that there are sufficient REDO log files for keeping records for at least 3 local checkpoints. In a production setting, there are always uncertainties — for instance, we cannot be sure that disks always operate at top speed or with maximum throughput. For this reason, it is best to err on the side of caution, so we double our requirement and calculate a number of fragment log files which should be enough to keep records covering 6 local checkpoints.

It is also important to remember that the disk also handles writes to the REDO log, so if you find that the amount of data being written to disk as detemined by the values of NoOfDiskPagesToDiskAfterRestartACC and NoOfDiskPagesToDiskAfterRestartTUP is approaching the amount of disk bandwidth available, you may wish to increase the time between local checkpoints.

Given 5 minutes (300 seconds) per local checkpoint, this means that we need to support writing log records at maximum speed for 6 * 300 = 1800 seconds. The size of a REDO log record is 72 bytes plus 4 bytes per updated column value plus the maximum size of the updated column, and there is one REDO log record for each table record updated in a transaction, on each node where the data reside. Using the numbers of operations set out previously in this section, we derive the following:

50000 select operations per hour yields 0 log records (and thus 0 bytes), since SELECT statements are not recorded in the REDO log.
15000 DELETE statements per hour is approximately 5 delete operations per second. (Since we wish to be conservative in our estimate, we round up here and in the following calculations.) No columns are updated by deletes, so these statements consume only 5 operations * 72 bytes per operation = 360 bytes per second.
15000 UPDATE statements per hour is roughly the same as 5 updates per second. Each update uses 72 bytes, plus 4 bytes per column * 5 columns updated, plus 32 bytes per column * 5 columns — this works out to 72 + 20 + 160 = 252 bytes per operation, and multiplying this by 5 operation per second yields 1260 bytes per second.
15000 INSERT statements per hour is equivalent to 5 insert operations per second. Each insert requires REDO log space of 72 bytes, plus 4 bytes per record * 40 columns, plus 32 bytes per column * 40 columns, which is 72 + 160 + 1280 = 1512 bytes per operation. This times 5 operations per second yields 7560 bytes per second.

So the total number of REDO log bytes being written per second is approximately 0 + 360 + 1260 + 7560 = 9180 bytes. Multiplied by 1800 seconds, this yields 16524000 bytes required for REDO logging, or approximately 15.75 MB. The unit used for NoOfFragmentLogFiles represents a set of 4 16-MB log files — that is, 64 MB. Thus, the minimum value (3) for this parameter is sufficient for the scenario envisioned in this example, since 3 times 64 = 192 MB, or about 12 times what is required; the default value of 8 (or 512 MB) is more than ample in this case.

20.4. MySQL Cluster Options and Variables

20.4.1. MySQL Cluster Server Option and Variable Reference
20.4.2. MySQL Cluster-Related Command Options for mysqld
20.4.3. MySQL Cluster System Variables
20.4.4. MySQL Cluster Status Variables

This section provides information about MySQL server options, server and status variables that are specific to MySQL Cluster. For general information on using these, and for other options and variables not specific to MySQL Cluster, see Section 5.1, “The MySQL Server”.

For MySQL Cluster configuration parameters used in the cluster configuration file (usually named config.ini), see Section 20.3, “MySQL Cluster Configuration”.

20.4.1. MySQL Cluster Server Option and Variable Reference

The following table provides a list of the command-line options, server and status variables applicable within mysqld when it is running as an SQL node in a MySQL Cluster. For a table showing all command-line options, server and status variables available for use with mysqld, see Section 5.1.1, “Option and Variable Reference”.

Name	Cmd-Line	Option file	System Var	Status Var	Var Scope	Dynamic
Handler_discover				Yes	Both	No
have_ndbcluster			Yes		Global	No
ndb_autoincrement_prefetch_sz	Yes	Yes	Yes		Both	Yes
ndb_cache_check_time	Yes	Yes	Yes		Global	Yes
ndbcluster	Yes	Yes	Yes		Both	Yes
ndb-cluster-connection-pool	Yes	Yes		Yes	Global	No
Ndb_cluster_node_id				Yes	Both	No
Ndb_config_from_host				Yes	Both	No
Ndb_config_from_port				Yes	Both	No
Ndb_conflict_fn_max				Yes	Both	No
Ndb_conflict_fn_old				Yes	Both	No
ndb-connectstring	Yes	Yes
ndb_execute_count				Yes	Global	No
ndb_extra_logging	Yes	Yes	Yes		Global	Yes
ndb_force_send	Yes	Yes	Yes		Both	Yes
ndb_index_stat_cache_entries	Yes	Yes
ndb_index_stat_enable	Yes	Yes
ndb_index_stat_update_freq	Yes	Yes
ndb_log_orig			Yes		Global	No
ndb_log_update_as_write	Yes	Yes	Yes		Global	Yes
ndb_log_updated_only	Yes	Yes	Yes		Global	Yes
Ndb_number_of_data_nodes				Yes	Both	No
ndb_optimized_node_selection	Yes	Yes
ndb_report_thresh_binlog_epoch_slip	Yes	Yes
ndb_report_thresh_binlog_mem_usage	Yes	Yes
ndb_use_copying_alter_table			Yes		Both	No
ndb_use_exact_count			Yes		Both	Yes
ndb_use_transactions	Yes	Yes
ndb_wait_connected	Yes	Yes	Yes			No
skip-ndbcluster	Yes	Yes
slave-allow-batching	Yes				Global	Yes
- Variable: slave_allow_batching			Yes		Global	Yes

20.4.2. MySQL Cluster-Related Command Options for mysqld

This section provides descriptions of mysqld server options relating to MySQL Cluster. For information about mysqld options not specific to MySQL Cluster, and for general information about the use of options with mysqld, see Section 5.1.2, “Command Options”.

For information about command-line options used with other MySQL Cluster processes (ndbd, ndb_mgmd, and ndb_mgm), see Section 20.6.5, “Command Options for MySQL Cluster Processes”. For information about command-line options used with NDB utility programs (such as ndb_desc, ndb_size.pl, and ndb_show_tables), see Section 20.10, “Cluster Utility Programs”.

--ndb-cluster-connection-pool=#
By setting this option to a value greater than 1 (the default), a mysqld process can use multiple connections to the cluster, effectively mimicking several SQL nodes. Each connection requires its own [api] or [mysqld] section in the cluster configuration (config.ini) file, and counts against the maximum number of API connections supported by the cluster. For example, suppose that you have 2 cluster host computers, each running an SQL node whose mysqld process was started with --ndb-cluster-connection-pool=4; this means that the cluster must have 8 API slots available for these connections (instead of 2). All of these connections are set up when the SQL node connects to the cluster, and are allocated to threads in a round-robin fashion.
Important
This option is useful only when running mysqld on host machines having multiple CPUs, multiple cores, or both. For best results, the value should be smaller than the total number of cores available on the host machine. Setting it to a value greater than this is likely to degrade performance severely.
Beginning with MySQL Cluster NDB 6.2.16 and MySQL Cluster NDB 6.3.13, the value used for this option is available as a global status variable. (Bug#35573)
--ndb-connectstring=connect_string
When using the NDBCLUSTER storage engine, this option specifies the management server that distributes cluster configuration data.
--ndbcluster
The NDB (or NDBCLUSTER) storage engine is necessary for using MySQL Cluster. If a mysqld binary includes support for the NDB storage engine, the engine is disabled by default. Use the --ndbcluster option to enable it. Use --skip-ndbcluster to explicitly disable the engine.
--skip-ndbcluster
Disable the NDB storage engine. This is the default for binaries that were built with NDB storage engine support; the server allocates memory and other resources for this storage engine only if the --ndbcluster option is given explicitly. See Section 20.3.3, “Quick Test Setup of MySQL Cluster”, for an example of usage.

20.4.3. MySQL Cluster System Variables

This section provides detailed information about MySQL server system variables that are specific to MySQL Cluster and the NDB storage engine. For system variables not specific to MySQL Cluster, see Section 5.1.3, “System Variables”. For general information on using system variables, see Section 5.1.5, “Using System Variables”.

have_ndbcluster
Variable Name have_ndbcluster
Variable Scope Global
Dynamic Variable No
Value Set
Type boolean
YES if mysqld supports NDB tables. DISABLED if --skip-ndbcluster is used.
multi_range_count
Option Sets Variable Yes, multi_range_count
Variable Name multi_range_count
Variable Scope Both
Dynamic Variable Yes
Value Set
Type numeric
Default 256
Range 1-4294967295
The maximum number of ranges to send to a table handler at once during range selects. The default value is 256. Sending multiple ranges to a handler at once can improve the performance of certain selects dramatically. This is especially true for the NDB table handler, which needs to send the range requests to all nodes. Sending a batch of those requests at once reduces communication costs significantly.
This variable is deprecated in MySQL 5.1, and is no longer supported in MySQL 6.0, in which arbitrarily long lists of ranges can be processed.

ndb_autoincrement_prefetch_sz

Option Sets Variable Yes, ndb_autoincrement_prefetch_sz

Variable Name ndb_autoincrement_prefetch_sz

Variable Scope Both

Dynamic Variable Yes

Value Set (<= 5.1.22)

Type	numeric
Default	`32`
Range	1-256

Value Set (>= 5.1.23)

Type	numeric
Default	`1`
Range	1-256

Determines the probability of gaps in an autoincremented column. Set it to 1 to minimize this. Setting it to a high value for optimization — makes inserts faster, but decreases the likelihood that consecutive autoincrement numbers will be used in a batch of inserts. Default value: 32. Minimum value: 1.

Beginning with MySQL Cluster NDB 6.2.10, MySQL Cluster NDB 6.3.7, and MySQL 5.1.23, this variable affects the number of AUTO_INCREMENT IDs that are fetched between statements only. Within a statement, at least 32 IDs are now obtained at a time. The default value for ndb_autoincrement_prefetch_sz is now 1, to increase the speed of statements inserting single rows. (Bug#31956)

ndb_cache_check_time
Option Sets Variable Yes, ndb_cache_check_time
Variable Name ndb_cache_check_time
Variable Scope Global
Dynamic Variable Yes
Value Set
Type numeric
Default 0
The number of milliseconds that elapse between checks of MySQL Cluster SQL nodes by the MySQL query cache. Setting this to 0 (the default and minimum value) means that the query cache checks for validation on every query.
The recommended maximum value for this variable is 1000, which means that the check is performed once per second. A larger value means that the check is performed and possibly invalidated due to updates on different SQL nodes less often. It is generally not desirable to set this to a value greater than 2000.
ndb_extra_logging
Version Introduced 5.1.6
Variable Name ndb_extra_logging
Variable Scope Global
Dynamic Variable Yes
Value Set
Type numeric
Default 0
This variable can be used to enable recording in the MySQL error log of information specific to the NDB storage engine. It is normally of interest only when debugging NDB storage engine code.
The default value is 0, which means that the only NDB-specific information written to the MySQL error log relates to transaction handling. If the value is greater than 0 but less than 10, NDB table schema and connection events are also logged, as well as whether or not conflict resolution is in use, and other NDB errors and information. If the value is set to 10 or more, information about NDB internals, such as the progress of data distribution among cluster nodes, is also written to the MySQL error log.
This variable was added in MySQL 5.1.6.
ndb_force_send
Option Sets Variable Yes, ndb_force_send
Variable Name ndb_force_send
Variable Scope Both
Dynamic Variable Yes
Value Set
Type boolean
Default TRUE
Forces sending of buffers to NDB immediately, without waiting for other threads. Defaults to ON.
ndb_index_stat_cache_entries
Value Set
Type numeric
Default 32
Range 0-4294967295
Sets the granularity of the statistics by determining the number of starting and ending keys to store in the statistics memory cache. Zero means no caching takes place; in this case, the data nodes are always queried directly. Default value: 32.
ndb_index_stat_enable
Value Set
Type boolean
Default ON
Use NDB index statistics in query optimization. Defaults to ON.
ndb_index_stat_update_freq
Value Set
Type numeric
Default 20
Range 0-4294967295
How often to query data nodes instead of the statistics cache. For example, a value of 20 (the default) means to direct every 20^th query to the data nodes.
ndb_optimized_node_selection
Value Set
Type boolean
Default ON
Causes an SQL node to use a data node on the same host machine as transaction coordinator. Enabled by default. Set to 0 or OFF to disable, in which case the SQL node uses each data node in the cluster in succession. When this option is disabled, or if there is no data node process running on the same host as the SQL node, the SQL node attempts to use a given data node 8 times before proceeding to the next one.
Begiunning with MySQL Cluster NDB 6.3.4, this option takes one of the integer values 0, 1, 2, or 3, with 1 being the default. These values affect node selection as follows:
- 0: Each data node is employed as the transaction coordinator 8 times before the SQL node proceeds to the next data node. (This is the same behavior as caused by setting this option to 0 or OFF in previous MySQL versions.)
- 1: If a data node process is running on the the same host as the SQL node, this data node is used as the transaction coordinator. (This is the same behavior as caused by setting this option to 1 or ON in previous MySQL versions.)
- 2: The SQL node follows the same behavior as if this option had been set to 1; however, the setting is global.
- 3: The data node housing the cluster partition accessed by the first statement of a given transaction is used as the transaction coordinator for the entire transaction. This is effective only if the first statement of the transaction accesses no more than one cluster partition; otherwise, the SQL node reverts to the round-robin behavior seen when this option is set to 0.
Important
Beginning with MySQL Cluster NDB 6.3.4, it is no longer possible to set --ndb_optimized_node_selection to ON or OFF; attempting to do so causes mysqld to abort with an error.
ndb_report_thresh_binlog_epoch_slip
Value Set
Type numeric
Default 3
Range 0-256
This is a threshold on the number of epochs to be behind before reporting binlog status. For example, a value of 3 (the default) means that if the difference between which epoch has been received from the storage nodes and which epoch has been applied to the binlog is 3 or more, a status message will be sent to the cluster log.
ndb_report_thresh_binlog_mem_usage
Value Set
Type numeric
Default 10
Range 0-10
This is a threshold on the percentage of free memory remaining before reporting binlog status. For example, a value of 10 (the default) means that if the amount of available memory for receiving binlog data from the data nodes falls below 10%, a status message will be sent to the cluster log.
ndb_use_copying_alter_table
Version Introduced 5.1.12
Variable Name ndb_use_copying_alter_table
Variable Scope Both
Dynamic Variable No
Forces NDB to use copying of tables in the event of problems with online ALTER TABLE operations. The default value is OFF.
This variable was added in MySQL 5.1.12.
ndb_use_exact_count
Variable Name ndb_use_exact_count
Variable Scope Both
Dynamic Variable Yes
Value Set
Type boolean
Default ON
Forces NDB to use a count of records during SELECT COUNT(*) query planning to speed up this type of query. The default value is ON. For faster queries overall, disable this feature by setting the value of ndb_use_exact_count to OFF.
ndb_use_transactions
Value Set
Type boolean
Default ON
You can disable NDB transaction support by setting this variable's values to OFF (not recommended). The default is ON.
ndb_wait_connected
Version Introduced 5.1.16-ndb-6.2.0
Option Sets Variable Yes, ndb_wait_connected
Variable Name ndb_wait_connected
Variable Scope
Dynamic Variable No
Value Set
Type numeric
Default 0
This variable can be used to cause the MySQL server to wait a given period of time for connections to MySQL Cluster management and data nodes to be established before accepting MySQL client connections. The time is specified in seconds. The default value is 0.

20.4.4. MySQL Cluster Status Variables

This section provides detailed information about MySQL server status variables that relate to MySQL Cluster and the NDB storage engine. For status variables not specific to MySQL Cluster, and for general information on using status variables, see Section 5.1.6, “Status Variables”.

Handler_discover
The MySQL server can ask the NDBCLUSTER storage engine if it knows about a table with a given name. This is called discovery. Handler_discover indicates the number of times that tables have been discovered via this mechanism.
Ndb_cluster_node_id
If the server is acting as a MySQL Cluster node, then the value of this variable its node ID in the cluster.
If the server is not part of a MySQL Cluster, then the value of this variable is 0.
Ndb_config_from_host
If the server is part of a MySQL Cluster, the value of this variable is the hostname or IP address of the Cluster management server from which it gets its configuration data.
If the server is not part of a MySQL Cluster, then the value of this variable is an empty string.
Prior to MySQL 5.1.12, this variable was named Ndb_connected_host.
Ndb_config_from_port
If the server is part of a MySQL Cluster, the value of this variable is the number of the port through which it is connected to the Cluster management server from which it gets its configuration data.
If the server is not part of a MySQL Cluster, then the value of this variable is 0.
Prior to MySQL 5.1.12, this variable was named Ndb_connected_port.
Ndb_execute_count
Provides the number of round trips to the NDB kernel made by operations. Added in MySQL Cluster NDB 6.3.6.
Ndb_number_of_data_nodes
If the server is part of a MySQL Cluster, the value of this variable is the number of data nodes in the cluster.
If the server is not part of a MySQL Cluster, then the value of this variable is 0.
Prior to MySQL 5.1.12, this variable was named Ndb_number_of_storage_nodes.
Slave_heartbeat_period
Shows the replication heartbeat interval (in seconds) on a replication slave.
This variable was added in MySQL Cluster NDB 6.3.4.
Slave_received_heartbeats
This counter increments with each replication heartbeat received by a replication slave since the last time that the slave was restarted or reset, or a CHANGE MASTER statement was issued.
This variable was added in MySQL Cluster NDB 6.3.4.

20.5. Upgrading and Downgrading MySQL Cluster

20.5.1. Performing a Rolling Restart of the Cluster
20.5.2. MySQL Cluster 5.1 and MySQL Cluster NDB 6.x Upgrade and Downgrade Compatibility

This portion of the MySQL Cluster chapter covers upgrading and downgrading a MySQL Cluster from one MySQL release to another. It discusses different types of Cluster upgrades and downgrades, and provides a Cluster upgrade/downgrade compatibility matrix (see Section 20.5.2, “MySQL Cluster 5.1 and MySQL Cluster NDB 6.x Upgrade and Downgrade Compatibility”). You are expected already to be familiar with installing and configuring a MySQL Cluster prior to attempting an upgrade or downgrade. See Section 20.3, “MySQL Cluster Configuration”.

For information about upgrading or downgrading between MySQL Cluster NDB releases, or between MySQL Cluster NDB releases and mainline MySQL releases, see the changelogs relating to MySQL Cluster NDB.

This section remains in development, and continues to be updated and expanded.

20.5.1. Performing a Rolling Restart of the Cluster

This section discusses how to perform a rolling restart of a MySQL Cluster installation, so called because it involves stopping and starting (or restarting) each node in turn, so that the cluster itself remains operational. This is often done as part of a rolling upgrade or rolling downgrade, where high availability of the cluster is mandatory and no downtime of the cluster as a whole is permissible. Where we refer to upgrades, the information provided here also generally applies to downgrades as well.

There are a number of reasons why a rolling restart might be desirable:

Cluster configuration change. To make a change in the cluster's configuration, such as adding an SQL node to the cluster, or setting a configuration parameter to a new value.
Cluster software upgrade/downgrade. To upgrade the cluster to a newer version of the MySQL Cluster software (or to downgrade it to an older version). This is usually referred to as a “rolling upgrade” (or “rolling downgrade”, when reverting to an older version of MySQL Cluster).
Change on node host. To make changes in the hardware or operating system on which one or more cluster nodes are running
Cluster reset. To reset the cluster because it has reached an undesirable state
Freeing of resources. To allow memory allocated to a table by successive INSERT and DELETE operations to be freed for re-use by other Cluster tables

The process for performing a rolling restart may be generalised as follows:

Stop all cluster management nodes (ndb_mgmd processes), reconfigure them, then restart them
Stop, reconfigure, then restart each cluster data node (ndbd process) in turn
Stop, reconfigure, then restart each cluster SQL node (mysqld process) in turn

The specifics for implementing a particular rolling upgrade depend upon the actual changes being made. A more detailed view of the process is presented here:

MySQL Cluster Rolling Restarts (By
Type)

In the previous diagram, Stop and Start steps indicate that the process must be stopped completely using a shell command (such as kill on most Unix systems) or the management client STOP command, then started again from a system shell by invoking the ndbd or ndb_mgmd executable as appropriate. Restart indicates the process may be restarted using the ndb_mgm management client RESTART command.

Important

When performing an upgrade or downgrade of the cluster software, you must upgrade or downgrade the management nodes first, then the data nodes, and finally the SQL nodes. Doing so in any other order may leave the cluster in an unusable state.

20.5.2. MySQL Cluster 5.1 and MySQL Cluster NDB 6.x Upgrade and Downgrade Compatibility

This section provides information about MySQL Cluster software and table file compatibility between MySQL 5.1 and MySQL Cluster NDB 6.x releases with regard to performing upgrades and downgrades.

Important

Only compatibility between MySQL versions with regard to NDBCLUSTER is taken into account in this section, and there are likely other issues to be considered. As with any other MySQL software upgrade or downgrade, you are strongly encouraged to review the relevant portions of the MySQL Manual for the MySQL versions from which and to which you intend to migrate, before attempting an upgrade or downgrade of the MySQL Cluster software. See Section 2.11, “Upgrading MySQL”.

The following table shows Cluster upgrade and downgrade compatibility between different releases of MySQL 5.1:

MySQL Cluster upgrade/downgrade compatibility,
MySQL 5.1

Notes — MySQL 5.1.

MySQL 5.1.3 was the first public release in this series.
Direct upgrades or downgrades between MySQL Cluster 5.0 and 5.1 are not supported; you must dump all NDBCLUSTER tables using mysqldump, install the new version of the software, and then reload the tables from the dump.
You cannot downgrade a MySQL 5.1.6 or later Cluster using Disk Data tables to MySQL 5.1.5 or earlier unless you convert all such tables to in-memory Cluster tables first.
MySQL 5.1.8, MySQL 5.1.10, and MySQL 5.1.13 were not released.
Online cluster upgrades and downgrades between MySQL 5.1.11 (or an earlier version) and 5.1.12 (or a later version) are not possible due to major changes in the cluster filesystem. In such cases, you must perform a backup or dump, upgrade (or downgrade) the software, start each data node with --initial, and then restore from the backup or dump. You can use NDB backup/restore or mysqldump for this purpose.
Online downgrades from MySQL 5.1.14 or later to versions previous to 5.1.14 are not supported due to incompatible changes in the cluster system tables.
Online upgrades from MySQL 5.1.17 and earlier to 5.1.18 and later are not supported for clusters using replication due to incompatible changes in the mysql.ndb_apply_status table. However, it should not be necessary to shut down the cluster entirely, if you follow this modified rolling restart procedure:
1. Stop the management server, update the ndb_mgmd binary, then start it again. For multiple management servers, repeat this step for each management server in turn.
2. For each data node in turn: Stop the data node, replace the ndbd binary with the new version, then restart the data node. It is not necessary to use --initial when restarting any of the data nodes.
3. Stop all SQL nodes. Replace the mysqld binary with the new version for all SQL nodes, then restart them. It is not necessary to start them one at a time, but they must all be shut down at the same time before starting any of them again using the 5.1.18 (or later) mysqld. Otherwise — due to the fact that mysql.ndb_apply_status uses the NDB storage engine and is thus shared between all SQL nodes — there may be conflicts between MySQL servers using the old and new versions of the table.
You can find more information about the changes to ndb_apply_status in Section 20.11.4, “Cluster Replication Schema and Tables”.
The internal specifications for columns in NDB tables changed in MySQL 5.1.18 to allow compatibility with future MySQL Cluster releases that are expected to implement online adding and dropping of columns. This change is not backwards compatible with earlier MySQL versions.
In order to make tables created in MySQL 5.1.17 and earlier compatible with online adding and dropping of columns when this features becomes available, it is necessary force MySQL 5.1.18 and later to convert the tables to the new format by following this procedure:
1. Upgrade the MySQL Cluster software on all data, management, and SQL nodes
2. Back up all NDB tables
3. Shut down the cluster (all data, management, and SQL nodes)
4. Restart the cluster, starting all data nodes with the --initial option (to clear and rebuild the data node filesystems)
5. Restore the tables from backup
This is not necessary for NDB tables created in MySQL 5.1.18 and later; such tables will automatically be compatible with online adding and dropping of columns (as implemented beginning with MySQL Cluster NDB 6.2.5 and MySQL Cluster NDB 6.3.2).
In order to minimise possible later difficulties, it is strongly advised that the procedure outlined above be followed as soon as possible after to upgrading from MySQL 5.1.17 or earlier to MySQL 5.1.18 or later.
Information about how this change effects users of MySQL Cluster NDB 6.x is provided later in this section.
MySQL Cluster is not supported in standard MySQL 5.1 releases, beginning with MySQL 5.1.25. If you are using MySQL Cluster in a standard MySQL 5.1 release, you should upgrade to the most recent MySQL Cluster NDB 6.2 or 6.3 release.

The following table shows Cluster upgrade and downgrade compatibility between different releases of MySQL Cluster NDB 6.x:

MySQL Cluster upgrade/downgrade compatibility,
MySQL Cluster NDB 6.x

Notes — MySQL Cluster NDB 6.x.

MySQL Cluster NDB 6.1 is no longer in production; if you are still using a MySQL Cluster NDB 6.1 release, you should upgrade to the most recent MySQL Cluster NDB 6.2 or 6.3 as soon as possible.
It is not possible to upgrade from MySQL Cluster NDB 6.1.2 (or an older 6.1 release) directly to 6.1.4 or a newer NDB 6.1 release, or to downgrade from 6.1.4 (or a newer 6.1 release) directly to 6.1.2 or an older NDB 6.1 release; in either case, you must upgrade or downgrade to MySQL Cluster NDB 6.1.3 first.
It is not possible to perform an online downgrade from MySQL Cluster NDB 6.1.8 (or a newer 6.1 release) to MySQL Cluster NDB 6.1.7 (or an older 6.1 release).
MySQL Cluster NDB 6.1.6 and 6.1.18 were not released.
It is not possible to perform an online upgrade or downgrade between MySQL Cluster NDB 6.2 and any previous release series (including mainline MySQL 5.1 and MySQL Cluster NDB 6.1); it is necessary to perform a dump and reload. However, it should be possible to perform online upgrades or downgrades between any MySQL Cluster NDB 6.2 release and any MySQL Cluster NDB 6.3 release up to and including 6.3.7.
The internal specifications for columns in NDB tables changed in MySQL Cluster NDB 6.1.17 and 6.2.1 to allow compatibility with future MySQL Cluster releases that are expected to implement online adding and dropping of columns. This change is not backwards compatible with earlier MySQL or MySQL Cluster NDB 6.x versions.
In order to make tables created in earlier versions compatible with online adding and dropping of columns in later versions, it is necessary to force MySQL Cluster to convert the tables to the new format by following this procedure following an upgrade:
1. Upgrade the MySQL Cluster software on all data, management, and SQL nodes
2. Back up all NDB tables
3. Shut down the cluster (all data, management, and SQL nodes)
4. Restart the cluster, starting all data nodes with the --initial option (to clear and rebuild the data node filesystems)
5. Restore the tables from backup
In order to minimise possible later difficulties, it is strongly advised that the procedure outlined above be followed as soon as possible after to upgrading between the versions indicated. The procedure is not necessary for NDBCLUSTER tables created in any of the following versions:
- MySQL Cluster NDB 6.1.8 or a later MySQL Cluster NDB 6.1 release
- MySQL Cluster 6.2.1 or a later MySQL Cluster NDB 6.2 release
- Any MySQL Cluster NDB 6.3 release
Tables created in the listed versions (or later ones, as indicated) are already compatible with online adding and dropping of columns (as implemented beginning with MySQL Cluster NDB 6.2.5 and MySQL Cluster NDB 6.3.2).
Online downgrades between MySQL Cluster NDB 6.3.8 and earlier releases are not supported.

20.6. Process Management in MySQL Cluster

20.6.1. MySQL Server Process Usage for MySQL Cluster
20.6.2. ndbd — The Storage Engine Node Process
20.6.3. ndb_mgmd — The Management Server Process
20.6.4. ndb_mgm — The Management Client Process
20.6.5. Command Options for MySQL Cluster Processes

Understanding how to manage MySQL Cluster requires a knowledge of four essential processes. In the next few sections of this chapter, we cover the roles played by these processes in a cluster, how to use them, and what startup options are available for each of them:

20.6.1. MySQL Server Process Usage for MySQL Cluster

mysqld is the traditional MySQL server process. To be used with MySQL Cluster, mysqld needs to be built with support for the NDBCLUSTER storage engine, as it is in the precompiled binaries available from http://dev.mysql.com/downloads/. If you build MySQL from source, you must invoke configure with the --with-ndbcluster option to enable NDB Cluster storage engine support.

If the mysqld binary has been built with Cluster support, the NDBCLUSTER storage engine is still disabled by default. You can use either of two possible options to enable this engine:

Use --ndbcluster as a startup option on the command line when starting mysqld.
Insert a line containing ndbcluster in the [mysqld] section of your my.cnf file.

An easy way to verify that your server is running with the NDBCLUSTER storage engine enabled is to issue the SHOW ENGINES statement in the MySQL Monitor (mysql). You should see the value YES as the Support value in the row for NDBCLUSTER. If you see NO in this row or if there is no such row displayed in the output, you are not running an NDB-enabled version of MySQL. If you see DISABLED in this row, you need to enable it in either one of the two ways just described.

To read cluster configuration data, the MySQL server requires at a minimum three pieces of information:

The MySQL server's own cluster node ID
The hostname or IP address for the management server (MGM node)
The number of the TCP/IP port on which it can connect to the management server

Node IDs can be allocated dynamically, so it is not strictly necessary to specify them explicitly.

The mysqld parameter ndb-connectstring is used to specify the connectstring either on the command line when starting mysqld or in my.cnf. The connectstring contains the hostname or IP address where the management server can be found, as well as the TCP/IP port it uses.

In the following example, ndb_mgmd.mysql.com is the host where the management server resides, and the management server listens for cluster messages on port 1186:

shell> mysqld --ndbcluster --ndb-connectstring=ndb_mgmd.mysql.com:1186

See Section 20.3.4.2, “The Cluster Connectstring”, for more information on connectstrings.

Given this information, the MySQL server will be a full participant in the cluster. (We often refer to a mysqld process running in this manner as an SQL node.) It will be fully aware of all cluster data nodes as well as their status, and will establish connections to all data nodes. In this case, it is able to use any data node as a transaction coordinator and to read and update node data.

You can see in the mysql client whether a MySQL server is connected to the cluster using SHOW PROCESSLIST. If the MySQL server is connected to the cluster, and you have the PROCESS privilege, then the first row of the output is as shown here:

mysql> SHOW PROCESSLIST \G
*************************** 1. row ***************************
     Id: 1
   User: system user
   Host:
     db:
Command: Daemon
   Time: 1
  State: Waiting for event from ndbcluster
   Info: NULL

Important

To participate in a MySQL Cluster, the mysqld process must be started with both the options --ndbcluster and --ndb-connectstring (or their equivalents in my.cnf). If mysqld is started with only the --ndbcluster option, or if it is unable to contact the cluster, it is not possible to work with NDB tables, nor is it possible to create any new tables regardless of storage engine. The latter restriction is a safety measure intended to prevent the creation of tables having the same names as NDB tables while the SQL node is not connected to the cluster. If you wish to create tables using a different storage engine while the mysqld process is not participating in a MySQL Cluster, you must restart the server without the --ndbcluster option.

20.6.2. ndbd — The Storage Engine Node Process

ndbd is the process that is used to handle all the data in tables using the NDB Cluster storage engine. This is the process that empowers a data node to accomplish distributed transaction handling, node recovery, checkpointing to disk, online backup, and related tasks.

In a MySQL Cluster, a set of ndbd processes cooperate in handling data. These processes can execute on the same computer (host) or on different computers. The correspondences between data nodes and Cluster hosts is completely configurable.

ndbd generates a set of log files which are placed in the directory specified by DataDir in the config.ini configuration file.

These log files are listed below. node_id is the node's unique identifier. Note that node_id represents the node's unique identifier. For example, ndb_2_error.log is the error log generated by the data node whose node ID is 2.

ndb_node_id_error.log is a file containing records of all crashes which the referenced ndbd process has encountered. Each record in this file contains a brief error string and a reference to a trace file for this crash. A typical entry in this file might appear as shown here:
```
Date/Time: Saturday 30 July 2004 - 00:20:01
Type of error: error
Message: Internal program error (failed ndbrequire)
Fault ID: 2341
Problem data: DbtupFixAlloc.cpp
Object of reference: DBTUP (Line: 173)
ProgramName: NDB Kernel
ProcessID: 14909
TraceFile: ndb_2_trace.log.2
***EOM***
```
Listings of possible ndbd exit codes and messages generated when a data node process shuts down prematurely can be found in ndbd Error Messages.
Important
The last entry in the error log file is not necessarily the newest one (nor is it likely to be). Entries in the error log are not listed in chronological order; rather, they correspond to the order of the trace files as determined in the ndb_node_id_trace.log.next file (see below). Error log entries are thus overwritten in a cyclical and not sequential fashion.
ndb_node_id_trace.log.trace_id is a trace file describing exactly what happened just before the error occurred. This information is useful for analysis by the MySQL Cluster development team.
It is possible to configure the number of these trace files that will be created before old files are overwritten. trace_id is a number which is incremented for each successive trace file.
ndb_node_id_trace.log.next is the file that keeps track of the next trace file number to be assigned.
ndb_node_id_out.log is a file containing any data output by the ndbd process. This file is created only if ndbd is started as a daemon, which is the default behavior.
ndb_node_id.pid is a file containing the process ID of the ndbd process when started as a daemon. It also functions as a lock file to avoid the starting of nodes with the same identifier.
ndb_node_id_signal.log is a file used only in debug versions of ndbd, where it is possible to trace all incoming, outgoing, and internal messages with their data in the ndbd process.

It is recommended not to use a directory mounted through NFS because in some environments this can cause problems whereby the lock on the .pid file remains in effect even after the process has terminated.

To start ndbd, it may also be necessary to specify the hostname of the management server and the port on which it is listening. Optionally, one may also specify the node ID that the process is to use.

shell> ndbd --connect-string="nodeid=2;host=ndb_mgmd.mysql.com:1186"

See Section 20.3.4.2, “The Cluster Connectstring”, for additional information about this issue. Section 20.6.5, “Command Options for MySQL Cluster Processes”, describes other options for ndbd.

When ndbd starts, it actually initiates two processes. The first of these is called the “angel process”; its only job is to discover when the execution process has been completed, and then to restart the ndbd process if it is configured to do so. Thus, if you attempt to kill ndbd via the Unix kill command, it is necessary to kill both processes, beginning with the angel process. The preferred method of terminating an ndbd process is to use the management client and stop the process from there.

The execution process uses one thread for reading, writing, and scanning data, as well as all other activities. This thread is implemented asynchronously so that it can easily handle thousands of concurrent activites. In addition, a watch-dog thread supervises the execution thread to make sure that it does not hang in an endless loop. A pool of threads handles file I/O, with each thread able to handle one open file. Threads can also be used for transporter connections by the transporters in the ndbd process. In a multi-processor system performing a large number of operations (including updates), the ndbd process can consume up to 2 CPUs if permitted to do so.

For a machine with many CPUs it is possible to use several ndbd processes which belong to different node groups; however, such a configuration is still considered experimental and is not supported for MySQL 5.1 in a production setting. See Section 20.14, “Known Limitations of MySQL Cluster”.

20.6.3. ndb_mgmd — The Management Server Process

The management server is the process that reads the cluster configuration file and distributes this information to all nodes in the cluster that request it. It also maintains a log of cluster activities. Management clients can connect to the management server and check the cluster's status.

It is not strictly necessary to specify a connectstring when starting the management server. However, if you are using more than one management server, a connectstring should be provided and each node in the cluster should specify its node ID explicitly.

See Section 20.3.4.2, “The Cluster Connectstring”, for information about using connectstrings. Section 20.6.5, “Command Options for MySQL Cluster Processes”, describes other options for ndb_mgmd.

The following files are created or used by ndb_mgmd in its starting directory, and are placed in the DataDir as specified in the config.ini configuration file. In the list that follows, node_id is the unique node identifier.

config.ini is the configuration file for the cluster as a whole. This file is created by the user and read by the management server. Section 20.3, “MySQL Cluster Configuration”, discusses how to set up this file.
ndb_node_id_cluster.log is the cluster events log file. Examples of such events include checkpoint startup and completion, node startup events, node failures, and levels of memory usage. A complete listing of cluster events with descriptions may be found in Section 20.7, “Management of MySQL Cluster”.
When the size of the cluster log reaches one million bytes, the file is renamed to ndb_node_id_cluster.log.seq_id, where seq_id is the sequence number of the cluster log file. (For example: If files with the sequence numbers 1, 2, and 3 already exist, the next log file is named using the number 4.)
ndb_node_id_out.log is the file used for stdout and stderr when running the management server as a daemon.
ndb_node_id.pid is the process ID file used when running the management server as a daemon.

20.6.4. ndb_mgm — The Management Client Process

The ndb_mgm management client process is actually not needed to run the cluster. Its value lies in providing a set of commands for checking the cluster's status, starting backups, and performing other administrative functions. The management client accesses the management server using a C API. Advanced users can also employ this API for programming dedicated management processes to perform tasks similar to those performed by ndb_mgm.

To start the management client, it is necessary to supply the hostname and port number of the management server:

shell> ndb_mgm [host_name [port_num]]

For example:

shell> ndb_mgm ndb_mgmd.mysql.com 1186

The default hostname and port number are localhost and 1186, respectively.

Additional information about using ndb_mgm can be found in Section 20.6.5.3, “Command Options for ndb_mgm”, and Section 20.7.2, “Commands in the MySQL Cluster Management Client”.

20.6.5. Command Options for MySQL Cluster Processes

20.6.5.1. Command Options for ndbd
20.6.5.2. Command Options for ndb_mgmd
20.6.5.3. Command Options for ndb_mgm

All MySQL Cluster executables (except for mysqld) take the options described in this section. Users of earlier MySQL Cluster versions should note that some of these options have been changed to make them consistent with one another as well as with mysqld. You can use the --help option with any MySQL Cluster executable to view a list of the options which it supports.

The following options are common to all MySQL Cluster executables:

--help --usage, -?
Prints a short list with descriptions of the available command options.
--connect-string=connect_string, -c connect_string
connect_string sets the connectstring to the management server as a command option.
```
shell> ndbd --connect-string="nodeid=2;host=ndb_mgmd.mysql.com:1186"
```
For more information, see Section 20.3.4.2, “The Cluster Connectstring”.
--debug[=options]
This option can be used only for versions compiled with debugging enabled. It is used to enable output from debug calls in the same manner as for the mysqld process.
--execute=command, -e command
Can be used to send a command to a Cluster executable from the system shell. For example, either of the following:
```
shell> ndb_mgm -e "SHOW"
```
or
```
shell> ndb_mgm --execute="SHOW"
```
is equivalent to
```
ndb_mgm> SHOW
```
This is analogous to how the --execute or -e option works with the mysql command-line client. See Section 4.2.3.1, “Using Options on the Command Line”.
--version, -V
Prints the MySQL Cluster version number of the executable. The version number is relevant because not all versions can be used together, and the MySQL Cluster startup process verifies that the versions of the binaries being used can co-exist in the same cluster. This is also important when performing an online (rolling) software upgrade or downgrade of MySQL Cluster. (See Section 20.5.1, “Performing a Rolling Restart of the Cluster”).

The next few sections describe options specific to individual NDB programs.

See Section 20.4.2, “MySQL Cluster-Related Command Options for mysqld”, for mysqld options relating to MySQL Cluster.

20.6.5.1. Command Options for ndbd

For options common to all NDB programs, see Section 20.6.5, “Command Options for MySQL Cluster Processes”.

--bind-address
Causes ndbd to bind to a specific network interface (hostname or IP address). This option has no default value.
This option was added in MySQL 5.1.12.
--daemon, -d
Instructs ndbd to execute as a daemon process. This is the default behavior. --nodaemon can be used to prevent the process from running as a daemon.
--initial
Instructs ndbd to perform an initial start. An initial start erases any files created for recovery purposes by earlier instances of ndbd. It also re-creates recovery log files. Note that on some operating systems this process can take a substantial amount of time.
An --initial start is to be used only when starting the ndbd process under very special circumstances; this is because this option causes all files to be removed from the Cluster filesystem and all redo log files to be re-created. These circumstances are listed here:
- When performing a software upgrade which has changed the contents of any files.
- When restarting the node with a new version of ndbd.
- As a measure of last resort when for some reason the node restart or system restart repeatedly fails. In this case, be aware that this node can no longer be used to restore data due to the destruction of the data files.
Important
This option does not affect either of the following:
- Backup files that have already been created by the affected node
- Cluster Disk Data files (see Section 20.12, “MySQL Cluster Disk Data Tables”).
It is permissible to use this option when starting the cluster for the very first time (that is, before any data node files have been created); however, it is not necessary to do so.
--initial-start
This option is used when performing a partial initial start of the cluster. Each node should be started with this option, as well as --nowait-nodes.
For example, suppose you have a 4-node cluster whose data nodes have the IDs 2, 3, 4, and 5, and you wish to perform a partial initial start using only nodes 2, 4, and 5 — that is, omitting node 3:
```
ndbd --ndbd-nodeid=2 --nowait-nodes=3 --initial-start
ndbd --ndbd-nodeid=4 --nowait-nodes=3 --initial-start
ndbd --ndbd-nodeid=5 --nowait-nodes=3 --initial-start
```
This option was added in MySQL 5.1.11.
Important
Prior to MySQL 5.1.19, it was not possible to perform DDL operations involving Disk Data tables on a partially started cluster. (See Bug#24631.)
--nowait-nodes=node_id_1[, node_id_2[, ...]]
This option takes a list of data nodes which for which the cluster will not wait for before starting.
This can be used to start the cluster in a partitioned state. For example, to start the cluster with only half of the data nodes (nodes 2, 3, 4, and 5) running in a 4-node cluster, you can start each ndbd process with --nowait-nodes=3,5. In this case, the cluster starts as soon as nodes 2 and 4 connect, and does not wait StartPartitionedTimeout milliseconds for nodes 3 and 5 to connect as it would otherwise.
If you wanted to start up the same cluster as in the previous example without one ndbd — say, for example, that the host machine for node 3 has suffered a hardware failure — then start nodes 2, 4, and 5 with --nowait-nodes=3. Then the cluster will start as soon as nodes 2, 4, and 5 connect and will not wait for node 3 to start.
This option was added in MySQL 5.1.9.
--nodaemon
Instructs ndbd not to start as a daemon process. This is useful when ndbd is being debugged and you want output to be redirected to the screen.
--nostart, -n
Instructs ndbd not to start automatically. When this option is used, ndbd connects to the management server, obtains configuration data from it, and initializes communication objects. However, it does not actually start the execution engine until specifically requested to do so by the management server. This can be accomplished by issuing the proper START command in the management client (see Section 20.7.2, “Commands in the MySQL Cluster Management Client”).

20.6.5.2. Command Options for ndb_mgmd

For options common to NDB programs, see Section 20.6.5, “Command Options for MySQL Cluster Processes”.

--bind-address=host[:port]
When specified, this option limits management server connections by management clients to clients at the specified hostname or IP address (and possibly port, if this is also specified). In such cases, a management client attempting to connect to the management server from any other address fails with the error Unable to setup port: host:port!
If the port is not specified, the management client attempts to use port 1186.
This option was added in MySQL Cluster NDB 6.2.5 and MySQL Cluster NDB 6.3.2.
--config-file=filename, -f filename
Instructs the management server as to which file it should use for its configuration file. By default, the management server looks for a file named config.ini in the same directory as the ndb_mgmd executable; otherwise the filename and location must be specified explicitly.
--daemon, -d
Instructs ndb_mgmd to start as a daemon process. This is the default behavior.
--nodaemon
Instructs ndb_mgmd not to start as a daemon process.
--print-full-config, -P
Shows extended information regarding the configuration of the cluster. With this option on the command line the ndb_mgmd process prints information about the cluster setup including an extensive list of the cluster configuration sections as well as parameters and their values. Normally used together with the --config-file (-f) option.

20.6.5.3. Command Options for ndb_mgm

For options common to NDB programs, see Section 20.6.5, “Command Options for MySQL Cluster Processes”.

--try-reconnect=number
If the connection to the management server is broken, the node tries to reconnect to it every 5 seconds until it succeeds. By using this option, it is possible to limit the number of attempts to number before giving up and reporting an error instead.

20.7. Management of MySQL Cluster

20.7.1. Summary of MySQL Cluster Start Phases
20.7.2. Commands in the MySQL Cluster Management Client
20.7.3. Event Reports Generated in MySQL Cluster
20.7.4. Single User Mode
20.7.5. Quick Reference: MySQL Cluster SQL Statements

Managing a MySQL Cluster involves a number of tasks, the first of which is to configure and start MySQL Cluster. This is covered in Section 20.3, “MySQL Cluster Configuration”, and Section 20.6, “Process Management in MySQL Cluster”.

The following sections cover the management of a running MySQL Cluster.

For information about security issues relating to management and deployment of a MySQL Cluster, see Section 20.8, “MySQL Cluster Security Issues”.

There are essentially two methods of actively managing a running MySQL Cluster. The first of these is through the use of commands entered into the management client whereby cluster status can be checked, log levels changed, backups started and stopped, and nodes stopped and started. The second method involves studying the contents of the cluster log ndb_node_id_cluster.log; this is usually found in the management server's DataDir directory, but this location can be overridden using the LogDestination option — see Section 20.3.4.4, “Defining the Management Server”, for details. (Recall that node_id represents the unique identifier of the node whose activity is being logged.) The cluster log contains event reports generated by ndbd. It is also possible to send cluster log entries to a Unix system log.

In addition, some aspects of the cluster's operation can be monitored from an SQL node using the SHOW ENGINE NDB STATUS statement. See Section 12.5.4.13, “SHOW ENGINE Syntax”, for more information.

20.7.1. Summary of MySQL Cluster Start Phases

This section provides a simplified outline of the steps involved when MySQL Cluster data nodes are started. More complete information can be found in MySQL Cluster Start Phases.

These phases are the same as those reported in the output from the node_id STATUS command in the management client. (See Section 20.7.2, “Commands in the MySQL Cluster Management Client”, for more information about this command.)

Start types. There are several different startup types and modes, as shown here:

Initial Start. The cluster starts with a clean filesystem on all data nodes. This occurs either when the cluster started for the very first time, or when all data nodes are restarted using the --initial option.
Note
Disk Data files are not removed when restarting a node using --initial.
System Restart. The cluster starts and reads data stored in the data nodes. This occurs when the cluster has been shut down after having been in use, when it is desired for the cluster to resume operations from the point where it left off.
Node Restart. This is the online restart of a cluster node while the cluster itself is running.
Initial Node Restart. This is the same as a node restart, except that the node is reinitialized and started with a clean filesystem.

Setup and initialization (Phase -1). Prior to startup, each data node (ndbd process) must be initialized. Initialization consists of the following steps:

Obtain a node ID
Fetch configuration data
Allocate ports to be used for inter-node communications
Allocate memory according to settings obtained from the configuration file

When a data node or SQL node first connects to the management node, it reserves a cluster node ID. To make sure that no other node allocates the same node ID, this ID is retained until the node has managed to connect to the cluster and at least one ndbd reports that this node is connected. This retention of the node ID is guarded by the connection between the node in question and ndb_mgmd.

Normally, in the event of a problem with the node, the node disconnects from the management server, the socket used for the connection is closed, and the reserved node ID is freed. However, if a node is disconnected abruptly — for example, due to a hardware failure in one of the cluster hosts, or because of network issues — the normal closing of the socket by the operating system may not take place. In this case, the node ID continues to be reserved and not released until a TCP timeout occurs 10 or so minutes later.

To take care of this problem, you can use PURGE STALE SESSIONS. Running this statement forces all reserved node IDs to be checked; any that are not being used by nodes actually connected to the cluster are then freed.

Beginning with MySQL 5.1.11, timeout handling of node ID assignments is implemented. This performs the ID usage checks automatically after approximately 20 seconds, so that PURGE STALE SESSIONS should no longer be necessary in a normal Cluster start.

After each data node has been initialized, the cluster startup process can proceed. The stages which the cluster goes through during this process are listed here:

Phase 0. The NDBFS and NDBCNTR blocks start (see NDB Kernel Blocks). The cluster filesystem is cleared, if the cluster was started with the --initial option.
Phase 1. In this stage, all remaining NDB kernel blocks are started. Cluster connections are set up, inter-block communications are established, and Cluster heartbeats are started. In the case of a node restart, API node connections are also checked.
Note
When one or more nodes hang in Phase 1 while the remaining node or nodes hang in Phase 2, this often indicates network problems. One possible cause of such issues is one or more cluster hosts having multiple network interfaces. Another common source of problems causing this condition is the blocking of TCP/IP ports needed for communications between cluster nodes. In the latter case, this is often due to a misconfigured firewall.
Phase 2. The NDBCNTR kernel block checks the states of all existing nodes. The master node is chosen, and the cluster schema file is initialized.
Phase 3. The DBLQH and DBTC kernel blocks set up communications between them. The startup type is determined; if this is a restart, the DBDIH block obtains permission to perform the restart.
Phase 4. For an initial start or initial node restart, the redo log files are created. The number of these files is equal to NoOfFragmentLogFiles.
For a system restart:
- Read schema or schemas.
- Read data from the local checkpoint.
- Apply all redo information until the latest restorable global checkpoint has been reached.
For a node restart, find the tail of the redo log.
Phase 5. Most of the database-related portion of a data node start is performed during this phase. For an initial start or system restart, a local checkpoint is executed, followed by a global checkpoint. Periodic checks of memory usage begin during this phase, and any required node takeovers are performed.
Phase 6. In this phase, node groups are defined and set up.
Phase 7. The arbitrator node is selected and begins to function. The next backup ID is set, as is the backup disk write speed. Nodes reaching this start phase are marked as Started. It is now possible for API nodes (including SQL nodes) to connect to the cluster. connect.
Phase 8. If this is a system restart, all indexes are rebuilt (by DBDIH).
Phase 9. The node internal startup variables are reset.
Phase 100 (OBSOLETE). Formerly, it was at this point during a node restart or initial node restart that API nodes could connect to the node and begin to receive events. Currently, this phase is empty.
Phase 101. At this point in a node restart or initial node restart, event delivery is handed over to the node joining the cluster. The newly-joined node takes over responsibility for delivering its primary data to subscribers. This phase is also referred to as SUMA handover phase.

After this process is completed for an initial start or system restart, transaction handling is enabled. For a node restart or initial node restart, completion of the startup process means that the node may now act as a transaction coordinator.

20.7.2. Commands in the MySQL Cluster Management Client

In addition to the central configuration file, a cluster may also be controlled through a command-line interface available through the management client ndb_mgm. This is the primary administrative interface to a running cluster.

Commands for the event logs are given in Section 20.7.3, “Event Reports Generated in MySQL Cluster”; commands for creating backups and restoring from backup are provided in Section 20.9, “On-line Backup of MySQL Cluster”.

The management client has the following basic commands. In the listing that follows, node_id denotes either a database node ID or the keyword ALL, which indicates that the command should be applied to all of the cluster's data nodes.

HELP
Displays information on all available commands.
SHOW
Displays information on the cluster's status.
Note
In a cluster where multiple management nodes are in use, this command displays information only for data nodes that are actually connected to the current management server.
node_id START
Brings online the data node identified by node_id (or all data nodes).
ALL START works on all data nodes only, and does not affect management nodes.
Important
To use this command to bring a data node online, the data node must have been started using ndbd --nostart or ndbd -n.
node_id STOP
Stops the data or management node identified by node_id. Note that ALL STOP works to stop all data nodes only, and does not affect management nodes.
A node affected by this command disconnects from the cluster, and its associated ndbd or ndb_mgmd process terminates.
node_id RESTART [-n] [-i] [-a]
Restarts the data node identified by node_id (or all data nodes).
Using the -i option with RESTART causes the data node to perform an initial restart; that is, the node's filesystem is deleted and recreated. The effect is the same as that obtained from stopping the data node process and then starting it again using ndbd --initial from the system shell. Note that backup files and Disk Data files are not removed when this option is used.
Using the -n option causes the data node process to be restarted, but the data node is not actually brought online until the appropriate START command is issued. The effect of this option is the same as that obtained from stopping the data node and then starting it again using ndbd --nostart or ndbd -n from the system shell.
Using the -a causes all current transactions relying on this node to be aborted. No GCP check is done when the node rejoins the cluster.
node_id STATUS
Displays status information for the data node identified by node_id (or for all data nodes).
node_id REPORT report-type
Displays a report of type report-type for the data node identified by node_id, or for all data nodes using ALL.
Currently, there are two accepted values for report-type:
- BackupStatus provides a status report on a cluster backup in progress
- MemoryUsage displays how much data memory and index memory is being used by each data node.
The REPORT command was introduced in MySQL Cluster NDB 6.2.3 and MySQL Cluster NDB 6.3.0.
ENTER SINGLE USER MODE node_id
Enters single user mode, whereby only the MySQL server identified by the node ID node_id is allowed to access the database.
Important
It is not possible in MySQL 5.1 for data nodes to join the cluster while it is running in single user mode. (See Bug#20395 for more information.)
EXIT SINGLE USER MODE
Exits single user mode, allowing all SQL nodes (that is, all running mysqld processes) to access the database.
QUIT, EXIT
Terminates the management client.
This command does not affect any nodes connected to the cluster.
SHUTDOWN
Shuts down all cluster data nodes and management nodes. To exit the management client after this has been done, use EXIT or QUIT.
This command does not shut down any SQL nodes or API nodes that are connected to the cluster.

20.7.3. Event Reports Generated in MySQL Cluster

20.7.3.1. Logging Management Commands
20.7.3.2. Log Events
20.7.3.3. Using CLUSTERLOG STATISTICS

In this section, we discuss the types of event logs provided by MySQL Cluster, and the types of events that are logged.

MySQL Cluster provides two types of event log:

The cluster log, which includes events generated by all cluster nodes. The cluster log is the log recommended for most uses because it provides logging information for an entire cluster in a single location.
By default, the cluster log is saved to a file named ndb_node_id_cluster.log, (where node_id is the node ID of the management server) in the same directory where the ndb_mgm binary resides.
Cluster logging information can also be sent to stdout or a syslog facility in addition to or instead of being saved to a file, as determined by the values set for the DataDir and LogDestination configuration parameters. See Section 20.3.4.4, “Defining the Management Server”, for more information about these parameters.
Node logs are local to each node.
Output generated by node event logging is written to the file ndb_node_id_out.log (where node_id is the node's node ID) in the node's DataDir. Node event logs are generated for both management nodes and data nodes.
Node logs are intended to be used only during application development, or for debugging application code.

Both types of event logs can be set to log different subsets of events.

Each reportable event can be distinguished according to three different criteria:

Category: This can be any one of the following values: STARTUP, SHUTDOWN, STATISTICS, CHECKPOINT, NODERESTART, CONNECTION, ERROR, or INFO.
Priority: This is represented by one of the numbers from 1 to 15 inclusive, where 1 indicates “most important” and 15 “least important.”
Severity Level: This can be any one of the following values: ALERT, CRITICAL, ERROR, WARNING, INFO, or DEBUG.

Both the cluster log and the node log can be filtered on these properties.

The format used in the cluster log is as shown here:

2007-01-26 19:35:55 [MgmSrvr] INFO     -- Node 1: Data usage is 2%(60 32K pages of total 2560)
2007-01-26 19:35:55 [MgmSrvr] INFO     -- Node 1: Index usage is 1%(24 8K pages of total 2336)
2007-01-26 19:35:55 [MgmSrvr] INFO     -- Node 1: Resource 0 min: 0 max: 639 curr: 0
2007-01-26 19:35:55 [MgmSrvr] INFO     -- Node 2: Data usage is 2%(76 32K pages of total 2560)
2007-01-26 19:35:55 [MgmSrvr] INFO     -- Node 2: Index usage is 1%(24 8K pages of total 2336)
2007-01-26 19:35:55 [MgmSrvr] INFO     -- Node 2: Resource 0 min: 0 max: 639 curr: 0
2007-01-26 19:35:55 [MgmSrvr] INFO     -- Node 3: Data usage is 2%(58 32K pages of total 2560)
2007-01-26 19:35:55 [MgmSrvr] INFO     -- Node 3: Index usage is 1%(25 8K pages of total 2336)
2007-01-26 19:35:55 [MgmSrvr] INFO     -- Node 3: Resource 0 min: 0 max: 639 curr: 0
2007-01-26 19:35:55 [MgmSrvr] INFO     -- Node 4: Data usage is 2%(74 32K pages of total 2560)
2007-01-26 19:35:55 [MgmSrvr] INFO     -- Node 4: Index usage is 1%(25 8K pages of total 2336)
2007-01-26 19:35:55 [MgmSrvr] INFO     -- Node 4: Resource 0 min: 0 max: 639 curr: 0
2007-01-26 19:39:42 [MgmSrvr] INFO     -- Node 4: Node 9 Connected
2007-01-26 19:39:42 [MgmSrvr] INFO     -- Node 1: Node 9 Connected
2007-01-26 19:39:42 [MgmSrvr] INFO     -- Node 1: Node 9: API version 5.1.15
2007-01-26 19:39:42 [MgmSrvr] INFO     -- Node 2: Node 9 Connected
2007-01-26 19:39:42 [MgmSrvr] INFO     -- Node 2: Node 9: API version 5.1.15
2007-01-26 19:39:42 [MgmSrvr] INFO     -- Node 3: Node 9 Connected
2007-01-26 19:39:42 [MgmSrvr] INFO     -- Node 3: Node 9: API version 5.1.15
2007-01-26 19:39:42 [MgmSrvr] INFO     -- Node 4: Node 9: API version 5.1.15
2007-01-26 19:59:22 [MgmSrvr] ALERT    -- Node 2: Node 7 Disconnected
2007-01-26 19:59:22 [MgmSrvr] ALERT    -- Node 2: Node 7 Disconnected

Each line in the cluster log contains the following information:

A timestamp in YYYY-MM-DD HH:MM:SS format.
The type of node which is performing the logging. In the cluster log, this is always [MgmSrvr].
The severity of the event.
The ID of the node reporting the event.
A description of the event. The most common types of events to appear in the log are connections and disconnections between different nodes in the cluster, and when checkpoints occur. In some cases, the description may contain status information.

20.7.3.1. Logging Management Commands

The following management commands are related to the cluster log:

CLUSTERLOG ON
Turns the cluster log on.
CLUSTERLOG OFF
Turns the cluster log off.
CLUSTERLOG INFO
Provides information about cluster log settings.
node_id CLUSTERLOG category=threshold
Logs category events with priority less than or equal to threshold in the cluster log.
CLUSTERLOG FILTER severity_level
Toggles cluster logging of events of the specified severity_level.

The following table describes the default setting (for all data nodes) of the cluster log category threshold. If an event has a priority with a value lower than or equal to the priority threshold, it is reported in the cluster log.

Note that events are reported per data node, and that the threshold can be set to different values on different nodes.

Category	Default threshold (All data nodes)
`STARTUP`	7
`SHUTDOWN`	7
`STATISTICS`	7
`CHECKPOINT`	7
`NODERESTART`	7
`CONNECTION`	7
`ERROR`	15
`INFO`	7

The STATISTICS category can provide a great deal of useful data. See Section 20.7.3.3, “Using CLUSTERLOG STATISTICS”, for more information.

Thresholds are used to filter events within each category. For example, a STARTUP event with a priority of 3 is not logged unless the threshold for STARTUP is set to 3 or higher. Only events with priority 3 or lower are sent if the threshold is 3.

The following table shows the event severity levels.

Note

These correspond to Unix syslog levels, except for LOG_EMERG and LOG_NOTICE, which are not used or mapped.

1	`ALERT`	A condition that should be corrected immediately, such as a corrupted system database
2	`CRITICAL`	Critical conditions, such as device errors or insufficient resources
3	`ERROR`	Conditions that should be corrected, such as configuration errors
4	`WARNING`	Conditions that are not errors, but that might require special handling
5	`INFO`	Informational messages
6	`DEBUG`	Debugging messages used for `NDBCLUSTER` development

Event severity levels can be turned on or off (using CLUSTERLOG FILTER — see above). If a severity level is turned on, then all events with a priority less than or equal to the category thresholds are logged. If the severity level is turned off then no events belonging to that severity level are logged.

Important

Cluster log levels are set on a per ndb_mgmd, per subscriber basis. This means that, in a MySQL Cluster with multiple management servers, using a CLUSTERLOG command in an instance of ndb_mgm connected to one management server affects only logs generated by that management server but not by any of the others. This also means that, should one of the management servers be restarted, only logs generated by that management server are affected by the resetting of log levels caused by the restart.

20.7.3.2. Log Events

An event report reported in the event logs has the following format:

datetime [string] severity -- message

For example:

09:19:30 2005-07-24 [NDB] INFO -- Node 4 Start phase 4 completed

This section discusses all reportable events, ordered by category and severity level within each category.

In the event descriptions, GCP and LCP mean “Global Checkpoint” and “Local Checkpoint”, respectively.

CONNECTION Events

These events are associated with connections between Cluster nodes.

Event	Priority	Severity Level	Description
data nodes connected	8	`INFO`	Data nodes connected
data nodes disconnected	8	`INFO`	Data nodes disconnected
Communication closed	8	`INFO`	SQL node or data node connection closed
Communication opened	8	`INFO`	SQL node or data node connection opened

CHECKPOINT Events

The logging messages shown here are associated with checkpoints.

Event	Priority	Severity Level	Description
LCP stopped in calc keep GCI	0	`ALERT`	LCP stopped
Local checkpoint fragment completed	11	`INFO`	LCP on a fragment has been completed
Global checkpoint completed	10	`INFO`	GCP finished
Global checkpoint started	9	`INFO`	Start of GCP: REDO log is written to disk
Local checkpoint completed	8	`INFO`	LCP completed normally
Local checkpoint started	7	`INFO`	Start of LCP: data written to disk

STARTUP Events

The following events are generated in response to the startup of a node or of the cluster and of its success or failure. They also provide information relating to the progress of the startup process, including information concerning logging activities.

Event	Priority	Severity Level	Description
Internal start signal received STTORRY	15	`INFO`	Blocks received after completion of restart
New REDO log started	10	`INFO`	GCI keep `X`, newest restorable GCI `Y`
New log started	10	`INFO`	Log part `X`, start MB `Y`, stop MB `Z`
Node has been refused for inclusion in the cluster	8	`INFO`	Node cannot be included in cluster due to misconfiguration, inability to establish communication, or other problem
data node neighbors	8	`INFO`	Shows neighboring data nodes
data node start phase `X` completed	4	`INFO`	A data node start phase has been completed
Node has been successfully included into the cluster	3	`INFO`	Displays the node, managing node, and dynamic ID
data node start phases initiated	1	`INFO`	NDB Cluster nodes starting
data node all start phases completed	1	`INFO`	NDB Cluster nodes started
data node shutdown initiated	1	`INFO`	Shutdown of data node has commenced
data node shutdown aborted	1	`INFO`	Unable to shut down data node normally

NODERESTART Events

The following events are generated when restarting a node and relate to the success or failure of the node restart process.

Event	Priority	Severity Level	Description
Node failure phase completed	8	`ALERT`	Reports completion of node failure phases
Node has failed, node state was `X`	8	`ALERT`	Reports that a node has failed
Report arbitrator results	2	`ALERT`	There are eight different possible results for arbitration attempts: Arbitration check failed — less than 1/2 nodes left Arbitration check succeeded — node group majority Arbitration check failed — missing node group Network partitioning — arbitration required Arbitration succeeded — affirmative response from node `X` Arbitration failed - negative response from node `X` Network partitioning - no arbitrator available Network partitioning - no arbitrator configured
Completed copying a fragment	10	`INFO`
Completed copying of dictionary information	8	`INFO`
Completed copying distribution information	8	`INFO`
Starting to copy fragments	8	`INFO`
Completed copying all fragments	8	`INFO`
GCP takeover started	7	`INFO`
GCP takeover completed	7	`INFO`
LCP takeover started	7	`INFO`
LCP takeover completed (state = `X`)	7	`INFO`
Report whether an arbitrator is found or not	6	`INFO`	There are seven different possible outcomes when seeking an arbitrator: Management server restarts arbitration thread [state=`X`] Prepare arbitrator node `X` [ticket=`Y`] Receive arbitrator node `X` [ticket=`Y`] Started arbitrator node `X` [ticket=`Y`] Lost arbitrator node `X` - process failure [state=`Y`] Lost arbitrator node `X` - process exit [state=`Y`] Lost arbitrator node `X` <error msg> [state=`Y`]

STATISTICS Events

The following events are of a statistical nature. They provide information such as numbers of transactions and other operations, amount of data sent or received by individual nodes, and memory usage.

Event	Priority	Severity Level	Description
Report job scheduling statistics	9	`INFO`	Mean internal job scheduling statistics
Sent number of bytes	9	`INFO`	Mean number of bytes sent to node `X`
Received # of bytes	9	`INFO`	Mean number of bytes received from node `X`
Report transaction statistics	8	`INFO`	Numbers of: transactions, commits, reads, simple reads, writes, concurrent operations, attribute information, and aborts
Report operations	8	`INFO`	Number of operations
Report table create	7	`INFO`
Memory usage	5	`INFO`	Data and index memory usage (80%, 90%, and 100%)

ERROR Events

These events relate to Cluster errors and warnings. The presence of one or more of these generally indicates that a major malfunction or failure has occurred.

Event	Priority	Severity	Description
Dead due to missed heartbeat	8	`ALERT`	Node `X` declared “dead” due to missed heartbeat
Transporter errors	2	ERROR
Transporter warnings	8	`WARNING`
Missed heartbeats	8	`WARNING`	Node `X` missed heartbeat #`Y`
General warning events	2	`WARNING`

INFO Events

These events provide general information about the state of the cluster and activities associated with Cluster maintenance, such as logging and heartbeat transmission.

Event	Priority	Severity	Description
Sent heartbeat	12	`INFO`	Heartbeat sent to node `X`
Create log bytes	11	`INFO`	Log part, log file, MB
General information events	2	`INFO`

20.7.3.3. Using `CLUSTERLOG STATISTICS`

The NDB management client's CLUSTERLOG STATISTICS command can provide a number of useful statistics in its output. The following statistics are reported by the transaction coordinator:

Statistic	*Description (Number of...)*
`Trans. Count`	Transactions attempted with this node as coordinator
`Commit Count`	Transactions committed with this node as coordinator
`Read Count`	Primary key reads (all)
`Simple Read Count`	Primary key reads reading the latest committed value
`Write Count`	Primary key writes (includes all `INSERT`, `UPDATE`, and `DELETE` operations)
`AttrInfoCount`	Data words used to describe all reads and writes received
`Concurrent Operations`	All concurrent operations ongoing at the moment the report is taken
`Abort Count`	Transactions with this node as coordinator that were aborted
`Scans`	Scans (all)
`Range Scans`	Index scans

The ndbd process has a scheduler that runs in an infinite loop. During each loop scheduler performs the following tasks:

Read any incoming messages from sockets into a job buffer.
Check whether there are any timed messages to be executed; if so, put these into the job buffer as well.
Execute (in a loop) any messages in the job buffer.
Send any distributed messages that were generated by executing the messages in the job buffer.
Wait for any new incoming messages.

The number of loops executed in the third step is reported as the Mean Loop Counter. This statistic increases in size as the utilisation of the TCP/IP buffer improves. You can use this to monitor performance as you add new processes to the cluster.

The Mean send size and Mean receive size statistics allow you to gauge the efficiency of writes and reads (respectively) between nodes. These values are given in bytes. Higher values mean a lower cost per byte sent or received; the maximum is 64k.

To cause all cluster log statistics to be logged, you can use the following command in the NDB management client:

ndb_mgm> ALL CLUSTERLOG STATISTICS=15

Note

Setting the threshold for STATISTICS to 15 causes the cluster log to become very verbose, and to gow quite rapidly in size, in direct proportion to the number of cluster nodes and the amount of activity on the cluster.

20.7.4. Single User Mode

Single user mode allows the database administrator to restrict access to the database system to a single API node, such as a MySQL server (SQL node) or an instance of ndb_restore. When entering single user mode, connections to all other API nodes are closed gracefully and all running transactions are aborted. No new transactions are permitted to start.

Once the cluster has entered single user mode, only the designated API node is granted access to the database.

You can use the ALL STATUS command to see when the cluster has entered single user mode.

Example:

ndb_mgm> ENTER SINGLE USER MODE 5

After this command has executed and the cluster has entered single user mode, the API node whose node ID is 5 becomes the cluster's only permitted user.

The node specified in the preceding command must be an API node; attempting to specify any other type of node will be rejected.

Note

When the preceding command is invoked, all transactions running on the designated node are aborted, the connection is closed, and the server must be restarted.

The command EXIT SINGLE USER MODE changes the state of the cluster's data nodes from single user mode to normal mode. API nodes — such as MySQL Servers — waiting for a connection (that is, waiting for the cluster to become ready and available), are again permitted to connect. The API node denoted as the single-user node continues to run (if still connected) during and after the state change.

Example:

ndb_mgm> EXIT SINGLE USER MODE

There are two recommended ways to handle a node failure when running in single user mode:

Method 1:
1. Finish all single user mode transactions
2. Issue the EXIT SINGLE USER MODE command
3. Restart the cluster's data nodes
Method 2:
Restart database nodes prior to entering single user mode.

20.7.5. Quick Reference: MySQL Cluster SQL Statements

This section discusses several SQL statements that can prove useful in managing and monitoring a MySQL server that is connected to a MySQL Cluster, and in some cases provide information about the cluster itself.

SHOW ENGINE NDB STATUS, SHOW ENGINE NDBCLUSTER STATUS
The output of this statement contains information about the server's connection to the cluster, creation and usage of MySQL Cluster objects, and binary logging for MySQL Cluster replication.
See Section 12.5.4.13, “SHOW ENGINE Syntax”, for a usage example and more detailed information.
SHOW ENGINES
This statement can be used to determine whether or not clustering support is enabled in the MySQL server, and if so, whether it is active.
See Section 12.5.4.14, “SHOW ENGINES Syntax”, for more detailed information.
Note
In MySQL 5.1, this statement no longer supports a LIKE clause. However, you can use LIKE to filter queries against the INFORMATION_SCHEMA.ENGINES, as discussed in the next item.
SELECT * FROM INFORMATION_SCHEMA.ENGINES [WHERE ENGINE LIKE 'NDB%']
This is the equivalent of SHOW ENGINES, but uses the ENGINES table of the INFORMATION_SCHEMA database (available beginning with MySQL 5.1.5). Unlike the case with the SHOW ENGINES statement, it is possible to filter the results using a LIKE clause, and to select specific columns to obtain information that may be of use in scripts. For example, the following query shows whether the server was built with NDB support and, if so, whether it is enabled:
```
mysql> SELECT SUPPORT FROM INFORMATION_SCHEMA.ENGINES
    ->   WHERE ENGINE LIKE 'NDB%';
+---------+
| support |
+---------+
| ENABLED |
+---------+
```
See Section 27.18, “The INFORMATION_SCHEMA ENGINES Table”, for more information.

SHOW VARIABLES LIKE 'NDB%'

This statement provides a list of most server system variables relating to the NDB storage engine, and their values, as shown here:

mysql> SHOW VARIABLES LIKE 'NDB%';
+-------------------------------------+-------+
| Variable_name                       | Value |
+-------------------------------------+-------+
| ndb_autoincrement_prefetch_sz       | 32    |
| ndb_cache_check_time                | 0     |
| ndb_extra_logging                   | 0     |
| ndb_force_send                      | ON    |
| ndb_index_stat_cache_entries        | 32    |
| ndb_index_stat_enable               | OFF   |
| ndb_index_stat_update_freq          | 20    |
| ndb_report_thresh_binlog_epoch_slip | 3     |
| ndb_report_thresh_binlog_mem_usage  | 10    |
| ndb_use_copying_alter_table         | OFF   |
| ndb_use_exact_count                 | ON    |
| ndb_use_transactions                | ON    |
+-------------------------------------+-------+

See Section 5.1.3, “System Variables”, for more information.

SELECT * FROM INFORMATION_SCHEMA.GLOBAL_VARIABLES WHERE VARIABLE_NAME LIKE 'NDB%';

This statement is the equivalent of the SHOW described in the previous item, and provides almost identical output, as shown here:

mysql> SELECT * FROM INFORMATION_SCHEMA.GLOBAL_VARIABLES 
    ->   WHERE VARIABLE_NAME LIKE 'NDB%';
+-------------------------------------+----------------+
| VARIABLE_NAME                       | VARIABLE_VALUE |
+-------------------------------------+----------------+
| NDB_AUTOINCREMENT_PREFETCH_SZ       | 32             |
| NDB_CACHE_CHECK_TIME                | 0              |
| NDB_EXTRA_LOGGING                   | 0              |
| NDB_FORCE_SEND                      | ON             |
| NDB_INDEX_STAT_CACHE_ENTRIES        | 32             |
| NDB_INDEX_STAT_ENABLE               | OFF            |
| NDB_INDEX_STAT_UPDATE_FREQ          | 20             |
| NDB_REPORT_THRESH_BINLOG_EPOCH_SLIP | 3              |
| NDB_REPORT_THRESH_BINLOG_MEM_USAGE  | 10             |
| NDB_USE_COPYING_ALTER_TABLE         | OFF            |
| NDB_USE_EXACT_COUNT                 | ON             |
| NDB_USE_TRANSACTIONS                | ON             |
+-------------------------------------+----------------+

Unlike the case with the SHOW statement, it is possible to select individual columns. For example:

mysql> SELECT VARIABLE_VALUE 
    ->   FROM INFORMATION_SCHEMA.GLOBAL_VARIABLES
    ->   WHERE VARIABLE_NAME = 'ndb_force_send';
+----------------+
| VARIABLE_VALUE |
+----------------+
| ON             |
+----------------+

See Section 27.25, “The INFORMATION_SCHEMA GLOBAL_VARIABLES and SESSION_VARIABLES Tables”, and Section 5.1.3, “System Variables”, for more information.

SHOW STATUS LIKE 'NDB%'

This statement shows at a glance whether or not the MySQL server is acting as a cluster SQL node, and if so, it provides the MySQL server's cluster node ID, the hostname and port for the cluster management server to which it is connected, and the number of data nodes in the cluster, as shown here:

mysql> SHOW STATUS LIKE 'NDB%';
+--------------------------+---------------+
| Variable_name            | Value         |
+--------------------------+---------------+
| Ndb_cluster_node_id      | 10            |
| Ndb_config_from_host     | 192.168.0.103 |
| Ndb_config_from_port     | 1186          |
| Ndb_number_of_data_nodes | 4             |
+--------------------------+---------------+

If the MySQL server was built with clustering support, but it is not connected to a cluster, all rows in the output of this statement contain a zero or an empty string:

mysql> SHOW STATUS LIKE 'NDB%';
+--------------------------+-------+
| Variable_name            | Value |
+--------------------------+-------+
| Ndb_cluster_node_id      | 0     |
| Ndb_config_from_host     |       |
| Ndb_config_from_port     | 0     |
| Ndb_number_of_data_nodes | 0     |
+--------------------------+-------+

SELECT * FROM INFORMATION_SCHEMA.GLOBAL_STATUS WHERE VARIABLE_NAME LIKE 'NDB%';
Beginning with MySQL 5.1.12, this statement provides similar output to the SHOW statement discussed in the previous item. However, unlike the case with SHOW STATUS, it is possible using the SELECT to extract values in SQL for use in scripts for monitoring and automation purposes.
See Section 27.24, “The INFORMATION_SCHEMA GLOBAL_STATUS and SESSION_STATUS Tables”, for more information.

20.8. MySQL Cluster Security Issues

20.8.1. MySQL Cluster Security and Networking Issues
20.8.2. MySQL Cluster and MySQL Privileges
20.8.3. MySQL Cluster and MySQL Security Procedures

This section discusses security considerations to take into account when setting up and running MySQL Cluster.

Topics to be covered in this chapter include the following:

MySQL Cluster and network security issues
Configuration issues relating to running MySQL Cluster securely
MySQL Cluster and the MySQL privilege system
MySQL standard security procedures as applicable to MySQL Cluster

20.8.1. MySQL Cluster Security and Networking Issues

In this section, we discuss basic network security issues as they relate to MySQL Cluster. It is extremely important to remember that MySQL Cluster “out of the box” is not secure; you or your network administrator must take the proper steps to insure that your cluster cannot be compromised over the network.

Cluster communication protocols are inherently insecure, and no encryption or similar security measures are used in communications between nodes in the cluster. Because network speed and latency have a direct impact on the cluster's efficiency, it is also not advisable to employ SSL or other encryption to network connections between nodes, as such schemes will effectively slow communications.

It is also true that no authentication is used for controlling API node access to a MySQL Cluster. As with encryption, the overhead of imposing authentication requirements would have an adverse impact on Cluster performance.

In addition, there is no checking of the source IP address for either of the following when accessing the cluster:

SQL or API nodes using “free slots” created by empty [mysqld] or [api] sections in the config.ini file
This means that, if there are any empty [mysqld] or [api] sections in the config.ini file, then any API nodes (including SQL nodes) that know the management server's hostname (or IP address) and port can connect to the cluster and access its data without restriction. (See Section 20.8.2, “MySQL Cluster and MySQL Privileges”, for more information about this and related issues.)
Note
You can exercise some control over SQL and API node access to the cluster by specifying a HostName parameter for all [mysqld] and [api] sections in the config.ini file. However, this also means that, should you wish to connect an API node to the cluster from a previously unused host, you need to add an [api] section containing its hostname to the config.ini file.
More information is available elsewhere in this chapter about the HostName parameter. Also see Section 20.3.3, “Quick Test Setup of MySQL Cluster”, for configuration examples using HostName with API nodes.
Any ndb_mgm client
This means that any cluster management client that is given the management server's hostname (or IP address) and port (if not the standard port) can connect to the cluster and execute any management client command. This includes commands such as ALL STOP and SHUTDOWN.

For these reasons, it is necessary to protect the cluster on the network level. The safest network configuration for Cluster is one which isolates connections between Cluster nodes from any other network communications. This can be accomplished by any of the following methods:

Keeping Cluster nodes on a network that is physically separate from any public networks. This option is the most dependable; however, it is the most expensive to implement.
We show an example of a MySQL Cluster setup using such a physically segregated network here:
This setup has two networks, one private (solid box) for the Cluster management servers and data nodes, and one public (dotted box) where the SQL nodes reside. (We show the management and data nodes connected using a gigabit switch since this provides the best performance.) Both networks are protected from the outside by a hardware firewall, sometimes also known as a network-based firewall.
This network setup is safest because no packets can reach the cluster's management or data nodes from outside the network — and none of the cluster's internal communications can reach the outside — without going through the SQL nodes, as long as the SQL nodes do not allow any packets to be forwarded. This means, of course, that all SQL nodes must be secured against hacking attempts.
Important
With regard to potential security vulnerabilities, an SQL node is no different from any other MySQL server. See Section 5.3.2, “Making MySQL Secure Against Attackers”, for a description of techniques you can use to secure MySQL servers.

Using one or more software firewalls (also known as host-based firewalls) to control which packets pass through to the cluster from portions of the network that do not require access to it. In this type of setup, a software firewall must be installed on every host in the cluster which might otherwise be accessible from outside the local network.

The host-based option is the least expensive to implement, but relies purely on software to provide protection and so is the most difficult to keep secure.

This type of network setup for MySQL Cluster is illustrated here:

MySQL Cluster deployed on a network
using software firewalls to create public and private
zones

Using this type of network setup means that there are two zones of MySQL Cluster hosts. Each cluster host must be able to communicate with all of the other machines in the cluster, but only those hosting SQL nodes (dotted box) can be permitted to have any contact with the outside, while those in the zone containing the data nodes and management nodes (solid box) must be isolated from any machines that are not part of the cluster. Applications using the cluster and user of those applications must not be permitted to have direct access to the management and data node hosts.

To accomplish this, you must set up software firewalls that limit the traffic to the type or types shown in the following table, according to the type of node that is running on each cluster host computer:

Type of Node to be Accessed	Traffic to Allow
SQL or API node	It originates from the IP address of a management or data node (using any TCP or UDP port). It originates from within the network in which the cluster resides and is on the port that your application is using.
Data node or Management node	It originates from the IP address of a management or data node (using any TCP or UDP port). It originates from the IP address of an SQL or API node.

Any traffic other than that shown in the table for a given node type should be denied.

The specifics of configuring a firewall vary from firewall application to firewall application, and are beyond the scope of this Manual. iptables is a very common and reliable firewall application, which is often used with APF as a front end to make configuration easier. You can (and should) consult the documentation for the software firewall that you employ, should you choose to implement a MySQL Cluster network setup of this type, or of a “mixed” type as discussed under the next item.

It is also possible to employ a combination of the first two methods, using both hardware and software to secure the cluster — that is, using both network-based and host-based firewalls. This is between the first two schemes in terms of both security level and cost. This type of network setup keeps the cluster behind the hardware firewall, but allows incoming packets to travel beyond the router connecting all cluster hosts in order to reach the SQL nodes.
One possible network deployment of a MySQL Cluster using hardware and software firewalls in combination is shown here:
In this case, you can set the rules in the hardware firewall to deny any external traffic except to SQL nodes and API nodes, and then allow traffic to them only on the ports required by your application.

Whatever network configuration you use, remember that your objective from the viewpoint of keeping the cluster secure remains the same — to prevent any unessential traffic from reaching the cluster while ensuring the most efficient communication between the nodes in the cluster.

Because MySQL Cluster requires large numbers of ports to be open for communications between nodes, the recommended option is to use a segregated network. This represents the simplest way to prevent unwanted traffic from reaching the cluster.

Note

If you wish to administer a MySQL Cluster remotely (that is, from outside the local network), the recommended way to do this is to use ssh or another secure login shell to access an SQL node host. From this host, you can then run the management client to access the management server safely, from within the Cluster's own local network.

Even though it is possible to do so in theory, it is not recommended to use ndb_mgm to manage a Cluster directly from outside the local network on which the Cluster is running. Since neither authentication nor encryption takes place between the management client and the management server, this represents an extremely insecure means of managing the cluster, and is almost certain to be compromised sooner or later.

20.8.2. MySQL Cluster and MySQL Privileges

In this section, we discuss how the MySQL privilege system works in relation to MySQL Cluster and the implications of this for keeping a MySQL Cluster secure.

Standard MySQL privileges apply to MySQL Cluster tables. This includes all MySQL privilege types (SELECT privilege, UPDATE privilege, DELETE privilege, and so on) granted on the database, table, and column level. As with any other MySQL Server, user and privilege information is stored in the mysql system database. The SQL statements used to grant and revoke privileges on NDB tables, databases containing such tables, and columns within such tables are identical in all respects with the GRANT and REVOKE statements used in connection with database objects involving any (other) MySQL storage engine. The same thing is true with respect to the CREATE USER and DROP USER statements.

It is important to keep in mind that the MySQL grant tables use the MyISAM storage engine. Because of this, those tables are not duplicated or shared among MySQL servers acting as SQL nodes in a MySQL Cluster. By way of example, suppose that two SQL nodes A and B are connected to the same MySQL Cluster, which has an NDB table named mytable in a database named mydb, and that you execute an SQL statement on server A that creates a new user jon@localhost and grants this user the SELECT privilege on that table:

mysql> GRANT SELECT ON mydb.mytable
    ->   TO jon@localhost IDENTIFIED BY 'mypass';

This user is not created on server B. In order for this to take place, the statement must also be run on server B. Similarly, statements run on server A and affecting the privileges of existing users on server A do not affect users on server B unless those statements are actually run on server B as well.

In other words, changes in users and their privileges do not automatically propagate between SQL nodes. Synchronization of privileges between SQL nodes must be done either manually or by scripting an application that periodically synchronizes the privilege tables on all SQL nodes in the cluster.

Conversely, because there is no way in MySQL to deny privileges (privileges can either be revoked or not granted in the first place, but not denied as such), there is no special protection for NDB tables on one SQL node from users that have privileges on another SQL node. The most far-reaching example of this is the MySQL root account, which can perform any action on any database object. In combination with empty [mysqld] or [api] sections of the config.ini file, this account can be especially dangerous. To understand why, consider the following scenario:

The config.ini file contains at least one empty [mysqld] or [api] section. This means that the Cluster management server performs no checking of the host from which a MySQL Server (or other API node) accesses the MySQL Cluster.
There is no firewall, or the firewall fails to protect against access to the Cluster from hosts external to the network.
The hostname or IP address of the Cluster's management server is known or can be determined from outside the network.

If these conditions are true, then anyone, anywhere can start a MySQL Server with --ndbcluster --ndb-connectstring=management_host and access the Cluster. Using the MySQL root account, this person can then perform the following actions:

Execute a SHOW DATABASES statement to obtain a list of all databases that exist in the cluster
Execute a SHOW TABLES FROM some_database statement to obtain a list of all NDB tables in a given database
Run any legal MySQL statements on any of those tables, such as:
- SELECT * FROM some_table to read all the data from any table
- DELETE FROM some_table to delete all the data from a table
- DESCRIBE some_table or SHOW CREATE TABLE some_table to determine the table schema
- UPDATE some_table SET column1 = any_value1 to fill a table column with “garbage” data; this could actually cause much greater damage than simply deleting all the data
  Even more insidious variations might include statements like these:
```
UPDATE some_table SET an_int_column = an_int_column + 1
```
  or
```
                  
UPDATE some_table SET a_varchar_column = REVERSE(a_varchar_column)
```
  Such malicious statements are limited only by the imagination of the attacker.
The only tables that would be safe from this sort of mayhem would be those tables that were created using storage engines other than NDB, and so not visible to a “rogue” SQL node.
Note
A user who can log in as root can also access the INFORMATION_SCHEMA database and its tables, and so obtain information about databases, tables, stored routines, scheduled events, and any other database objects for which metadata is stored in INFORMATION_SCHEMA.
It is also a very good idea to use different passwords for the root accounts on different cluster SQL nodes.

In sum, you cannot have a safe MySQL Cluster if it is directly accessible from outside your local network.

Important

Never leave the MySQL root account password empty. This is just as true when running MySQL as a MySQL Cluster SQL node as it is when running it as a standalone (non-Cluster) MySQL Server, and should be done as part of the MySQL installation process before configuring the MySQL Server as an SQL node in a MySQL Cluster.

You should never convert the system tables in the mysql database to use the NDB storage engine. There are a number of reasons why you should not do this, but the most important reason is this: Many of the SQL statements that affect mysql tables storing information about user privileges, stored routines, scheduled events, and other database objects cease to function if these tables are changed to use any storage engine other than MyISAM. This is a consequence of various MySQL Server internals which are not expected to change in the foreseeable future.

If you need to synchronize mysql system tables between SQL nodes, you can use standard MySQL replication to do so, or employ a script to copy table entries between the MySQL servers.

Summary. The two most important points to remember regarding the MySQL privilege system with regard to MySQL Cluster are:

Users and privileges established on one SQL node do not automatically exist or take effect on other SQL nodes in the cluster.
Conversely, removing a user or privilege on one SQL node in the cluster does not remove the user or privilege from any other SQL nodes.
Once a MySQL user is granted privileges on an NDB table from one SQL node in a MySQL Cluster, that user can “see” any data in that table regardless of the SQL node from which the data originated.

20.8.3. MySQL Cluster and MySQL Security Procedures

In this section, we discuss MySQL standard security procedures as they apply to running MySQL Cluster.

In general, any standard procedure for running MySQL securely also applies to running a MySQL Server as part of a MySQL Cluster. First and foremost, you should always run a MySQL Server as the mysql system user; this is no different from running MySQL in a standard (non-Cluster) environment. The mysql system account should be uniquely and clearly defined. Fortunately, this is the default behavior for a new MySQL installation. You can verify that the mysqld process is running as the system user mysql by using the system command such as the one shown here:

shell> ps aux | grep mysql
root     10467  0.0  0.1   3616  1380 pts/3    S    11:53   0:00 \ 
  /bin/sh ./mysqld_safe --ndbcluster --ndb-connectstring=localhost:1186
mysql    10512  0.2  2.5  58528 26636 pts/3    Sl   11:53   0:00 \ 
  /usr/local/mysql/libexec/mysqld --basedir=/usr/local/mysql \ 
  --datadir=/usr/local/mysql/var --user=mysql --ndbcluster \ 
  --ndb-connectstring=localhost:1186 --pid-file=/usr/local/mysql/var/mothra.pid \ 
  --log-error=/usr/local/mysql/var/mothra.err
jon      10579  0.0  0.0   2736   688 pts/0    S+   11:54   0:00 grep mysql

If the mysqld process is running as any other user than mysql, you should immediately shut it down and restart it as the mysql user. If this user does not exist on the system, the mysql user account should be created, and this user should be part of the mysql user group; in this case, you should also make sure that the MySQL DataDir on this system is owned by the mysql user, and that the SQL node's my.cnf file includes user=mysql in the [mysqld] section. Alternatively, you can start the server with --user=mysql on the command line, but it is preferable to use the my.cnf option, since you might forget to use the command-line option and so have mysqld running as another user unintentionally. The mysqld_safe startup script forces MySQL to run as the mysql user.

Important

Never run mysqld as the system root user. Doing so means that potentially any file on the system can be read by MySQL, and thus — should MySQL be compromised — by an attacker.

As mentioned in the previous section (see Section 20.8.2, “MySQL Cluster and MySQL Privileges”), you should always set a root password for the MySQL Server as soon as you have it running. You should also delete the anonymous user account that is installed by default. You can accomplish these tasks via the following statements:

shell< mysql -u root

mysql> UPDATE mysql.user 
    ->     SET Password=PASSWORD('secure_password')
    ->     WHERE User='root';

mysql> DELETE FROM mysql.user
    ->     WHERE User='';

mysql> FLUSH PRIVILEGES;

Be very careful when executing the DELETE statement not to omit the WHERE clause, or you risk deleting all MySQL users. Be sure to run the FLUSH PRIVILEGES statement as soon as you have modified the mysql.user table, so that the changes take immediate effect. Without FLUSH PRIVILEGES, the changes do not take effect until the next time that the server is restarted.

Note

Many of the MySQL Cluster utilities such as ndb_show_tables, ndb_desc, and ndb_select_all also work without authentication and can reveal table names, schemas, and data. By default these are installed on Unix-style systems with the permissions wxr-xr-x (755), which means they can be executed by any user that can access the mysql/bin directory.

See Section 20.10, “Cluster Utility Programs”, for more information about these utilities.

20.9. On-line Backup of MySQL Cluster

20.9.1. Cluster Backup Concepts
20.9.2. Using The Management Client to Create a Backup
20.9.3. ndb_restore — Restore a Cluster Backup
20.9.4. Configuration for Cluster Backup
20.9.5. Backup Troubleshooting

This section describes how to create a backup and how to restore the database from a backup at a later time.

20.9.1. Cluster Backup Concepts

A backup is a snapshot of the database at a given time. The backup consists of three main parts:

Metadata. The names and definitions of all database tables
Table records. The data actually stored in the database tables at the time that the backup was made
Transaction log. A sequential record telling how and when data was stored in the database

Each of these parts is saved on all nodes participating in the backup. During backup, each node saves these three parts into three files on disk:

BACKUP-backup_id.node_id.ctl
A control file containing control information and metadata. Each node saves the same table definitions (for all tables in the cluster) to its own version of this file.
BACKUP-backup_id-0.node_id.data
A data file containing the table records, which are saved on a per-fragment basis. That is, different nodes save different fragments during the backup. The file saved by each node starts with a header that states the tables to which the records belong. Following the list of records there is a footer containing a checksum for all records.
BACKUP-backup_id.node_id.log
A log file containing records of committed transactions. Only transactions on tables stored in the backup are stored in the log. Nodes involved in the backup save different records because different nodes host different database fragments.

In the listing above, backup_id stands for the backup identifier and node_id is the unique identifier for the node creating the file.

20.9.2. Using The Management Client to Create a Backup

Before starting a backup, make sure that the cluster is properly configured for performing one. (See Section 20.9.4, “Configuration for Cluster Backup”.)

Creating a backup using the management client involves the following steps:

Start the management client (ndb_mgm), if it not running already.
Execute the START BACKUP command. This produces several lines of output indicating the progress of the backup, as shown here:
```
ndb_mgm> START BACKUP
Waiting for completed, this may take several minutes
Node 2: Backup 1 started from node 1
Node 2: Backup 1 started from node 1 completed
 StartGCP: 177 StopGCP: 180
 #Records: 7362 #LogRecords: 0
 Data: 453648 bytes Log: 0 bytes
ndb_mgm>
```
The first line printed indicates that the management client is waiting for the backup to be completed before returning control to the client. This behavior is the default for the START BACKUP command, but can be changed. To specify when START BACKUP command should return control to the client, append NOWAIT, WAIT STARTED, or WAIT COMPLETED to the command. The effects that each of these has differs as follows:
- If NOWAIT is specified, the management client displays a prompt immediately, as seen here:
```
ndb_mgm> START BACKUP NOWAIT
ndb_mgm> 
```
  In this case, the management client can be used even while it prints progress information from the backup process.
- With WAIT STARTED the management client waits until the backup has started before returning control to the user, as shown here:
```
ndb_mgm> START BACKUP WAIT STARTED
Waiting for started, this may take several minutes
Node 2: Backup 3 started from node 1
ndb_mgm>
```
- WAIT COMPLETED explicitly specifies the default behavior — that is, it causes the management client to wait until the backup process is complete before returning control to the user.
When the backup has started the management client displays this message:
```
Backup backup_id started from node node_id
```
backup_id is the unique identifier for this particular backup. This identifier is saved in the cluster log, if it has not been configured otherwise. node_id is the identifier of the management server that is coordinating the backup with the data nodes. At this point in the backup process the cluster has received and processed the backup request. It does not mean that the backup has finished. An example of this statement is shown here:
```
	
Node 2: Backup 1 started from node 1
```
The management client indicates that the backup has completed with a message in the following format:
```
Backup backup_id started from node node_id completed
```
As is the case for the notification that the backup has started, backup_id is the unique identifier for this particular backup, and node_id is the node ID of the management server that is coordinating the backup with the data nodes. This output is accompanied by additional information including relevant global checkpoints, the number of records backed up, and the size of the data, as shown here:
```
	
Node 2: Backup 1 started from node 1 completed
 StartGCP: 177 StopGCP: 180
 #Records: 7362 #LogRecords: 0
 Data: 453648 bytes Log: 0 bytes
```

Cluster backups are created by default in the BACKUP subdirectory of the DataDir on each data node. This can be overridden for one or more data nodes individually, or for all cluster data nodes in the config.ini file using the BackupDataDir configuration parameter as discussed in Identifying Data Nodes. The backup files created for a backup with a given backup_id are stored in a subdirectory named BACKUP-backup_id in the backup directory.

To abort a backup that is already in progress:

Start the management client.
Execute this command:
```
ndb_mgm> ABORT BACKUP backup_id
```
The number backup_id is the identifier of the backup that was included in the response of the management client when the backup was started (in the message Backup backup_id started from node management_node_id).
The management client will acknowledge the abort request with Abort of backup backup_id ordered.
Note
At this point, the management client has not yet received a response from the cluster data nodes to this request, and the backup has not yet actually been aborted.
After the backup has been aborted, the management client will report this fact in a manner similar to what is shown here:
```
Node 1: Backup 3 started from 5 has been aborted. Error: 1321 - Backup aborted by user request: Permanent error: User defined error
Node 3: Backup 3 started from 5 has been aborted. Error: 1323 - 1323: Permanent error: Internal error
Node 2: Backup 3 started from 5 has been aborted. Error: 1323 - 1323: Permanent error: Internal error
Node 4: Backup 3 started from 5 has been aborted. Error: 1323 - 1323: Permanent error: Internal error
```
In this example, we have shown sample output for a cluster with 4 data nodes, where the sequence number of the backup to be aborted is 3, and the management node to which the cluster management client is connected has the node ID 5. The first node to complete its part in aborting the backup reports that the reason for the abort was due to a request by the user. (The remaining nodes report that the backup was aborted due to an unspecified internal error.)
Note
There is no guarantee that the cluster nodes respond to an ABORT BACKUP command in any particular order.
The Backup backup_id started from node management_node_id has been aborted messages mean that the backup has been terminated and that all files relating to this backup have been removed from the cluster filesystem.

It is also possible to abort a backup in progress from a system shell using this command:

shell> ndb_mgm -e "ABORT BACKUP backup_id"

Note

If there is no backup with ID backup_id running when an ABORT BACKUP is issued, the management client makes no response, nor is it indicated in the cluster log that an invalid abort command was sent.

20.9.3. ndb_restore — Restore a Cluster Backup

The cluster restoration program is implemented as a separate command-line utility ndb_restore, which can normally be found in the MySQL bin directory. This program reads the files created as a result of the backup and inserts the stored information into the database.

ndb_restore must be executed once for each of the backup files that were created by the START BACKUP command used to create the backup (see Section 20.9.2, “Using The Management Client to Create a Backup”). This is equal to the number of data nodes in the cluster at the time that the backup was created.

Note

Before using ndb_restore, it is recommended that the cluster be running in single user mode, unless you are restoring multiple data nodes in parallel. See Section 20.7.4, “Single User Mode”, for more information about single user mode.

Typical options for this utility are shown here:

ndb_restore [-c connectstring] -n node_id [-s] [-m] -b backup_id -r [backup_path=]/path/to/backup/files [-e]

The -c option is used to specify a connectstring which tells ndb_restore where to locate the cluster management server. (See Section 20.3.4.2, “The Cluster Connectstring”, for information on connectstrings.) If this option is not used, then ndb_restore attempts to connect to a management server on localhost:1186. This utility acts as a cluster API node, and so requires a free connection “slot” to connect to the cluster management server. This means that there must be at least one [api] or [mysqld] section that can be used by it in the cluster config.ini file. It is a good idea to keep at least one empty [api] or [mysqld] section in config.ini that is not being used for a MySQL server or other application for this reason (see Section 20.3.4.6, “Defining SQL and Other API Nodes”).

You can verify that ndb_restore is connected to the cluster by using the SHOW command in the ndb_mgm management client. You can also accomplish this from a system shell, as shown here:

shell> ndb_mgm -e "SHOW"

-n is used to specify the node ID of the data node on which the backups were taken.

The first time you run the ndb_restore restoration program, you also need to restore the metadata. In other words, you must re-create the database tables — this can be done by running it with the -m option. Note that the cluster should have an empty database when starting to restore a backup. (In other words, you should start ndbd with --initial prior to performing the restore. You should also remove manually any Disk Data files present in the data node's DataDir.)

It is possible to restore data without restoring table metadata. Prior to MySQL 5.1.17, ndb_restore did not perform any checks of table schemas; if a table was altered between the time the backup was taken and when ndb_restore was run, ndb_restore would still attempt to restore the data to the altered table.

Beginning with MySQL 5.1.17, the default behavior is for ndb_restore is to fail with an error if table data do not match the table schema; this can be overridden using the --skip-table-check or -s option. Prior to MySQL 5.1.21, if this option is used, then ndb_restore attempts to fit data into the existing table schema, but the result of restoring a backup to a table schema that does not match the original is unspecified.

Beginning with MySQL Cluster NDB 6.3.8, ndb_restore supports limited attribute promotion in much the same way that it is supported by MySQL replication; that is, data backed up from a column of a given type can generally be restored to a column using a “larger, similar” type. For example, data from a CHAR(20) column can be restored to a column declared as VARCHAR(20), VARCHAR(30), or CHAR(30); data from a MEDIUMINT column can be restored to a column of type INT or BIGINT. See Section 19.3.1.22.2, “Replication of Columns Having Different Data Types”, for a table of type conversions currently supported by attribute promotion.

Attribute promotion by ndb_restore must be enabled explicitly, as follows:

Prepare the table to which the backup is to be restored. ndb_restore cannot be used to re-create the table with a different definition from the original; this means that you must either create the table manually, or alter the columns which you wish to promote using ALTER TABLE after restoring the table metadata but before restoring the data.
Invoke ndb_restore with the --promote-attributes option (short form -A) when restoring the table data. Attribute promotion does not occur if this option is not used; instead, the restore operation fails with an error.

In addition to --promote-attributes, a --preserve-trailing-spaces option is also available for use with ndb_restore beginning with MySQL 5.1.23-ndb-6.3.8. This option (short form -R) causes trailing spaces to be preserved when promoting a CHAR column to VARCHAR or a BINARY column to VARBINARY. Otherwise, any trailing spaces are dropped from column values when they are inserted into the new columns.

Note

Although you can promote CHAR columns to VARCHAR and BINARY columns to VARBINARY, you cannot promote VARCHAR columns to CHAR or VARBINARY columns to BINARY.

The -b option is used to specify the ID or sequence number of the backup, and is the same number shown by the management client in the Backup backup_id completed message displayed upon completion of a backup. (See Section 20.9.2, “Using The Management Client to Create a Backup”.)

-e adds (or restores) epoch information to the cluster replication status table. This is useful for starting replication on a MySQL Cluster replication slave. When this option is used, the row in the mysql.ndb_apply_status having 0 in the id column is updated if it already exists; such a row is inserted if it does not already exist. (See Section 20.11.9, “MySQL Cluster Backups With Replication”.)

The path to the backup directory is required, and must include the subdirectory corresponding to the ID backup of the backup to be restored. For example, if the data node's DataDir is /var/lib/mysql-cluster, then the backup directory is /var/lib/mysql-cluster/BACKUP, and the backup files for the backup with the ID 3 can be found in /var/lib/mysql-cluster/BACKUP/BACKUP-3. The path may be absolute or relative to the directory in which the ndb_restore executable is located, and may be optionally prefixed with backup_path=.

Important

When restoring cluster backups, you must be sure to restore all data nodes from backups having the same backup ID. Using files from different backups will at best result in restoring the cluster to an inconsistent state, and may fail altogether.

Important

It is not possible to restore a backup made from a newer version of MySQL Cluster using an older version of ndb_restore. You can restore a backup made from a newer version of MySQL to an older cluster, but you must use a copy of ndb_restore from the newer MySQL Cluster version to do so.

For example, to restore a cluster backup taken from a cluster running MySQL 5.1.23 to a cluster running MySQL Cluster 5.1.20, you must use a copy of ndb_restore from the 5.1.23 distribution.

It is possible to restore a backup to a database with a different configuration than it was created from. For example, suppose that a backup with backup ID 12, created in a cluster with two database nodes having the node IDs 2 and 3, is to be restored to a cluster with four nodes. Then ndb_restore must be run twice — once for each database node in the cluster where the backup was taken. However, ndb_restore cannot always restore backups made from a cluster running one version of MySQL to a cluster running a different MySQL version. See Section 20.5.2, “MySQL Cluster 5.1 and MySQL Cluster NDB 6.x Upgrade and Downgrade Compatibility”, for more information.

Note

For more rapid restoration, the data may be restored in parallel, provided that there is a sufficient number of cluster connections available. That is, when restoring to multiple nodes in parallel, you must have an [api] or [mysqld] section in the cluster config.ini file available for each concurrent ndb_restore process. However, the data files must always be applied before the logs.

Formerly, when using ndb_restore to restore a backup made from a MySQL 5.0 cluster to a 5.1 cluster, VARCHAR columns were not resized and were recreated using the 5.0 fixed format. Beginning with MySQL 5.1.19, ndb_restore recreates such VARCHAR columns using MySQL Cluster 5.1's variable-width format. Also beginning with MySQL 5.1.19, this behavior can be overridden using the --no-upgrade option (short form: -u) when running ndb_restore.

Most of the options available for this program are shown in the following table:

Long Form	Short Form	Description	Default Value
`--backup-id`	`-b`	Backup sequence ID	`0`
`--backup_path`	None	Path to backup files	`./`
`--character-sets-dir`	None	Specify the directory where character set information can be found	None
`--connect`, `--connectstring`, or `--ndb-connectstring`	`-c` or `-C`	Set the connectstring in `[nodeid=node_id;][host=]host[:port]` format	`localhost:1186`
`--core-file`	None	Write a core file in the event of an error	`TRUE`
`--debug`	`-#`	Output debug log	`d:t:O,/tmp/ndb_restore.trace`
`--dont_ignore_systab_0`	`-f`	Do not ignore system table during restore — EXPERIMENTAL; not for production use	`FALSE`
`--help` or `--usage`	`-?`	Display help message with available options and current values, then exit	[N/A]
`--ndb-mgmd-host`	None	Set the host and port in `host[:port]` format for the management server to connect to; this is the same as `--connect`, `--connectstring`, or `--ndb-connectstring`, but without a way to specify the `nodeid`	None
`--ndb-nodegroup-map`	`-z`	Specifies a nodegroup map — Syntax: list of (`source_nodegroup`, `destination_nodegroup`)	None
`--ndb-nodeid`	None	Specify a node ID for the ndb_restore process	`0`
`--ndb-optimized-node-selection`	None	Optimize selection of nodes for transactions	`TRUE`
`--ndb-shm`	None	Use shared memory connections when available	`FALSE`
`--no-binlog`	None	Do not write anything to mysqld binary logs (added in MySQL Cluster NDB 6.2.16 and 6.3.16)	`FALSE` (in other words, write to binary logs unless this option is used)
`--no-restore-disk-objects`	`-d`	Do not restore Disk Data objects such as tablespaces and log file groups	`FALSE` (in other words, restore Disk Data objects unless this option is used)
`--no-upgrade`	`-u`	Do not re-create `VARSIZE` columns from a MySQL 5.0 Cluster backup as variable-width columns (added in MySQL 5.1.19)	`FALSE` (in other words, re-create `VARSIZE` columns from a MySQL 5.0 Cluster backup as variable-width columns unless this option is used)
`--nodeid`	`-n`	Use backup files from node with the specified ID	`0`
`--parallelism`	`-p`	Set from 1 to 1024 parallel transactions to be used during the restoration process	`128`
`--print`	None	Print metadata, data, and log to `stdout`	`FALSE`
`--print_data`	None	Print data to `stdout`	`FALSE`
`--print_log`	None	Print log to `stdout`	`FALSE`
`--print_meta`	None	Print metadata to `stdout`	`FALSE`
`--restore_data`	`-r`	Restore data and logs	`FALSE`
`--restore_epoch`	`-e`	Restore epoch data into the status table; the row in the `cluster.apply_status` having the id `0` is inserted or updated as appropriate — this is convenient when starting up replication on a MySQL Cluster replication slave	`FALSE`
`--restore_meta`	`-m`	Restore table metadata	`FALSE`
`--skip-table-check`	`-s`	Do not check table schemas (Added in MySQL 5.1.17)	`FALSE`
`--version`	`-V`	Output version information and exit	[N/A]

Beginning with MySQL 5.1.18, several additional options are available for use with the --print_data option in generating data dumps, either to stdout, or to a file. These are similar to some of the options used with mysqldump, and are shown in the following table:

Long Form	Short Form	Description	Default Value
`--tab`	`-T`	Creates dumpfiles, one per table, each named `tbl_name.txt`. Takes as its argument the path to the directory where the files should be saved (required; use `.` for the current directory).	None
`--fields-enclosed-by`	None	String used to enclose all column values	None
`--fields-optionally-enclosed-by`	None	String used to enclose column values containing character data (such as `CHAR`, `VARCHAR`, `BINARY`, `TEXT`, or `ENUM`)	None
`--fields-terminated-by`	None	String used to separate column values	`\t` (tab character)
`--hex`	None	Use hex format for binary values	[N/A]
`--lines-terminated-by`	None	String used to terminate each line	`\n` (linefeed character)
`--append`	None	When used with `--tab`, causes the data to be appended to existing files of the same name	[N/A]

Note

If a table has no explicit primary key, then the output generated when using the --print includes the table's hidden primary key.

Beginning with MySQL 5.1.18, it is possible to restore selected databases, or to restore selected tables from a given database using the syntax shown here:

ndb_restore other_options db_name_1 [db_name_2[, db_name_3][, ...] | tbl_name_1[, tbl_name_2][, ...]]

In other words, you can specify either of the following to be restored:

All tables from one or more databases
One or more tables from a single database

Note

ndb_restore reports both temporary and permanent errors. In the case of temporary errors, it may able to recover from them. Beginning with MySQL 5.1.12, it reports Restore successful, but encountered temporary error, please look at configuration in such cases.

20.9.4. Configuration for Cluster Backup

Five configuration parameters are essential for backup:

BackupDataBufferSize
The amount of memory used to buffer data before it is written to disk.
BackupLogBufferSize
The amount of memory used to buffer log records before these are written to disk.
BackupMemory
The total memory allocated in a database node for backups. This should be the sum of the memory allocated for the backup data buffer and the backup log buffer.
BackupWriteSize
The default size of blocks written to disk. This applies for both the backup data buffer and the backup log buffer.
BackupMaxWriteSize
The maximum size of blocks written to disk. This applies for both the backup data buffer and the backup log buffer.

More detailed information about these parameters can be found in Backup Parameters.

20.9.5. Backup Troubleshooting

If an error code is returned when issuing a backup request, the most likely cause is insufficient memory or disk space. You should check that there is enough memory allocated for the backup.

Important

If you have set BackupDataBufferSize and BackupLogBufferSize and their sum is greater than 4MB, then you must also set BackupMemory as well. See BackupMemory.

You should also make sure that there is sufficient space on the hard drive partition of the backup target.

NDB does not support repeatable reads, which can cause problems with the restoration process. Although the backup process is “hot”, restoring a MySQL Cluster from backup is not a 100% “hot” process. This is due to the fact that, for the duration of the restore process, running transactions get non-repeatable reads from the restored data. This means that the state of the data is inconsistent while the restore is in progress.

MySQL Enterprise MySQL Enterprise subscribers will find more information about Cluster backup in the Knowledge Base article, How Do I Backup my Cluster Database. Access to the MySQL Knowledge Base collection of articles is one of the advantages of subscribing to MySQL Enterprise. For more information, see http://www.mysql.com/products/enterprise/advisors.html.

20.10. Cluster Utility Programs

20.10.1. ndb_config — Extract NDB Configuration Information
20.10.2. ndb_cpcd — Automate Testing for NDB Development
20.10.3. ndb_delete_all — Delete All Rows from NDB Table
20.10.4. ndb_desc — Describe NDB Tables
20.10.5. ndb_drop_index — Drop Index from NDB Table
20.10.6. ndb_drop_table — Drop NDB Table
20.10.7. ndb_error_reporter — NDB Error-Reporting Utility
20.10.8. ndb_print_backup_file — Print NDB Backup File Contents
20.10.9. ndb_print_schema_file — Print NDB Schema File Contents
20.10.10. ndb_print_sys_file — Print NDB System File Contents
20.10.11. ndbd_redo_log_reader — Check and Print Content of Cluster Redo Log
20.10.12. ndb_select_all — Print Rows from NDB Table
20.10.13. ndb_select_count — Print Row Counts for NDB Tables
20.10.14. ndb_show_tables — Display List of NDB Tables
20.10.15. ndb_size.pl — NDBCLUSTER Size Requirement Estimator
20.10.16. ndb_waiter — Wait for Cluster to Reach a Given Status

This section discusses the MySQL Cluster utility programs that can be found in the mysql/bin directory. Each of these — except for ndb_size.pl and ndb_error_reporter — is a standalone binary that can be used from a system shell, and that does not need to connect to a MySQL server (nor even requires that a MySQL server be connected to the cluster).

These utilities can also serve as examples for writing your own applications using the NDB API. The source code for most of these programs may be found in the storage/ndb/tools directory of the MySQL 5.1 tree (see Section 2.9, “MySQL Installation Using a Source Distribution”). The NDB API is not covered in this manual; please refer to the NDB API Guide for information about this API.

All of the NDB utilities are listed here with brief descriptions:

ndb_config: Retrieves Cluster configuration option values.
ndb_cpcd: Used in testing and debugging MySQL Cluster.
ndb_delete_all: Deletes all rows from a given table.
ndb_desc: Lists all properties of an NDB table.
ndb_drop_index: Drops the specified index from an NDB table.
ndb_drop_table: Drops an NDB table.
ndb_error_reporter: Can be used to gather information useful for diagnosing problems with the cluster.
ndb_mgm: This is the MySQL Cluster management client, which is discussed in Section 20.7.2, “Commands in the MySQL Cluster Management Client”.
ndb_print_backup_file: Prints diagnostic information obtained from cluster backup files.
ndb_print_schema_file: Prints diagnostic information obtained from cluster schema files.
ndb_print_sys_file: Prints diagnostic information obtained from cluster system files.
ndb_restore: This utility is used to restore a cluster from backup. See Section 20.9.3, “ndb_restore — Restore a Cluster Backup”, for more information.
ndb_select_all: Prints all rows from an NDB table.
ndb_select_count: Gets the number of rows in one or more NDB tables.
ndb_show_tables: Shows all NDB tables anywhere in the cluster.
ndb_size.pl: Examines all the tables in a given non-Cluster database and calculates the amount of storage each would require if it were converted to use the NDB storage engine.
ndb_waiter: Reports on the status of cluster data nodes in a manner similar to that of the management client command ALL STATUS.
ndbd_redo_log_reader: Reads a redo log file, checking it for errors, printing it in a human-readable format, or both.
Note
An alpha version of this utility was made available in MySQL Cluster NDB 6.1.3. Currently, it should be considered experimental.

Most of these utilities need to connect to a Cluster management server in order to function. The exceptions are ndb_size.pl (see below), and the following utilities which access a cluster data node filesystem and so need to be run on a data node host:

ndb_print_backup_file
ndb_print_schema_file
ndb_print_sys_file
ndbd_redo_log_reader

ndb_size.pl is a Perl script which is also intended to be used from the shell; however it is a MySQL application and must be able to connect to a MySQL server. See Section 20.10.15, “ndb_size.pl — NDBCLUSTER Size Requirement Estimator”, for additional requirements for using this script.

ndb_error_reporter is also a Perl script. It is used to gather cluster data node and management node logs together into a tarball to submit along with a bug report. It can use ssh or scp to access the node filesystems remotely.

Additional information about each of these utilities (except for ndb_mgm and ndb_restore) can be found in the sections that follow.

Note

All of these utilities (except for ndb_size.pl and ndb_config) can use the options discussed in Section 20.6.5, “Command Options for MySQL Cluster Processes”. Additional options specific to each utility program are discussed in the individual program listings.

The order in which these options are used is generally not important. For example, all of these commands produce exactly the same output:

ndb_desc -c localhost fish -d test
ndb_desc fish -c localhost -d test
ndb_desc -d test fish -c localhost

20.10.1. ndb_config — Extract NDB Configuration Information

This tool extracts configuration information for data nodes, SQL nodes, and API nodes from a cluster management node (and possibly its config.ini file).

Usage:

ndb_config options

The options available for this utility differ somewhat from those used with the other utilities, and so are listed in their entirety in the next section, followed by some examples.

Options:

--usage, --help, or -?
Causes ndb_config to print a list of available options, and then exit.
--version, -V
Causes ndb_config to print a version information string, and then exit.
--ndb-connectstring=connect_string
Specifies the connectstring to use in connecting to the management server. The format for the connectstring is the same as described in Section 20.3.4.2, “The Cluster Connectstring”, and defaults to localhost:1186.
The use of -c as a short version for this option is supported for ndb_config beginning with MySQL 5.1.12.
--config-file=path-to-file
Gives the path to the management server's configuration file (config.ini). This may be a relative or absolute path. If the management node resides on a different host from the one on which ndb_config is invoked, then an absolute path must be used.
--query=query-options, -q query-options
This is a comma-delimited list of query options — that is, a list of one or more node attributes to be returned. These include id (node ID), type (node type — that is, ndbd, mysqld, or ndb_mgmd), and any configuration parameters whose values are to be obtained.
For example, --query=id,type,indexmemory,datamemory would return the node ID, node type, DataMemory, and IndexMemory for each node.
Note
If a given parameter is not applicable to a certain type of node, than an empty string is returned for the corresponding value. See the examples later in this section for more information.
--host=hostname
Specifies the hostname of the node for which configuration information is to be obtained.
--id=node_id, --nodeid=node_id
Used to specify the node ID of the node for which configuration information is to be obtained.
--nodes
(Tells ndb_config to print information from parameters defined in [ndbd] sections only. Currently, using this option has no affect, since these are the only values checked, but it may become possible in future to query parameters set in [tcp] and other sections of cluster configuration files.)
--type=node_type
Filters results so that only configuration values applying to nodes of the specified node_type (ndbd, mysqld, or ndb_mgmd) are returned.
--fields=delimiter, -f delimiter
Specifies a delimiter string used to separate the fields in the result. The default is “,” (the comma character).
Note
If the delimiter contains spaces or escapes (such as \n for the linefeed character), then it must be quoted.
--rows=separator, -r separator
Specifies a separator string used to separate the rows in the result. The default is a space character.
Note
If the separator contains spaces or escapes (such as \n for the linefeed character), then it must be quoted.

Examples:

To obtain the node ID and type of each node in the cluster:
```
shell> ./ndb_config --query=id,type --fields=':' --rows='\n'
1:ndbd
2:ndbd
3:ndbd
4:ndbd
5:ndb_mgmd
6:mysqld
7:mysqld
8:mysqld
9:mysqld
```
In this example, we used the --fields options to separate the ID and type of each node with a colon character (:), and the --rows options to place the values for each node on a new line in the output.

To produce a connectstring that can be used by data, SQL, and API nodes to connect to the management server:

shell> ./ndb_config --config-file=usr/local/mysql/cluster-data/config.ini --query=hostname,portnumber --fields=: --rows=, --type=ndb_mgmd
192.168.0.179:1186

This invocation of ndb_config checks only data nodes (using the --type option), and shows the values for each node's ID and hostname, and its DataMemory, IndexMemory, and DataDir parameters:

shell> ./ndb_config --type=ndbd --query=id,host,datamemory,indexmemory,datadir -f ' : ' -r '\n'
1 : 192.168.0.193 : 83886080 : 18874368 : /usr/local/mysql/cluster-data
2 : 192.168.0.112 : 83886080 : 18874368 : /usr/local/mysql/cluster-data
3 : 192.168.0.176 : 83886080 : 18874368 : /usr/local/mysql/cluster-data
4 : 192.168.0.119 : 83886080 : 18874368 : /usr/local/mysql/cluster-data

In this example, we used the short options -f and -r for setting the field delimiter and row separator, respectively.

To exclude results from any host except one in particular, use the --host option:
```
shell> ./ndb_config --host=192.168.0.176 -f : -r '\n' -q id,type
3:ndbd
5:ndb_mgmd
```
In this example, we also used the short form -q to determine the attributes to be queried.
Similarly, you can limit results to a node with a specific ID using the --id or --nodeid option.

20.10.2. ndb_cpcd — Automate Testing for NDB Development

This utility is found in the libexec directory. It is part of an internal automated test framework used in testing and bedugging MySQL Cluster. Because it can control processes on remote systems, it is not advisable to use ndb_cpcd in a production cluster.

The source files for ndb_cpcd may be found in the directory storage/ndb/src/cw/cpcd, in the MySQL 5.1 source tree.

20.10.3. ndb_delete_all — Delete All Rows from NDB Table

ndb_delete_all deletes all rows from the given NDB table. In some cases, this can be much faster than DELETE or even TRUNCATE.

Usage:

ndb_delete_all -c connect_string tbl_name -d db_name

This deletes all rows from the table named tbl_name in the database named db_name. It is exactly equivalent to executing TRUNCATE db_name.tbl_name in MySQL.

Additional Options:

--transactional, -t
Use of this option causes the delete operation to be performed as a single transaction.
Warning
With very large tables, using this option may cause the number of operations available to the cluster to be exceeded.

20.10.4. ndb_desc — Describe NDB Tables

ndb_desc provides a detailed description of one or more NDB tables.

Usage:

ndb_desc -c connect_string tbl_name -d db_name [-p]

Sample Output:

MySQL table creation and population statements:

USE test;

CREATE TABLE fish (
    id INT(11) NOT NULL AUTO_INCREMENT,
    name VARCHAR(20),

    PRIMARY KEY pk (id),
    UNIQUE KEY uk (name)
) ENGINE=NDBCLUSTER;

INSERT INTO fish VALUES 
    ('','guppy'), ('','tuna'), ('','shark'), 
    ('','manta ray'), ('','grouper'), ('','puffer');

Output from ndb_desc:

shell> ./ndb_desc -c localhost fish -d test -p
-- fish --
Version: 16777221
Fragment type: 5
K Value: 6
Min load factor: 78
Max load factor: 80
Temporary table: no
Number of attributes: 2
Number of primary keys: 1
Length of frm data: 268
Row Checksum: 1
Row GCI: 1
TableStatus: Retrieved
-- Attributes --
id Int PRIMARY KEY DISTRIBUTION KEY AT=FIXED ST=MEMORY
name Varchar(20;latin1_swedish_ci) NULL AT=SHORT_VAR ST=MEMORY

-- Indexes --
PRIMARY KEY(id) - UniqueHashIndex
uk(name) - OrderedIndex
PRIMARY(id) - OrderedIndex
uk$unique(name) - UniqueHashIndex

-- Per partition info --
Partition  Row count  Commit count  Frag fixed memory  Frag varsized memory
2          2          2             65536              327680
1          2          2             65536              327680
3          2          2             65536              327680

NDBT_ProgramExit: 0 - OK

Additional Options:

--extra-partition-info, -p
Prints additional information about the table's partitions.
Information about multiple tables can be obtained in a single invocation of ndb_desc by using their names, separated by spaces. All of the tables must be in the same database.

20.10.5. ndb_drop_index — Drop Index from NDB Table

ndb_drop_index drops the specified index from an NDB table. It is recommended that you use this utility only as an example for writing NDB API applications — see the Warning later in this section for details.

Usage:

ndb_drop_index -c connect_string table_name index -d db_name

The statement shown above drops the index named index from the table in the database.

Additional Options: None that are specific to this application.

Warning

Operations performed on Cluster table indexes using the NDB API are not visible to MySQL and make the table unusable by a MySQL server. If you use this program to drop an index, then try to access the table from an SQL node, an error results, as shown here:

shell> ./ndb_drop_index -c localhost dogs ix -d ctest1
Dropping index dogs/idx...OK

NDBT_ProgramExit: 0 - OK

shell> ./mysql -u jon -p ctest1
Enter password: *******
Reading table information for completion of table and column names
You can turn off this feature to get a quicker startup with -A

Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 7 to server version: 5.1.12-beta-20060817

Type 'help;' or '\h' for help. Type '\c' to clear the buffer.

mysql> SHOW TABLES;
+------------------+
| Tables_in_ctest1 |
+------------------+
| a                |
| bt1              |
| bt2              |
| dogs             |
| employees        |
| fish             |
+------------------+
6 rows in set (0.00 sec)

mysql> SELECT * FROM dogs;
ERROR 1296 (HY000): Got error 4243 'Index not found' from NDBCLUSTER

In such a case, your only option for making the table available to MySQL again is to drop the table and re-create it. You can use either the SQL statementDROP TABLE or the ndb_drop_table utility (see Section 20.10.6, “ndb_drop_table — Drop NDB Table”) to drop the table.

20.10.6. ndb_drop_table — Drop NDB Table

ndb_drop_table drops the specified NDB table. (If you try to use this on a table created with a storage engine other than NDB, it fails with the error 723: No such table exists.) This operation is extremely fast — in some cases, it can be an order of magnitude faster than using DROP TABLE on an NDB table from MySQL.

Usage:

ndb_drop_table -c connect_string tbl_name -d db_name

Additional Options: None.

20.10.7. ndb_error_reporter — NDB Error-Reporting Utility

ndb_error_reporter creates an archive from data node and management node log files that can be used to help diagnose bugs or other problems with a cluster. It is highly recommended that you make use of this utility when filing reports of bugs in MySQL Cluster.

Usage:

ndb_error_reporter path/to/config-file [username] [--fs]

This utility is intended for use on a management node host, and requires the path to the management host configuration file (config.ini). Optionally, you can supply the name of a user that is able to access the cluster's data nodes via SSH, in order to copy the data node log files. ndb_error_reporter then includes all of these files in archive that is created in the same directory in which it is run. The archive is named ndb_error_report_YYYYMMDDHHMMSS.tar.bz2, where YYYYMMDDHHMMSS is a datetime string.

If the --fs is used, then the data node filesystems are also copied to the management host and included in the archive that is produced by this script. As data node filesystems can be extremely large even after being compressed, we ask that you please do not send archives created using this option to MySQL AB unless you are specifically requested to do so.

20.10.8. ndb_print_backup_file — Print NDB Backup File Contents

ndb_print_backup_file obtains diagnostic information from a cluster backup file.

Usage:

ndb_print_backup_file file_name

file_name is the name of a cluster backup file. This can be any of the files (.Data, .ctl, or .log file) found in a cluster backup directory. These files are found in the data node's backup directory under the subdirectory BACKUP-#, where # is the sequence number for the backup. For more information about cluster backup files and their contents, see Section 20.9.1, “Cluster Backup Concepts”.

Like ndb_print_schema_file and ndb_print_sys_file (and unlike most of the other NDB utilities that are intended to be run on a management server host or to connect to a management server) ndb_print_backup_file must be run on a cluster data node, since it accesses the data node filesystem directly. Because it does not make use of the management server, this utility can be used when the management server is not running, and even when the cluster has been completely shut down.

Additional Options: None.

20.10.9. ndb_print_schema_file — Print NDB Schema File Contents

ndb_print_schema_file obtains diagnostic information from a cluster schema file.

Usage:

ndb_print_schema_file file_name

file_name is the name of a cluster schema file. For more information about cluster schema files, see Cluster Data Node FileSystemDir Files.

Like ndb_print_backup_file and ndb_print_sys_file (and unlike most of the other NDB utilities that are intended to be run on a management server host or to connect to a management server) ndb_schema_backup_file must be run on a cluster data node, since it accesses the data node filesystem directly. Because it does not make use of the management server, this utility can be used when the management server is not running, and even when the cluster has been completely shut down.

Additional Options: None.

20.10.10. ndb_print_sys_file — Print NDB System File Contents

ndb_print_sys_file obtains diagnostic information from a cluster system file.

Usage:

ndb_print_sys_file file_name

file_name is the name of a cluster system file (sysfile). Cluster system files are located in a data node's data directory (DataDir); the path under this directory to system files matches the pattern ndb_#_fs/D#/DBDIH/P#.sysfile. In each case, the # represents a number (not necessarily the same number). For more information, see Cluster Data Node FileSystemDir Files.

Like ndb_print_backup_file and ndb_print_schema_file (and unlike most of the other NDB utilities that are intended to be run on a management server host or to connect to a management server) ndb_print_backup_file must be run on a cluster data node, since it accesses the data node filesystem directly. Because it does not make use of the management server, this utility can be used when the management server is not running, and even when the cluster has been completely shut down.

Additional Options: None.

20.10.11. ndbd_redo_log_reader — Check and Print Content of Cluster Redo Log

Reads a redo log file, checking it for errors, printing its contents in a human-readable format, or both. ndbd_redo_log_reader is intended for use primarily by MySQL developers and support personnel in debugging and diagnosing problems.

This utility was made available as part of default builds beginning with MySQL Cluster NDB 6.1.3. It remains under development, and its syntax and behavior are subject to change in future releases. For this reason, it should be considered experimental at this time.

The C++ source files for ndbd_redo_log_reader can be found in the directory /storage/ndb/src/kernel/blocks/dblqh/redoLogReader.

Usage:

ndbd_redo_log_reader file_name [options]

file_name is the name of a cluster REDO log file. REDO log files are located in the numbered directories under the data node's data directory (DataDir); the path under this directory to the REDO log files matches the pattern ndb_#_fs/D#/LCP/#/T#F#.Data. In each case, the # represents a number (not necessarily the same number). For more information, see Cluster Data Node FileSystemDir Files.

Additional Options:

The name of the file to be read may be followed by one or more of the options listed here:

-noprint: Do not print the contents of the log file.
-nocheck: Do not check the log filre for errors.

Like ndb_print_backup_file and ndb_print_schema_file (and unlike most of the NDB utilities that are intended to be run on a management server host or to connect to a management server) ndbd_redo_log_reader must be run on a cluster data node, since it accesses the data node filesystem directly. Because it does not make use of the management server, this utility can be used when the management server is not running, and even when the cluster has been completely shut down.

20.10.12. ndb_select_all — Print Rows from NDB Table

ndb_select_all prints all rows from an NDB table to stdout.

Usage:

ndb_select_all -c connect_string tbl_name -d db_name [> file_name]

Additional Options:

--lock=lock_type, -l lock_type
Employs a lock when reading the table. Possible values for lock_type are:
- 0: Read lock
- 1: Read lock with hold
- 2: Exclusive read lock
There is no default value for this option.
--order=index_name, -o index_name
Orders the output according to the index named index_name. Note that this is the name of an index, not of a column, and that the index must have been explicitly named when created.
--descending, -z
Sorts the output in descending order. This option can be used only in conjunction with the -o (--order) option.
--header=FALSE
Excludes column headers from the output.
--useHexFormat -x
Causes all numeric values to be displayed in hexadecimal format. This does not affect the output of numerals contained in strings or datetime values.
--delimiter=character, -D character
Causes the character to be used as a column delimiter. Only table data columns are separated by this delimiter.
The default delimiter is the tab character.
--disk
Adds a disk reference column to the output. The column is non-empty only for Disk Data tables having non-indexed columns.
--rowid
Adds a ROWID column providing information about the fragments in which rows are stored.
--gci
Adds a column to the output showing the global checkpoint at which each row was last updated. See Section 20.16, “MySQL Cluster Glossary”, and Section 20.7.3.2, “Log Events”, for more information about checkpoints.
--tupscan, -t
Scan the table in the order of the tuples.
--nodata
Causes any table data to be omitted.

Sample Output:

Output from a MySQL SELECT statement:

mysql> SELECT * FROM ctest1.fish;
+----+-----------+
| id | name      |
+----+-----------+
|  3 | shark     |
|  6 | puffer    |
|  2 | tuna      |
|  4 | manta ray |
|  5 | grouper   |
|  1 | guppy     |
+----+-----------+
6 rows in set (0.04 sec)

Output from the equivalent invocation of ndb_select_all:

shell> ./ndb_select_all -c localhost fish -d ctest1
id      name
3       [shark]
6       [puffer]
2       [tuna]
4       [manta ray]
5       [grouper]
1       [guppy]
6 rows returned

NDBT_ProgramExit: 0 - OK

Note that all string values are enclosed by square brackets (“[...]”) in the output of ndb_select_all. For a further example, consider the table created and populated as shown here:

CREATE TABLE dogs (
    id INT(11) NOT NULL AUTO_INCREMENT,
    name VARCHAR(25) NOT NULL,
    breed VARCHAR(50) NOT NULL,
    PRIMARY KEY pk (id),
    KEY ix (name)
) 
TABLESPACE ts STORAGE DISK 
ENGINE=NDB;

INSERT INTO dogs VALUES 
    ('', 'Lassie', 'collie'),
    ('', 'Scooby-Doo', 'Great Dane'),
    ('', 'Rin-Tin-Tin', 'Alsatian'),
    ('', 'Rosscoe', 'Mutt');

This demonstrates the use of several additional ndb_select_all options:

shell> ./ndb_select_all -d ctest1 dogs -o ix -z --gci --disk        
GCI     id name          breed        DISK_REF
834461  2  [Scooby-Doo]  [Great Dane] [ m_file_no: 0 m_page: 98 m_page_idx: 0 ]
834878  4  [Rosscoe]     [Mutt]       [ m_file_no: 0 m_page: 98 m_page_idx: 16 ]
834463  3  [Rin-Tin-Tin] [Alsatian]   [ m_file_no: 0 m_page: 34 m_page_idx: 0 ]
835657  1  [Lassie]      [Collie]     [ m_file_no: 0 m_page: 66 m_page_idx: 0 ]
4 rows returned

NDBT_ProgramExit: 0 - OK

20.10.13. ndb_select_count — Print Row Counts for NDB Tables

ndb_select_count prints the number of rows in one or more NDB tables. With a single table, the result is equivalent to that obtained by using the MySQL statement SELECT COUNT(*) FROM tbl_name.

Usage:

ndb_select_count [-c connect_string] -ddb_name tbl_name[, tbl_name2[, ...]]

Additional Options: None that are specific to this application. However, you can obtain row counts from multiple tables in the same database by listing the table names separated by spaces when invoking this command, as shown under Sample Output.

Sample Output:

shell> ./ndb_select_count -c localhost -d ctest1 fish dogs
6 records in table fish
4 records in table dogs

NDBT_ProgramExit: 0 - OK

20.10.14. ndb_show_tables — Display List of NDB Tables

ndb_show_tables displays a list of all NDB database objects in the cluster. By default, this includes not only both user-created tables and NDB system tables, but NDB-specific indexes, internal triggers, and Cluster Disk Data objects as well.

Usage:

ndb_show_tables [-c connect_string]

Additional Options:

--loops, -l
Specifies the number of times the utility should execute. This is 1 when this option is not specified, but if you do use the option, you must supply an integer argument for it.
--parsable, -p
Using this option causes the output to be in a format suitable for use with LOAD DATA INFILE.
--type, -t
Can be used to restrict the output to one type of object, specified by an integer type code as shown here:
- 1: System table
- 2: User-created table
- 3: Unique hash index
Any other value causes all NDB database objects to be listed (the default).
--unqualified, -u
If specified, this causes unqualified object names to be displayed.

Note

Only user-created Cluster tables may be accessed from MySQL; system tables such as SYSTAB_0 are not visible to mysqld. However, you can examine the contents of system tables using NDB API applications such as ndb_select_all (see Section 20.10.12, “ndb_select_all — Print Rows from NDB Table”).

20.10.15. ndb_size.pl — NDBCLUSTER Size Requirement Estimator

This is a Perl script that can be used to estimate the amount of space that would be required by a MySQL database if it were converted to use the NDBCLUSTER storage engine. Unlike the other utilities discussed in this section, it does not require access to a MySQL Cluster (in fact, there is no reason for it to do so). However, it does need to access the MySQL server on which the database to be tested resides.

Requirements:

A running MySQL server. The server instance does not have to provide support for MySQL Cluster.
A working installation of Perl.
The DBI module, which can be obtained from CPAN if it is not already part of your Perl installation. (Many Linux and other operating system distributions provide their own packages for this library.)
Note
Previous to MySQL 5.1.18, ndb_size.pl also required the HTML::Template module and an associated template file share/mysql/ndb_size.tmpl. Beginning with MySQL 5.1.18, ndb_size.tmpl is no longer needed (or included).
A MySQL user account having the necessary privileges. If you do not wish to use an existing account, then creating one using GRANT USAGE ON db_name.* — where db_name is the name of the database to be examined — is sufficient for this purpose.

ndb_size.pl can also be found in the MySQL sources in storage/ndb/tools. If this file is not present in your MySQL installation, you can obtain it from the MySQL Forge project page.

Usage:

ndb_size.pl [--database=db_name|ALL] [--hostname=host[:port]] [--socket=socket] [--user=user] \
    [--password=password] [--help|-h] [--format=(html|text)] [--loadqueries=file_name] [--savequeries=file_name]

By default, this utility attempts to analyze all databases on the server. You can specify a single database using the --database option; the default behavior can be made explicit by using ALL for the name of the database. You can also exclude one or more databases by using the --excludedbs with a comma-separated list of the names of the databases to be skipped. Similarly, you can cause specific tables to be skipped by listing their names, separated by commas, following the optional --excludetables option. A hostname (and possibly a port as well) can be specified using --hostname; the default is localhost:3306. If necessary, you can specify a socket; the default is /var/lib/mysql.sock. A MySQL username and password can be specified the corresponding options shown. It also possible to control the format of the output using the --format option; this can take either of the values html or text, with text being the default. An example of the text output is shown here:

shell> ndb_size.pl --database=test --socket=/tmp/mysql.sock
ndb_size.pl report for database: 'test' (1 tables)
--------------------------------------------------
Connected to: DBI:mysql:host=localhost;mysql_socket=/tmp/mysql.sock

Including information for versions: 4.1, 5.0, 5.1

test.t1
-------

DataMemory for Columns (* means varsized DataMemory):
              Column Name                 Type  Varsized   Key       4.1        5.0        5.1
          HIDDEN_NDB_PKEY               bigint             PRI         8          8          8
                       c2          varchar(50)         Y              52         52         4*
                       c1              int(11)                         4          4          4
                                                                      --         --         --
Fixed Size Columns DM/Row                                             64         64         12
   Varsize Columns DM/Row                                              0          0          4

DataMemory for Indexes:
               Index Name                 Type        4.1        5.0        5.1
                  PRIMARY                BTREE         16         16         16
                                                       --         --         --
       Total Index DM/Row                              16         16         16

IndexMemory for Indexes:
               Index Name        4.1        5.0        5.1
                  PRIMARY         33         16         16
                                  --         --         --
           Indexes IM/Row         33         16         16

Summary (for THIS table):
                                 4.1        5.0        5.1
    Fixed Overhead DM/Row         12         12         16
           NULL Bytes/Row          4          4          4
           DataMemory/Row         96         96         48  (Includes overhead, bitmap and indexes)

  Varsize Overhead DM/Row          0          0          8
   Varsize NULL Bytes/Row          0          0          4
       Avg Varside DM/Row          0          0         16

                 No. Rows          0          0          0

        Rows/32kb DM Page        340        340        680
Fixedsize DataMemory (KB)          0          0          0

Rows/32kb Varsize DM Page          0          0       2040
  Varsize DataMemory (KB)          0          0          0

         Rows/8kb IM Page        248        512        512
         IndexMemory (KB)          0          0          0

Parameter Minimum Requirements
------------------------------
* indicates greater than default

                Parameter          Default             4.1              5.0              5.1
          DataMemory (KB)            81920               0                0                0
       NoOfOrderedIndexes              128               1                1                1
               NoOfTables              128               1                1                1
         IndexMemory (KB)            18432               0                0                0
    NoOfUniqueHashIndexes               64               0                0                0
           NoOfAttributes             1000               3                3                3
             NoOfTriggers              768               5                5                5

For debugging purposes, the Perl arrays containing the queries run by this script can be read from the file specified using can be saved to a file using --savequeries; a file containing such arrays to be read in during script execution can be specified using --loadqueries. Neither of these options has a default value.

To produce output in HTML format, use the --format option and redirect the output to a file, as shown in this example:

shell> ndb_size.pl --database=test --socket=/tmp/mysql.sock --format=html > ndb_size.html

(Without the redirection, the output is sent to stdout.) This figure shows a portion of the generated ndb_size.html output file, as viewed in a Web browser:

Partial sample output from
ndb_size.pl as viewed in a Web
browser.

The output from this script includes:

Minimum values for the DataMemory, IndexMemory, MaxNoOfTables, MaxNoOfAttributes, MaxNoOfOrderedIndexes, MaxNoOfUniqueHashIndexes, and MaxNoOfTriggers configuration parameters required to accommodate the tables analyzed.
Memory requirements for all of the tables, attributes, ordered indexes, and unique hash indexes defined in the database.
The IndexMemory and DataMemory required per table and table row.

Note

Prior to MySQL 5.1.23, MySQL Cluster NDB 6.2.5, and MySQL Cluster NDB 6.3.7, ndb_size.pl was invoked as shown here:

perl ndb_size.pl db_name hostname username password > file_name.html

For more information about this change, see Bug#28683 and Bug#28253.

20.10.16. ndb_waiter — Wait for Cluster to Reach a Given Status

ndb_waiter repeatedly (each 100 milliseconds) prints out the status of all cluster data nodes until either the cluster reaches a given status or the --timeout limit is exceeded, then exits. By default, it waits for the cluster to achieve STARTED status, in which all nodes have started and connected to the cluster. This can be overridden using the --no-contact and --not-started options (see Additional Options).

The node states reported by this utility are as follows:

NO_CONTACT: The node cannot be contacted.
UNKNOWN: The node can be contacted, but its status is not yet known. Usually, this means that the node has received a START or RESTART command from the management server, but has not yet acted on it.
NOT_STARTED: The node has stopped, but remains in contact with the cluster. This is seen when restarting the node using the management client's RESTART command.
STARTING: The node's ndbd process has started, but the node has not yet joined the cluster.
STARTED: The node is operational, and has joined the cluster.
SHUTTING_DOWN: The node is shutting down.
SINGLE USER MODE: This is shown for all cluster data nodes when the cluster is in single user mode.

Usage:

ndb_waiter [-c connect_string]

Additional Options:

--no-contact, -n
Instead of waiting for the STARTED state, ndb_waiter continues running until the cluster reaches NO_CONTACT status before exiting.
--not-started
Instead of waiting for the STARTED state, ndb_waiter continues running until the cluster reaches NOT_STARTED status before exiting.
--timeout=seconds, -t seconds
Time to wait. The program exits if the desired state is not achieved within this number of seconds. The default is 120 seconds (1200 reporting cycles).

Sample Output. Shown here is the output from ndb_waiter when run against a 4-node cluster in which two nodes have been shut down and then started again manually. Duplicate reports (indicated by “...”) are omitted.

shell> ./ndb_waiter -c localhost

Connecting to mgmsrv at (localhost)
State node 1 STARTED
State node 2 NO_CONTACT
State node 3 STARTED
State node 4 NO_CONTACT
Waiting for cluster enter state STARTED

...

State node 1 STARTED
State node 2 UNKNOWN
State node 3 STARTED
State node 4 NO_CONTACT
Waiting for cluster enter state STARTED

...

State node 1 STARTED
State node 2 STARTING
State node 3 STARTED
State node 4 NO_CONTACT
Waiting for cluster enter state STARTED

...

State node 1 STARTED
State node 2 STARTING
State node 3 STARTED
State node 4 UNKNOWN
Waiting for cluster enter state STARTED

...

State node 1 STARTED
State node 2 STARTING
State node 3 STARTED
State node 4 STARTING
Waiting for cluster enter state STARTED

...

State node 1 STARTED
State node 2 STARTED
State node 3 STARTED
State node 4 STARTING
Waiting for cluster enter state STARTED

...

State node 1 STARTED
State node 2 STARTED
State node 3 STARTED
State node 4 STARTED
Waiting for cluster enter state STARTED

NDBT_ProgramExit: 0 - OK

Note

If no connectstring is specified, then ndb_waiter tries to connect to a management on localhost, and reports Connecting to mgmsrv at (null).

20.11. MySQL Cluster Replication

20.11.1. Abbreviations and Symbols
20.11.2. Assumptions and General Requirements
20.11.3. Known Issues in MySQL Cluster Replication
20.11.4. Cluster Replication Schema and Tables
20.11.5. Preparing the Cluster for Replication
20.11.6. Starting Replication (Single Replication Channel)
20.11.7. Using Two Replication Channels
20.11.8. Implementing Failover with MySQL Cluster
20.11.9. MySQL Cluster Backups With Replication
20.11.10. MySQL Cluster Replication Conflict Resolution

Previous to MySQL 5.1.6, asynchronous replication, more usually referred to simply as “replication”, was not available when using MySQL Cluster. MySQL 5.1.6 introduces master-slave replication of this type for MySQL Cluster databases. This section explains how to set up and manage a configuration wherein one group of computers operating as a MySQL Cluster replicates to a second computer or group of computers. We assume some familiarity on the part of the reader with standard MySQL replication as discussed elsewhere in this Manual. (See Chapter 19, Replication).

Normal (non-clustered) replication involves a “master” server and a “slave” server, the master being the source of the operations and data to be replicated and the slave being the recipient of these. In MySQL Cluster, replication is conceptually very similar but can be more complex in practice, as it may be extended to cover a number of different configurations including replicating between two complete clusters. Although a MySQL Cluster itself depends on the NDBCLUSTER storage engine for clustering functionality, it is not necessary to use the Cluster storage engine on the slave. However, for maximum availability, it is possible to replicate from one MySQL Cluster to another, and it is this type of configuration that we discuss, as shown in the following figure:

MySQL Cluster-to-Cluster Replication
Layout

In this scenario, the replication process is one in which successive states of a master cluster are logged and saved to a slave cluster. This process is accomplished by a special thread known as the NDB binlog injector thread, which runs on each MySQL server and produces a binary log (binlog). This thread ensures that all changes in the cluster producing the binary log — and not just those changes that are effected via the MySQL Server — are inserted into the binary log with the correct serialization order. We refer to the MySQL replication master and replication slave servers as replication servers or replication nodes, and the data flow or line of communication between them as a replication channel.

20.11.1. Abbreviations and Symbols

Throughout this section, we use the following abbreviations or symbols for referring to the master and slave clusters, and to processes and commands run on the clusters or cluster nodes:

Symbol or Abbreviation	Description (Refers to...)
`M`	The cluster serving as the (primary) replication master
`S`	The cluster acting as the (primary) replication slave
`shellM>`	Shell command to be issued on the master cluster
`mysqlM>`	MySQL client command issued on a single MySQL server running as an SQL node on the master cluster
`mysqlM*>`	MySQL client command to be issued on all SQL nodes participating in the replication master cluster
`shellS>`	Shell command to be issued on the slave cluster
`mysqlS>`	MySQL client command issued on a single MySQL server running as an SQL node on the slave cluster
`mysqlS*>`	MySQL client command to be issued on all SQL nodes participating in the replication slave cluster
`C`	Primary replication channel
`C'`	Secondary replication channel
`M'`	Secondary replication master
`S'`	Secondary replication slave

20.11.2. Assumptions and General Requirements

A replication channel requires two MySQL servers acting as replication servers (one each for the master and slave). For example, this means that in the case of a replication setup with two replication channels (to provide an extra channel for redundancy), there will be a total of four replication nodes, two per cluster.

Replication of a MySQL Cluster as described in this section and those following is dependent on row-based replication. This means that the replication master MySQL server must be started with --binlog-format=ROW or --binlog-format=MIXED, as described in Section 20.11.6, “Starting Replication (Single Replication Channel)”. For general information about row-based replication, see Section 19.1.2, “Replication Formats”.

(It is possible to replicate a MySQL Cluster using statement-based replication. However, in this case, the following restrictions apply: All updates to data rows on the cluster acting as the master must be directed to a single MySQL server; It is not possible to replicate a cluster using several MySQL replication processes at the same time; Only changes made at the SQL level are replicated.)

Each MySQL server used for replication in either cluster must be uniquely identified among all the MySQL replication servers participating in either cluster (you cannot have replication servers on both the master and slave clusters sharing the same ID). This can be done by starting each SQL node using the --server-id=id option, where id is a unique integer. Although it is not strictly necessary, we will assume for purposes of this discussion that all MySQL installations are the same version.

In any event, both MySQL servers involved in replication must be compatible with one another with respect to both the version of the replication protocol used and the SQL feature sets which they support; the simplest and easiest way to assure that this is the case is to use the same MySQL version for all servers involved. Note that in many cases it is not possible to replicate to a slave running a version of MySQL with a lower version number than that of the master — see Section 19.3.2, “Replication Compatibility Between MySQL Versions”, for details.

We assume that the slave server or cluster is dedicated to replication of the master, and that no other data is being stored on it.

20.11.3. Known Issues in MySQL Cluster Replication

The following are known problems or issues when using replication with MySQL Cluster in MySQL 5.1:

Loss of master-slave connection. Prior to MySQL 5.1.18, a MySQL Cluster replication slave mysqld had no way of detecting that the connection from the master had been interrupted (due to, for instance, the master going down or a network failure). For this reason, it was possible for the slave to become inconsistent with the master.
Beginning with MySQL 5.1.18, the master issues a “gap” event when connecting to the cluster. When the slave encounters a gap in the replication log, it stops with an error message. This message is available in the output of SHOW SLAVE STATUS, and indicates that the SQL thread has stopped due to an incident registered in the replication stream, and that manual intervention is required. In order to restart the slave, it is necessary to issue the following commands:
```
SET GLOBAL SQL_SLAVE_SKIP_COUNTER = 1;
START SLAVE; 
```
The slave then resumes reading the master binlog from the point where the gap was recorded.
Important
If high availability is a requirement for the slave server or cluster, then it is still advisable to set up multiple replication lines, to monitor the master mysqld on the primary replication line, and to fail over to a secondary line if and as necessary. For information about implementing this type of setup, see Section 20.11.7, “Using Two Replication Channels”, and Section 20.11.8, “Implementing Failover with MySQL Cluster”.
However, if you are replicating from a standalone MySQL server to a MySQL Cluster, one channel is usually sufficient.
Multi-byte character sets. There are several known issues with regard to the use of multi-byte characters sets with MySQL Cluster Replication. See Bug#27404 (fixed in MySQL 5.1.21), Bug#29562, Bug#29563, and Bug#29564 for more information.
Circular replication. Prior to MySQL 5.1.18, circular replication was not supported with MySQL Cluster replication, due to the fact that all log events created in a particular MySQL Cluster were wrongly tagged with the server ID of the MySQL server used as master and not with the server ID of the originating server.
Beginning with MySQL 5.1.18, this limitation is lifted, as discussed in the next few paragraphs, in which we consider the example of a replication setup involving three MySQL Clusters numbered 1, 2, and 3, in which Cluster 1 acts as the replication master for Cluster 2, Cluster 2 acts as the master for Cluster 3, and Cluster 3 acts as the master for Cluster 1. Each cluster has two SQL nodes, with SQL nodes A and B belonging to Cluster 1, SQL nodes C and D belonging to Cluster 2, and SQL nodes E and F belonging to Cluster 3.
Circular replication using these clusters is supported as long as:
- the SQL nodes on all masters and slaves are the same
- All SQL nodes acting as replication masters and slaves are started using the --log-slave-updates option
This type of circular replication setup is shown in the following diagram:
In this scenario, SQL node A in Cluster 1 replicates to SQL node C in Cluster 2; SQL node C replicates to SQL node E in Cluster 3; SQL node E replicates to SQL node A. In other words, the replication line (indicated by the red arrows in the diagram) directly connects all SQL nodes used as replication masters and slaves.
It should also be possible to set up circular replication in which not all master SQL nodes are also slaves, as shown here:
In this case, different SQL nodes in each cluster are used as replication masters and slaves. However, you must not start any of the SQL nodes using --log-slave-updates (see the description of this option for more information). This type of circular replication scheme for MySQL Cluster, in which the line of replication (again indicated by the red arrows in the diagram) is discontinuous, should be possible, but it should be noted that it has not yet been thoroughly tested and must therefore still be considered experimental.
Important
Beginning with MySQL 5.1.24, you should execute the following statement before starting circular replication:
```
mysql> SET GLOBAL SLAVE_EXEC_MODE = 'IDEMPOTENT';
```
This is necessary to suppress duplicate-key and other errors that otherwise break circular replication of MySQL Cluster. IDEMPOTENT mode is also required for multi-master replication when using MySQL Cluster. (Bug#31609)
See Slave_exec_mode, for more information.
DDL statements. The use of data definition statements, such as CREATE TABLE, DROP TABLE, and ALTER TABLE, are recorded in the binary log for only the MySQL server on which they are issued.
Cluster replication and primary keys. In MySQL 5.1.6, only those NDB tables having explicit primary keys could be replicated. This limitation was lifted in MySQL 5.1.7. However, in the event of a node failure, errors in replication of NDB tables without primary keys can still occur, due to the possibility of duplicate rows being inserted in such cases. For this reason, it is highly recommended that all NDB tables being replicated have primary keys.
Restarting with --initial. Restarting the cluster with the --initial option causes the sequence of GCI and epoch numbers to start over from 0. (This is generally true of MySQL Cluster and not limited to replication scenarios involving Cluster.) The MySQL servers involved in replication should in this case be restarted. After this, you should use the RESET MASTER and RESET SLAVE statements to clear the invalid ndb_binlog_index and ndb_apply_status tables. respectively.
auto_increment_offset and auto_increment_increment variables. The use of the auto_increment_offset and auto_increment_increment server system variables is supported beginning with MySQL 5.1.20. Previously, these produced unpredictable results when used with NDB tables or MySQL Cluster replication.
Replication from NDBCLUSTER to other storage engines. If you attempt to replicate from a MySQL Cluster to a slave that uses a storage engine that does not handle its own binary logging, the replication process aborts with the error Binary logging not possible ... Statement cannot be written atomically since more than one engine involved and at least one engine is self-logging (Error 1595). It is possible to work around this issue in one of the following ways:
- Turn off binary logging on the slave. This can be accomplished by setting SQL_LOG_BIN = 0.
- Change the storage engine used for the mysql.ndb_apply_status table. Causing this table to use an engine that does not handle its own binary logging can also eliminate the conflict. This can be done by issuing a statement such as ALTER TABLE mysql.ndb_apply_status ENGINE=MyISAM on the slave. It is safe to do this when using a non-NDB storage engine on the slave, since you do not then need to worry about keeping multiple slave SQL nodes synchronized.
- Filter out changes to the mysql.ndb_apply_status table on the slave. This can be done by starting the slave SQL node with the option --replicate-ignore-table=mysql.ndb_apply_status. If you need for other tables to be ignored by replication, you might wish to use an appropriate --replicate-wild-ignore-table option instead.
Important
You should not disable replication or binary logging of mysql.ndb_apply_status or change the storage engine used for this table when replicating from one MySQL Cluster to another. See Replication and binary log filtering rules with replication between MySQL Clusters elsewhere in this section for details.
Replication and binary log filtering rules with replication between MySQL Clusters. If you are using any of the options --replicate-do-*, --replicate-ignore-*, --binlog-do-db, or --binlog-ignore-db to filter databases or tables being replicated, care must be taken not to block replication or binary logging of the mysql.ndb_apply_status, which is required for replication between MySQL Clusters to operate properly. In particular, you must keep in mind the following:
1. Using --replicate-do-db=db_name (and no other --replicate-do-* or --replicate-ignore-* options) means that only tables in database db_name are replicated. In this case, you should also use --replicate-do-db=mysql, --binlog-do-db=mysql, or --replicate-do-table=mysql.ndb_apply_status to insure that mysql.ndb_apply_status is populated on slaves.
  Using --binlog-do-db=db_name (and no other --binlog-do-db options) means that changes only to tables in database db_name are written to the binary log. In this case, you should also use --replicate-do-db=mysql, --binlog-do-db=mysql, or --replicate-do-table=mysql.ndb_apply_status to insure that mysql.ndb_apply_status is populated on slaves.
2. Using --replicate-ignore-db=mysql means that no tables in the mysql database are replicated. In this case, you should also use --replicate-do-table=mysql.ndb_apply_status to insure that mysql.ndb_apply_status is replicated.
  Using --binlog-ignore-db=mysql means that no changes to tables in the mysql database are written to the binary log. In this case, you should also use --replicate-do-table=mysql.ndb_apply_status to insure that mysql.ndb_apply_status is replicated.
You should also remember that:
1. Each replication filtering rule requires its own --replicate-do-* or --replicate-ignore-* option, and that multiple rules cannot be expressed in a single replication filtering option. For information about these rules, see Section 19.1.3, “Replication Options and Variables”.
2. Each binary log filtering rule requires its own --binlog-do-db or --binlog-ignore-db option, and that multiple rules cannot be expressed in a single binary log filtering option. For information about these rules, see Section 5.2.4, “The Binary Log”.
Note
If you are replicating a MySQL Cluster to a slave that uses a storage engine other than NDBCLUSTER, the considerations just given previously may not apply. See Replication from NDBCLUSTER to other storage engines elsewhere in this this section for details.

20.11.4. Cluster Replication Schema and Tables

Replication in MySQL Cluster makes use of a number of dedicated tables in the mysql database on each MySQL Server instance acting as an SQL node in both the cluster being replicated and the replication slave (whether the slave is a single server or a cluster). These tables are created during the MySQL installation process by the mysql_install_db script, and include a table for storing the binary log's indexing data. Since the ndb_binlog_index table is local to each MySQL server and does not participate in clustering, it uses the MyISAM storage engine. This means that it must be created separately on each mysqld participating in the master cluster. (However, the binlog itself contains updates from all MySQL servers in the cluster to be replicated.) This table is defined as follows:

        
CREATE TABLE `ndb_binlog_index` (
    `Position`  BIGINT(20) UNSIGNED NOT NULL,
    `File`      VARCHAR(255) NOT NULL,
    `epoch`     BIGINT(20) UNSIGNED NOT NULL,
    `inserts`   BIGINT(20) UNSIGNED NOT NULL,
    `updates`   BIGINT(20) UNSIGNED NOT NULL,
    `deletes`   BIGINT(20) UNSIGNED NOT NULL,
    `schemaops` BIGNINT(20) UNSIGNED NOT NULL,
    PRIMARY KEY (`epoch`)
) ENGINE=MYISAM  DEFAULT CHARSET=latin1;

Important

Prior to MySQL 5.1.14, the ndb_binlog_index table was known as binlog_index, and was kept in a separate cluster database, which in MySQL 5.1.7 and earlier was known as the cluster_replication database. Similarly, the ndb_apply_status and ndb_schema tables were known as apply_status and schema, and were also found in the cluster (earlier cluster_replication) database. However, beginning with MySQL 5.1.14, all MySQL Cluster replication tables reside in the mysql system database.

Information about how this change affects upgrades from MySQL Cluster 5.1.13 and earlier to 5.1.14 and later versions can be found in Section C.1.15, “Changes in MySQL 5.1.14 (05 December 2006)”.

Beginning with MySQL Cluster NDB 6.3.2, this table has been changed to facilitate 3-way replication recovery. Two columns orig_server_id and orig_epoch have been added to this table; when mysqld is started with the --ndb-log-orig option, these columns store, respectively, the ID of the server on which the event originated and the epoch in which the event took place on the originating server. In addition, the table's primary key now includes these two columns. The modified table definition is shown here:

CREATE TABLE `ndb_binlog_index` (
    `Position` BIGINT(20) UNSIGNED NOT NULL,
    `File` VARCHAR(255) NOT NULL,
    `epoch` BIGINT(20) UNSIGNED NOT NULL,
    `inserts` INT(10) UNSIGNED NOT NULL,
    `updates` INT(10) UNSIGNED NOT NULL,
    `deletes` INT(10) UNSIGNED NOT NULL,
    `schemaops` INT(10) UNSIGNED NOT NULL,
    `orig_server_id` INT(10) UNSIGNED NOT NULL,
    `orig_epoch` BIGINT(20) UNSIGNED NOT NULL,
    `gci` INT(10) UNSIGNED NOT NULL
    PRIMARY KEY (`epoch`,`orig_server_id`,`orig_epoch`)
) ENGINE=MyISAM DEFAULT CHARSET=latin1;

The gci column was added in MySQL Cluster NDB 6.2.6 and MySQL Cluster NDB 6.3.2.

The following figure shows the relationship of the MySQL Cluster replication master server, its binlog injector thread, and the mysql.ndb_binlog_index table.

The replication master cluster, the
binlog-injector thread, and the
ndb_binlog_index table

An additional table, named ndb_apply_status, is used to keep a record of the operations that have been replicated from the master to the slave. Unlike the case with ndb_binlog_index, the data in this table is not specific to any one SQL node in the (slave) cluster, and so ndb_apply_status can use the NDB Cluster storage engine, as shown here:

CREATE TABLE `ndb_apply_status` (
    `server_id`   INT(10) UNSIGNED NOT NULL,
    `epoch`       BIGINT(20) UNSIGNED NOT NULL,
    `log_name`    VARCHAR(255) CHARACTER SET latin1 COLLATE latin1_bin NOT NULL,
    `start_pos`   BIGINT(20) UNSIGNED NOT NULL,
    `end_pos`     BIGINT(20) UNSIGNED NOT NULL,
    PRIMARY KEY (`server_id`) USING HASH
) ENGINE=NDBCLUSTER   DEFAULT CHARSET=latin1;

This table is populated only on slaves; on the master, no DataMemory is allocated to it. However, the table is populated from the master. For this reason, this table must be replicated and any replication filtering or binary log filtering rules that prevent this prevent replication between clusters from operating properly. For more information about potential problems arising from such filtering rules, see Section 20.11.3, “Known Issues in MySQL Cluster Replication”.

The log_name, start_pos, and end_pos columns were added in MySQL 5.1.18.

Important

If you are using MySQL Cluster replication, see Section 20.5.2, “MySQL Cluster 5.1 and MySQL Cluster NDB 6.x Upgrade and Downgrade Compatibility” before upgrading to MySQL 5.1.18 or later from an earlier version.

The ndb_binlog_index and ndb_apply_status tables are created in the mysql database because they should not be replicated. No user intervention is normally required to create or maintain either of them. Both the ndb_binlog_index and the ndb_apply_status tables are maintained by the NDB injector thread. This keeps the master mysqld process updated to changes performed by the NDB storage engine. The NDB binlog injector thread receives events directly from the NDB storage engine. The NDB injector is responsible for capturing all the data events within the cluster, and ensures that all events which change, insert, or delete data are recorded in the ndb_binlog_index table. The slave I/O thread transfers the events from the master's binary log to the slave's relay log.

However, it is advisable to check for the existence and integrity of these tables as an initial step in preparing a MySQL Cluster for replication. It is possible to view event data recorded in the binary log by querying the mysql.ndb_binlog_index table directly on the master. This can be also be accomplished using the SHOW BINLOG EVENTS statement on either the replication master or slave MySQL servers. (See Section 12.6.1.4, “SHOW BINLOG EVENTS Syntax”.)

You can also obtain useful information from the output of SHOW ENGINE NDB STATUS.

The ndb_schema table is used to track schema changes made to NDB tables. It is defined as shown here:

CREATE TABLE ndb_schema ( 
    `db` VARBINARY(63) NOT NULL,
    `name` VARBINARY(63) NOT NULL,
    `slock` BINARY(32) NOT NULL,
    `query` BLOB NOT NULL,
    `node_id` INT UNSIGNED NOT NULL,
    `epoch` BIGINT UNSIGNED NOT NULL,
    `id` INT UNSIGNED NOT NULL,
    `version` INT UNSIGNED NOT NULL,
    `type` INT UNSIGNED NOT NULL,
    PRIMARY KEY USING HASH (db,name) 
) ENGINE=NDB   DEFAULT CHARSET=latin1;

Unlike the two tables previously mentioned in this section, the ndb_schema table is not visible either to MySQL SHOW statements, or in any INFORMATION_SCHEMA tables; however, it can be seen in the output of ndb_show_tables, as shown here:

shell> ndb_show_tables -t 2
id    type                 state    logging database     schema   name
4     UserTable            Online   Yes     mysql        def      ndb_apply_status
5     UserTable            Online   Yes     ndbworld     def      City
6     UserTable            Online   Yes     ndbworld     def      Country
3     UserTable            Online   Yes     mysql        def      NDB$BLOB_2_3
7     UserTable            Online   Yes     ndbworld     def      CountryLanguage
2     UserTable            Online   Yes     mysql        def      ndb_schema

NDBT_ProgramExit: 0 - OK

It is also possible to SELECT from this table in mysql and other MySQL client applications, as shown here:

mysql> SELECT * FROM mysql.ndb_schema WHERE name='City' \G
*************************** 1. row ***************************
     db: ndbworld
   name: City
  slock:
  query: alter table City engine=ndb
node_id: 4
  epoch: 0
     id: 0
version: 0
   type: 7
1 row in set (0.00 sec)

This can sometimes be useful when debugging applications.

Note

When performing schema changes on NDB tables, applications should wait until the ALTER TABLE statement has returned in the MySQL client connection that issued the statement before attempting to use the updated definition of the table.

The ndb_schema table was added in MySQL 5.1.8.

Beginning with MySQL 5.1.14, if either of the ndb_apply_status or ndb_schema tables does not exist on the slave, it is created by ndb_restore. (Bug#14612)

Conflict resolution for MySQL Cluster Replication requires the presence of an additional mysql.ndb_replication table. Currently, this table must be created manually. For details, see Section 20.11.10, “MySQL Cluster Replication Conflict Resolution”.

20.11.5. Preparing the Cluster for Replication

Preparing the MySQL Cluster for replication consists of the following steps:

Check all MySQL servers for version compatibility (see Section 20.11.2, “Assumptions and General Requirements”).
Create a slave account on the master Cluster with the appropriate privileges:
```
mysqlM> GRANT REPLICATION SLAVE
     -> ON *.* TO 'slave_user'@'slave_host'
     -> IDENTIFIED BY 'slave_password';
```
In the previous statement, slave_user is the slave account username, slave_host is the hostname or IP address of the replication slave, and slave_password is the password to assign to this account.
For example, to create a slave user account with the name “myslave,” logging in from the host named “rep-slave,” and using the password “53cr37,” use the following GRANT statement:
```
mysqlM> GRANT REPLICATION SLAVE
     -> ON *.* TO 'myslave'@'rep-slave'
     -> IDENTIFIED BY '53cr37';
```
For security reasons, it is preferable to use a unique user account — not employed for any other purpose — for the replication slave account.
Configure the slave to use the master. Using the MySQL Monitor, this can be accomplished with the CHANGE MASTER TO statement:
```
mysqlS> CHANGE MASTER TO
     -> MASTER_HOST='master_host',
     -> MASTER_PORT=master_port,
     -> MASTER_USER='slave_user',
     -> MASTER_PASSWORD='slave_password';
```
In the previous statement, master_host is the hostname or IP address of the replication master, master_port is the port for the slave to use for connecting to the master, slave_user is the username set up for the slave on the master, and slave_password is the password set for that user account in the previous step.
For example, to tell the slave to replicate from the MySQL server whose hostname is “rep-master,” using the replication slave account created in the previous step, use the following statement:
```
mysqlS> CHANGE MASTER TO
     -> MASTER_HOST='rep-master'
     -> MASTER_PORT=3306,
     -> MASTER_USER='myslave'
     -> MASTER_PASSWORD='53cr37';
```
For a complete list of clauses that can be used with this statement, see Section 12.6.2.1, “CHANGE MASTER TO Syntax”.
You can also configure the slave to use the master by setting the corresponding startup options in the slave server's my.cnf file. To configure the slave in the same way as the preceding example CHANGE MASTER TO statement, the following information would need to be included in the slave's my.cnf file:
```
[mysqld]
master-host=rep-master
master-port=3306
master-user=myslave
master-password=53cr37
```
For additional options that can be set in my.cnf for replication slaves, see Section 19.1.3, “Replication Options and Variables”.
Note
To provide replication backup capability, you will also need to add an ndb-connectstring option to the slave's my.cnf file prior to starting the replication process. See Section 20.11.9, “MySQL Cluster Backups With Replication”, for details.
If the master cluster is already in use, you can create a backup of the master and load this onto the slave to cut down on the amount of time required for the slave to synchronize itself with the master. If the slave is also running MySQL Cluster, this can be accomplished using the backup and restore procedure described in Section 20.11.9, “MySQL Cluster Backups With Replication”.
```
ndb-connectstring=management_host[:port]
```
In the event that you are not using MySQL Cluster on the replication slave, you can create a backup with this command on the replication master:
```
shellM> mysqldump --master-data=1
```
Then import the resulting data dump onto the slave by copying the dump file over to the slave. After this, you can use the mysql client to import the data from the dumpfile into the slave database as shown here, where dump_file is the name of the file that was generated using mysqldump on the master, and db_name is the name of the database to be replicated:
```
shellS> mysql -u root -p db_name < dump_file
```
For a complete list of options to use with mysqldump, see Section 4.5.4, “mysqldump — A Database Backup Program”.
Note
If you copy the data to the slave in this fashion, you should make sure that the slave is started with the --skip-slave-start option on the command line, or else include skip-slave-start in the slave's my.cnf file to keep it from trying to connect to the master to begin replicating before all the data has been loaded. Once the data loading has completed, follow the additional steps outlined in the next two sections.
Ensure that each MySQL server acting as a replication master is configured with a unique server ID, and with binary logging enabled, using the row format. (See Section 19.1.2, “Replication Formats”.) These options can be set either in the master server's my.cnf file, or on the command line when starting the master mysqld process. See Section 20.11.6, “Starting Replication (Single Replication Channel)”, for information regarding the latter option.

20.11.6. Starting Replication (Single Replication Channel)

This section outlines the procedure for starting MySQL CLuster replication using a single replication channel.

Start the MySQL replication master server by issuing this command:
```
shellM> mysqld --ndbcluster --server-id=id \ 
        --log-bin --binlog-format=ROW &
```
In the previous statement, id is this server's unique ID (see Section 20.11.2, “Assumptions and General Requirements”). This starts the server's mysqld process with binary logging enabled using the proper logging format.
Note
You can also start the master with --binlog-format=MIXED, in which case row-based replication is used automatically when replicating between clusters.
Start the MySQL replication slave server as shown here:
```
shellS> mysqld --ndbcluster --server-id=id &
```
In the previous statement, id is the slave server's unique ID. It is not necessary to enable logging on the replication slave.
Note
You should use the --skip-slave-start option with this command or else you should include skip-slave-start in the slave server's my.cnf file, unless you want replication to begin immediately. With the use of this option, the start of replication is delayed until the appropriate START SLAVE statement has been issued, as explained in Step 4 below.
It is necessary to synchronize the slave server with the master server's replication binlog. If binary logging has not previously been running on the master, run the following statement on the slave:
```
mysqlS> CHANGE MASTER TO
     -> MASTER_LOG_FILE='',
     -> MASTER_LOG_POS=4;
```
This instructs the slave to begin reading the master's binary log from the log's starting point. Otherwise — that is, if you are loading data from the master using a backup — see Section 20.11.8, “Implementing Failover with MySQL Cluster”, for information on how to obtain the correct values to use for MASTER_LOG_FILE and MASTER_LOG_POS in such cases.
Finally, you must instruct the slave to begin applying replication by issuing this command from the mysql client on the replication slave:
```
mysqlS> START SLAVE;
```
This also initiates the transmission of replication data from the master to the slave.

It is also possible to use two replication channels, in a manner simlar to the procedure described in the next section; the differences between this and using a single replication channel are covered in Section 20.11.7, “Using Two Replication Channels”.

Beginning with MySQL Cluster NDB 6.2.3, it is possible to improve cluster replication performance by enabling batched updates. This can be accomplished by starting slave mysqld processes with the --slave-allow-batching option. Normally, updates are applied as soon as they are received. However, the use of batching causes updates to be applied in 32 KB batches, which can result in higher throughput and less CPU usage, particularly where individual updates are relatively small.

Note

Slave batching works on a per-epoch basis; updates belonging to more than one transaction can be sent as part of the same batch.

All outstanding updates are applied when the end of an epoch is reached, even if the updates total less than 32 KB.

Batching can be turned on and off at runtime. To activate it at runtime, you can use either of these two statements:

SET GLOBAL slave_allow_batching = 1;
SET GLOBAL slave_allow_batching = ON;

If a particular batch causes problems (such as a statement whose effects do not appear to be replicated correctly), slave batching can be deactivated using either of the following statements:

SET GLOBAL slave_allow_batching = 0;
SET GLOBAL slave_allow_batching = OFF;

You can check whether slave batching is currently being used by means of an appropriate SHOW VARIABLES statement, like this one:

mysql> SHOW VARIABLES LIKE 'slave%';
+---------------------------+-------+
| Variable_name             | Value |
+---------------------------+-------+
| slave_allow_batching      | ON    |
| slave_compressed_protocol | OFF   |
| slave_load_tmpdir         | /tmp  |
| slave_net_timeout         | 3600  |
| slave_skip_errors         | OFF   |
| slave_transaction_retries | 10    |
+---------------------------+-------+
6 rows in set (0.00 sec)

20.11.7. Using Two Replication Channels

In a more complete example scenario, we envision two replication channels to provide redundancy and thereby guard against possible failure of a single replication channel. This requires a total of four replication servers, two masters for the master cluster and two slave servers for the slave cluster. For purposes of the discussion that follows, we assume that unique identifiers are assigned as shown here:

Server ID	Description
1	Master - primary replication channel (M)
2	Master - secondary replication channel (M')
3	Slave - primary replication channel (S)
4	Slave - secondary replication channel (S')

Setting up replication with two channels is not radically different from setting up a single replication channel. First, the mysqld processes for the primary and secondary replication masters must be started, followed by those for the primary and secondary slaves. Then the replication processes may be initiated by issuing the START SLAVE statement on each of the slaves. The commands and the order in which they need to be issued are shown here:

Start the primary replication master:

shellM> mysqld --ndbcluster --server-id=1 \ 
               --log-bin --binlog-format=row &

Start the secondary replication master:

shellM'> mysqld --ndbcluster --server-id=2 \
               --log-bin --binlog-format=row &

Start the primary replication slave server:

shellS> mysqld --ndbcluster --server-id=3 \
               --skip-slave-start &

Start the secondary replication slave:

shellS'> mysqld --ndbcluster --server-id=4 \
                --skip-slave-start &

Finally, initiate replication on the primary channel by executing the START SLAVE statement on the primary slave as shown here:
```
mysqlS> START SLAVE;
```
Warning
Only the primary channel is to be started at this point. The secondary replication channel is to be started only in the event that the primary replication channel fails, as described in Section 20.11.8, “Implementing Failover with MySQL Cluster”. Running multiple replication channels simultaneously can result in unwanted duplicate records being created on the replication slaves.

As mentioned previously, it is not necessary to enable binary logging on replication slaves.

20.11.8. Implementing Failover with MySQL Cluster

In the event that the primary Cluster replication process fails, it is possible to switch over to the secondary replication channel. The following procedure describes the steps required to accomplish this.

Obtain the time of the most recent global checkpoint (GCP). That is, you need to determine the most recent epoch from the ndb_apply_status table on the slave cluster, which can be found using the following query:
```
mysqlS'> SELECT @latest:=MAX(epoch)
      ->        FROM mysql.ndb_apply_status;
```
Using the information obtained from the query shown in Step 1, obtain the corresponding records from the ndb_binlog_index table on the master cluster as shown here:
```
mysqlM'> SELECT 
      ->     @file:=SUBSTRING_INDEX(File, '/', -1),
      ->     @pos:=Position
      -> FROM mysql.ndb_binlog_index
      -> WHERE epoch >= @latest
      -> ORDER BY epoch ASC LIMIT 1;
```
These are the records saved on the master since the failure of the primary replication channel. We have employed a user variable @latest here to represent the value obtained in Step 1. Of course, it is not possible for one mysqld instance to access user variables set on another server instance directly. These values must be “plugged in” to the second query manually or in application code.
Now it is possible to synchronize the secondary channel by running the following query on the secondary slave server:
```
mysqlS'> CHANGE MASTER TO
      ->     MASTER_LOG_FILE='@file',
      ->     MASTER_LOG_POS=@pos;
```
Again we have employed user variables (in this case @file and @pos) to represent the values obtained in Step 2 and applied in Step 3; in practice these values must be inserted manually or using application code that can access both of the servers involved.
Note
@file is a string value such as '/var/log/mysql/replication-master-bin.00001', and so must be quoted when used in SQL or application code. However, the value represented by @pos must not be quoted. Although MySQL normally attempts to convert strings to numbers, this case is an exception.
You can now initiate replication on the secondary channel by issuing the appropriate command on the secondary slave mysqld:
```
mysqlS'> START SLAVE;
```

Once the secondary replication channel is active, you can investigate the failure of the primary and effect repairs. The precise actions required to do this will depend upon the reasons for which the primary channel failed.

Warning

The secondary replication channel is to be started only if and when the primary replication channel has failed. Running multiple replication channels simultaneously can result in unwanted duplicate records being created on the replication slaves.

If the failure is limited to a single server, it should (in theory) be possible to replicate from M to S', or from M' to S; however, this has not yet been tested.

20.11.9. MySQL Cluster Backups With Replication

20.11.9.1. Automating Synchronization of the Slave to the Master binlog

This section discusses making backups and restoring from them using MySQL Cluster replication. We assume that the replication servers have already been configured as covered previously (see Section 20.11.5, “Preparing the Cluster for Replication”, and the sections immediately following). This having been done, the procedure for making a backup and then restoring from it is as follows:

There are two different methods by which the backup may be started.
- Method A. This method requires that the cluster backup process was previously enabled on the master server, prior to starting the replication process. This can be done by including the following line in a [mysql_cluster] section in the my.cnf file, where management_host is the IP address or hostname of the NDB management server for the master cluster, and port is the management server's port number:
```
ndb-connectstring=management_host[:port]
```
  Note
  The port number needs to be specified only if the default port (1186) is not being used. See Section 20.2.3, “Multi-Computer Configuration”, for more information about ports and port allocation in MySQL Cluster.
  In this case, the backup can be started by executing this statement on the replication master:
```
shellM> ndb_mgm -e "START BACKUP"
```
- Method B. If the my.cnf file does not specify where to find the management host, you can start the backup process by passing this information to the NDB management client as part of the START BACKUP command. This can be done as shown here, where management_host and port are the hostname and port number of the management server:
```
shellM> ndb_mgm management_host:port -e "START BACKUP"
```
  In our scenario as outlined earlier (see Section 20.11.5, “Preparing the Cluster for Replication”), this would be executed as follows:
```
shellM> ndb_mgm rep-master:1186 -e "START BACKUP"
```
Copy the cluster backup files to the slave that is being brought on line. Each system running an ndbd process for the master cluster will have cluster backup files located on it, and all of these files must be copied to the slave to ensure a successful restore. The backup files can be copied into any directory on the computer where the slave management host resides, so long as the MySQL and NDB binaries have read permissions in that directory. In this case, we will assume that these files have been copied into the directory /var/BACKUPS/BACKUP-1.
It is not necessary that the slave cluster have the same number of ndbd processes (data nodes) as the master; however, it is highly recommended this number be the same. It is necessary that the slave be started with the --skip-slave-start option, to prevent premature startup of the replication process.
Create any databases on the slave cluster that are present on the master cluster that are to be replicated to the slave.
Important
A CREATE DATABASE (or CREATE SCHEMA) statement corresponding to each database to be replicated must be executed on each SQL node in the slave cluster.
Reset the slave cluster using this statement in the MySQL Monitor:
```
mysqlS> RESET SLAVE;
```
It is important to make sure that the slave's apply_status table does not contain any records prior to running the restore process. You can accomplish this by running this SQL statement on the slave:
```
mysqlS> DELETE FROM mysql.ndb_apply_status;
```
You can now start the cluster restoration process on the replication slave using the ndb_restore command for each backup file in turn. For the first of these, it is necessary to include the -m option to restore the cluster metadata:
```
shellS> ndb_restore -c slave_host:port -n node-id \
        -b backup-id -m -r dir
```
dir is the path to the directory where the backup files have been placed on the replication slave. For the ndb_restore commands corresponding to the remaining backup files, the -m option should not be used.
For restoring from a master cluster with four data nodes (as shown in the figure in Section 20.11, “MySQL Cluster Replication”) where the backup files have been copied to the directory /var/BACKUPS/BACKUP-1, the proper sequence of commands to be executed on the slave might look like this:
```
shellS> ndb_restore -c rep-slave:1186 -n 2 -b 1 -m \
        -r ./var/BACKUPS/BACKUP-1
shellS> ndb_restore -c rep-slave:1186 -n 3 -b 1 \
        -r ./var/BACKUPS/BACKUP-1
shellS> ndb_restore -c rep-slave:1186 -n 4 -b 1 \
        -r ./var/BACKUPS/BACKUP-1
shellS> ndb_restore -c rep-slave:1186 -n 5 -b 1 -e \
        -r ./var/BACKUPS/BACKUP-1
```
Important
The -e (or --restore-epoch) option in the final invocation of ndb_restore in this example is required in order that the epoch is written to the slave mysql.ndb_apply_status. Without this information, the slave will not be able to synchronize properly with the master. (See Section 20.9.3, “ndb_restore — Restore a Cluster Backup”.)
Now you need to obtain the most recent epoch from the ndb_apply_status table on the slave (as discussed in Section 20.11.8, “Implementing Failover with MySQL Cluster”):
```
mysqlS> SELECT @latest:=MAX(epoch)
        FROM mysql.ndb_apply_status;
```
Using @latest as the epoch value obtained in the previous step, you can obtain the correct starting position @pos in the correct binary log file @file from the master's mysql.ndb_binlog_index table using the query shown here:
```
mysqlM> SELECT 
     ->     @file:=SUBSTRING_INDEX(File, '/', -1),
     ->     @pos:=Position
     -> FROM mysql.ndb_binlog_index
     -> WHERE epoch > @latest
     -> ORDER BY epoch ASC LIMIT 1;
```
In the event that there is currently no replication traffic, you can get this information by running SHOW MASTER STATUS on the master and using the value in the Position column for the file whose name has the suffix with the greatest value for all files shown in the File column. However, in this case, you must determine this and supply it in the next step manually or by parsing the output with a script.
Using the values obtained in the previous step, you can now issue the appropriate CHANGE MASTER TO statement in the slave's mysql client:
```
mysqlS> CHANGE MASTER TO
     ->     MASTER_LOG_FILE='@file',
     ->     MASTER_LOG_POS=@pos;
```
Now that the slave “knows” from what point in which binlog file to start reading data from the master, you can cause the slave to begin replicating with this standard MySQL statement:
```
mysqlS> START SLAVE;
```

To perform a backup and restore on a second replication channel, it is necessary only to repeat these steps, substituting the hostnames and IDs of the secondary master and slave for those of the primary master and slave replication servers where appropriate, and running the preceding statements on them.

For additional information on performing Cluster backups and restoring Cluster from backups, see Section 20.9, “On-line Backup of MySQL Cluster”.

20.11.9.1. Automating Synchronization of the Slave to the Master binlog

It is possible to automate much of the process described in the previous section (see Section 20.11.9, “MySQL Cluster Backups With Replication”). The following Perl script reset-slave.pl serves as an example of how you can do this.

#!/user/bin/perl -w

#  file: reset-slave.pl

#  Copyright ©2005 MySQL AB

#  This program is free software; you can redistribute it and/or modify
#  it under the terms of the GNU General Public License as published by
#  the Free Software Foundation; either version 2 of the License, or
#  (at your option) any later version.

#  This program is distributed in the hope that it will be useful,
#  but WITHOUT ANY WARRANTY; without even the implied warranty of
#  MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
#  GNU General Public License for more details.

#  You should have received a copy of the GNU General Public License
#  along with this program; if not, write to: 
#  Free Software Foundation, Inc. 
#  59 Temple Place, Suite 330 
#  Boston, MA 02111-1307 USA
#
#  Version 1.1


######################## Includes ###############################

use DBI;

######################## Globals ################################

my  $m_host='';
my  $m_port='';
my  $m_user='';
my  $m_pass='';
my  $s_host='';
my  $s_port='';
my  $s_user='';
my  $s_pass='';
my  $dbhM='';
my  $dbhS='';

####################### Sub Prototypes ##########################

sub CollectCommandPromptInfo;
sub ConnectToDatabases;
sub DisconnectFromDatabases;
sub GetSlaveEpoch;
sub GetMasterInfo;
sub UpdateSlave;

######################## Program Main ###########################

CollectCommandPromptInfo;
ConnectToDatabases;
GetSlaveEpoch;
GetMasterInfo;
UpdateSlave;
DisconnectFromDatabases;

################## Collect Command Prompt Info ##################

sub CollectCommandPromptInfo
{
  ### Check that user has supplied correct number of command line args
  die "Usage:\n
       reset-slave >master MySQL host< >master MySQL port< \n
                   >master user< >master pass< >slave MySQL host< \n
                   >slave MySQL port< >slave user< >slave pass< \n
       All 8 arguments must be passed. Use BLANK for NULL passwords\n"
       unless @ARGV == 8;

  $m_host  =  $ARGV[0];
  $m_port  =  $ARGV[1];
  $m_user  =  $ARGV[2];
  $m_pass  =  $ARGV[3];
  $s_host  =  $ARGV[4];
  $s_port  =  $ARGV[5];
  $s_user  =  $ARGV[6];
  $s_pass  =  $ARGV[7];

  if ($m_pass eq "BLANK") { $m_pass = '';}
  if ($s_pass eq "BLANK") { $s_pass = '';}
}

###############  Make connections to both databases #############

sub ConnectToDatabases
{
  ### Connect to both master and slave cluster databases

  ### Connect to master
  $dbhM
    = DBI->connect(
    "dbi:mysql:database=mysql;host=$m_host;port=$m_port",
    "$m_user", "$m_pass")
      or die "Can't connect to Master Cluster MySQL process!
              Error: $DBI::errstr\n";

  ### Connect to slave
  $dbhS 
    = DBI->connect(
          "dbi:mysql:database=mysql;host=$s_host",
          "$s_user", "$s_pass")
    or die "Can't connect to Slave Cluster MySQL process!
            Error: $DBI::errstr\n";
}

################  Disconnect from both databases ################

sub DisconnectFromDatabases
{
  ### Disconnect from master

  $dbhM->disconnect
  or warn " Disconnection failed: $DBI::errstr\n";

  ### Disconnect from slave

  $dbhS->disconnect
  or warn " Disconnection failed: $DBI::errstr\n";
}

######################  Find the last good GCI ##################

sub GetSlaveEpoch
{
  $sth = $dbhS->prepare("SELECT MAX(epoch)
                         FROM mysql.ndb_apply_status;")
      or die "Error while preparing to select epoch from slave: ", 
             $dbhS->errstr;

  $sth->execute
      or die "Selecting epoch from slave error: ", $sth->errstr;

  $sth->bind_col (1, \$epoch);
  $sth->fetch;
  print "\tSlave Epoch =  $epoch\n";
  $sth->finish;
}

#######  Find the position of the last GCI in the binlog ########

sub GetMasterInfo
{
  $sth = $dbhM->prepare("SELECT 
                           SUBSTRING_INDEX(File, '/', -1), Position
                         FROM mysql.ndb_binlog_index
                         WHERE epoch > $epoch 
                         ORDER BY epoch ASC LIMIT 1;")
      or die "Prepare to select from master error: ", $dbhM->errstr;

  $sth->execute
      or die "Selecting from master error: ", $sth->errstr;

  $sth->bind_col (1, \$binlog);
  $sth->bind_col (2, \$binpos);
  $sth->fetch;
  print "\tMaster bin log =  $binlog\n";
  print "\tMaster Bin Log position =  $binpos\n";
  $sth->finish;
}

##########  Set the slave to process from that location #########

sub UpdateSlave
{
  $sth = $dbhS->prepare("CHANGE MASTER TO
                         MASTER_LOG_FILE='$binlog',
                         MASTER_LOG_POS=$binpos;")
      or die "Prepare to CHANGE MASTER error: ", $dbhS->errstr;

  $sth->execute
       or die "CHNAGE MASTER on slave error: ", $sth->errstr;
  $sth->finish;
  print "\tSlave has been updated. You may now start the slave.\n";
}

# end reset-slave.pl

20.11.10. MySQL Cluster Replication Conflict Resolution

When using a replication setup involving multiple masters (including circular replication), it is possible that different masters may try to update the same row on the slave with different data. Conflict resolution in MySQL Cluster Replication provides a means of resolving such conflicts by allowing a user defined resolution column to be used to determine whether or not an update to the row on a given master should be applied on the slave. (This column is sometimes referred to as a “timestamp” column, even though this column' type cannot be TIMESTAMP, as explained later in this section.) Different methods can be used to compare resolution column values on the slave when conflicts occur, as explained later in this section; the method used can be set on a per-table basis.

Important

Conflict resolution as described in this section is always applied on a row-by-row basis rather than a transactional basis. In addition, it is the application's responsibility to ensure that the resolution column is correctly populated with relevant values, so that the resolution function can make the appropriate choice when determining whether to apply an update.

Requirements. Preparations for conflict resolution must be made on both the master and the slave:

On the master writing the binlogs, you must determine which columns are sent (all columns or only those that have been updated). This is done for the MySQL Server as a whole by applying the mysqld startup option -–ndb-log-updated-only (described later in this section) or on a per-table basis by entries in the mysql.ndb_replication table.
On the slave, you must determine which type of conflict resolution to apply (“latest timestamp wins”, “same timestamp wins”, or none). This is done using the mysql.ndb_replication system table, on a per-table basis.

If only some but not all columns are sent, then the master and slave can diverge.

Note

We refer to the column used for determining updates as a “timestamp” column, but the data type of this column is never TIMESTAMP; rather, its data type should be INT (INTEGER) or BIGINT. This column should be UNSIGNED and NOT NULL.

Master column control. We can see update operations in terms of “before” and “after” images — that is, the states of the table before and after the update is applied. Normally, when updating a table with a primary key, the “before” image is not of great interest; however, when we need to determine on a per-update basis whether or not to use the updated values on a replication slave, we need to make sure that both images are written to the master's binary log. This is done with the --ndb-log-update-as-write startup option for mysqld, as described later in this section.

Important

Whether logging of complete rows or of updated columns only is done is decided when the MySQL server is started, and cannot be changed online; you must either restart mysqld, or start a new mysqld instance with different logging options.

Logging full or partial rows (--ndb-log-updated-only option). For purposes of conflict resolution, there are two basic methods of logging rows, as determined by the setting of the --ndb-log-updated-only option for mysqld:

Log complete rows
Log only column data that has been updated — that is, column data whose value has been set, regardless of whether or not this value was actually changed.

It is more efficient to log updated columns only; however, if you need to log full rows, you can do so by setting --ndb-log-updated-only to 0 or OFF.

Logging changed data as updates (--ndb-log-update-as-write option). Either of these logging methods can be configured to be done with or without the “before” image as determined by the setting of another MySQL Server option --ndb-log-update-as-write. Because conflict resolution is done in the MySQL Server's update handler, it is necessary to control logging on the master such that updates are updates and not writes; that is, such that updates are treated as changes in existing rows rather than the writing of new rows (even though these replace existing rows). This option is turned on by default; to turn it off, start the server with --ndb-log-update-as-write=0 or --ndb-log-update-as-write=OFF.

Conflict resolution control. Conflict resolution is usually enabled on the server where conflicts can occur. Like logging method selection, it is enabled by entries in the mysql.ndb_replication table.

The ndb_replication system table. To enable conflict resolution, it is necessary to create an ndb_replication table in the mysql system database on the master, the slave, or both, depending on the conflict resolution type and method to be employed. This table is used to control logging and conflict resolution functions on a per-table basis, and has one row per table invoved in replication. ndb_replication is created and filled with control information on the server where the conflict is to be resolved. In a simple master-slave setup where data can also be changed locally on the slave this will typically be the slave. In a more complex master-master (2-way) replication schema this will usually be all of the masters involved. Each row in mysql.ndb_replication corresponds to a table being replicated, and specifies how to log and resolve conflicts (that is, which conflict resolution function, if any, to use) for that table. The definition of the mysql.ndb_replication table is shown here:

CREATE TABLE mysql.ndb_replication  (
    db VARBINARY(63),
    table_name VARBINARY(63),
    server_id INT UNSIGNED,
    binlog_type INT UNSIGNED,
    conflict_fn VARBINARY(128),
    PRIMARY KEY USING HASH (db, table_name, server_id)
)   ENGINE=NDB 
PARTITION BY KEY(db,table_name);

The columns in this table are described in the following list:

db. The name of the database containing the table to be replicated.
table_name. The name of the table to be replicated.
server_id. The unique server ID of the MySQL instance (SQL node) where the table resides.

binlog_type. The type of binary logging to be employed. This is determined as shown in the following table:

Value	Internal Value	Description
0	`NBT_DEFAULT`	Use server default
1	`NBT_NO_LOGGING`	Do not log this table in the binary log
2	`NBT_UPDATED_ONLY`	Only updated attributes are logged
3	`NBT_FULL`	Log full row, even if not updated (MySQL server default behavior)
4	`NBT_USE_UPDATE`	(For generating `NBT_UPDATED_ONLY_USE_UPDATE` and `NBT_FULL_USE_UPDATE` values only — not intended for separate use)
5	[Not used]	---
6	`NBT_UPDATED_ONLY_USE_UPDATE` (equal to `NBT_UPDATED_ONLY \| NBT_USE_UPDATE`)	Use updated attributes, even if values are unchanged
7	`NBT_FULL_USE_UPDATE` (equal to `NBT_FULL \| NBT_USE_UPDATE`)	Use full row, even if values are unchanged

conflict_fn. The conflict resolution function to be applied. This function must be specified as one of the following:
- NDB$MAX(column_name). Indicates that “greatest timestamp wins” conflict resolution is to be used — that is, if the “timestamp” for a given row coming from the master is higher than that on the slave, it is applied; otherwise it is not applied on the slave. This ensures that, in the event of a conflict, the version of the row that was most recently updated is the version that persists.
  This conflict resolution function is available beginning with MySQL Cluster NDB 6.3.0.
- NDB$OLD(column_name). Indicates that an update is applied only if the value of column_name is the same on both the master and the slave. This ensures that updates are not applied from the wrong master.
  This conflict resolution function is available beginning with MySQL Cluster NDB 6.3.4.
- NULL: Indicates that conflict resolution is not to be used for the corresponding table
.

Status information. Beginning with MySQL Cluster NDB 6.3.3, a server status variable Ndb_conflict_fn_max provides a count of the number of times that a row was not applied on the current SQL node due to “greatest timestamp wins” conflict resolution since the last time that mysqld was started.

Beginning with MySQL Cluster NDB 6.3.4, the number of times that a row was not applied as the result of “same timestamp wins” conflict resolution on a given mysqld since the last time it was restarted is given by the global status variable Ndb_conflict_fn_old. In addition to incrementing Ndb_conflict_fn_old, the primary key of the row that was not used is inserted into an exceptions table, as explained later in this section.

Additional requirements for “Same timestamp wins” conflict resolution. To use the NDB$OLD() conflict resolution function, it is also necessary to create an exceptions table corresponding to each NDB table for which this type of conflict resolution is to be employed. The name of this table is that of the table for which “same timestamp wins” conflict resolution is to be applied, with the string $EX appended. (For example, if the name of the original table is mytable, the name of the corresponding exception table name should be mytable$EX.) This table is created as follows:

CREATE TABLE original_table$EX  (
    server_id INT UNSIGNED,
    master_server_id INT UNSIGNED, 
    master_epoch BIGINT UNSIGNED,
    count INT UNSIGNED,
    original_table_pk_columns,
    [additional_columns,]
    PRIMARY KEY(server_id, master_server_id, master_epoch, count)
) ENGINE=NDB;

The first four columns are required. Following these columns, the columns making up the original table's primary key should be copied in the order in which they are used to define the primary key of the original table.

Note

The names of the first four columns and the columns matching the original table's primary key columns are not critical; however, we suggest for reasons of clarity and consistency, that you use the names shown here for the server_id, master_server_id, master_epoch, and count columns, and that you use the same names as in the original table for the columns matching those in the original table's primary key.

The data types for the columns duplicating the primary key columns of the original table should be the same as for (or larger than) the original columns.

Additional columns may optionally be defined following these columns, but not before any of them; any such extra columns cannot be NOT NULL. The exception table's primary key must be defined as shown. The exception table must use the NDB storage engine. An example of use for NDB$OLD() and an exception table is given later in this section.

Important

The mysql.ndb_replication table is read when a data table is set up for replication, so the row corresponding to a table to be replicated must be inserted into mysql.ndb_replication before the table to be replicated is created.

Examples. The following examples assume that you have already a working MySQL Cluster replication setup, as described in Section 20.11.5, “Preparing the Cluster for Replication”, and Section 20.11.6, “Starting Replication (Single Replication Channel)”.

NDB$MAX() example. Suppose you wish to enable “greatest timestamp wins” conflict resolution on table test.t1, using column mycol as the “timestamp”. This can be done using the following steps:
1. Make sure that you have started the master mysqld with -–ndb-log-update-as-write=OFF.
2. On the master, perform this INSERT statement:
```
INSERT INTO mysql.ndb_replication 
    VALUES ('test', 't1', 0, NULL, 'NDB$MAX(mycol)');
```
  Inserting a 0 into the server_id indicates that all SQL nodes accessing this table should use conflict resolution. If you want to use conflict resolution on a specific mysqld only, use the actual server ID.
  Inserting NULL into the binlog_type column has the same effect as inserting 0 (NBT_DEFAULT); the server default is used.
3. Create the test.t1 table:
```
CREATE TABLE test.t1 (
    columns
    mycol INT UNSIGNED,
    columns
) ENGINE=NDB;
```
  Now, when updates are done on this table, conflict resolution will be applied, and the version of the row having the greatest value for mycol will be written to the slave.
Note
Other binlog_type options — such as NBT_UPDATED_ONLY_USE_UPDATE should be used in order to control logging on the master via the ndb_replication table rather than by using command-line options.
NDB$OLD() example. Suppose an NDB table such as the one defined here is being replicated, and you wish to enable “same timestamp wins” conflict resolution for updates to this table:
```
CREATE TABLE test.t2  (
    a INT UNSIGNED NOT NULL,
    b CHAR(25) NOT NULL,
    columns,
    mycol INT UNSIGNED NOT NULL,
    columns,
    PRIMARY KEY pk (a, b)
)   ENGINE=NDB;
```
The following steps are required, in the order shown:
1. First — and prior to creating test.t2 — you must insert a row into the mysql.ndb_replication table, as shown here:
```
INSERT INTO mysql.ndb_replication 
    VALUES ('test', 't2', 0, NULL, 'NDB$OLD(mycol)');
```
  Possible values for the binlog_type column are shown earlier in this section. The value 'NDB$OLD(mycol)' should be inserted into the conflict_fn column.
2. Create an appropriate exceptions table for test.t2. The table creation statement shown here includes all required columns; any additional columns must be declared following these columns, and before the definition of the table's primary key.
```
CREATE TABLE test.t2$EX  (
    server_id SMALLINT UNSIGNED,
    master_server_id SMALLINT UNSIGNED, 
    master_epoch BIGINT UNSIGNED,
    count BIGINT UNSIGNED,
    a INT UNSIGNED NOT NULL,
    b CHAR(25) NOT NULL,
    [additional_columns,]
    PRIMARY KEY(server_id, master_server_id, master_epoch, count)
)   ENGINE=NDB;
```
3. Create the table test.t2 as shown previously.
These steps must be followed for every table for which you wish to perform conflict resolution using NDB$OLD(). For each such table, there must be a corresponding row in mysql.ndb_replication, and there must be an exceptions table in the same database as the table being replicated.

20.12. MySQL Cluster Disk Data Tables

20.12.1. Disk Data Objects
20.12.2. Disk Data Storage Requirements
20.12.3. Disk Data Configuration Parameters

Beginning with MySQL 5.1.6, it is possible to store the non-indexed columns of NDB tables on disk, rather than in RAM as with previous versions of MySQL Cluster.

As part of implementing Cluster Disk Data work, a number of improvements were made in MySQL Cluster for the efficient handling of very large amounts (terabytes) of data during node recovery and restart. These include a “no-steal” algorithm for synchronising a starting node with very large data sets. For more information, see the paper Recovery Principles of MySQL Cluster 5.1, by MySQL Cluster developers Mikael Ronström and Jonas Oreland.

20.12.1. Disk Data Objects

This section discusses Disk Data objects — which include tables, log file groups, and tablespaces — as well as how to create and drop them.

Assuming that you have already set up a MySQL Cluster with all nodes (including management and SQL nodes) running MySQL 5.1.6 or newer, the basic steps for creating a Cluster table on disk are as follows:

Create a log file group, and assign one or more undo log files to it (an undo log file is also referred as an undofile).
Note
In MySQL 5.1, undo log files are necessary only for Disk Data tables. They are no longer used for tables that are stored in memory.
Create a tablespace, and assign the log file group to it, as well as one or more data files.
Create a Disk Data table that uses this tablespace for data storage.

Each of these tasks can be accomplished using SQL statements, as shown in the following example.

We create a log file group named lg_1 using CREATE LOGFILE GROUP. This log file group is to be made up of two undo log files, which we name undo_1.dat and undo_2.dat, whose initial sizes are 16 MB and 12 MB, respectively. (The default initial size for an undo log file is 128 MB.) Optionally, you can also specify a size for the log file group's UNDO buffer, or allow it to assume the default value of 8 MB. In this example, we set the UNDO buffer's size at 2 MB. A log file group must be created with an undo log file; so we add undo_1.dat to lg_1 in this CREATE LOGFILE GROUP statement:
```
CREATE LOGFILE GROUP lg_1
    ADD UNDOFILE 'undo_1.dat'
    INITIAL_SIZE 16M
    UNDO_BUFFER_SIZE 2M
    ENGINE NDB;
```
To add undo_2.dat to the log file group, use the following ALTER LOGFILE GROUP statement:
```
ALTER LOGFILE GROUP lg_1
    ADD UNDOFILE 'undo_2.dat'
    INITIAL_SIZE 12M
    ENGINE NDB;
```
Some items of note:
- The .dat file extension used here is not required. We use it merely to make the log and data files easily recognisable.
- Every CREATE LOGFILE GROUP and ALTER LOGFILE GROUP statement must include an ENGINE clause. In MySQL 5.1, the permitted values for this clause are NDB and NDBCLUSTER.
  Important
  In MySQL 5.1.8 and later, there can exist only one log file group at any given time.
- When you add an undo log file to a log file group using ADD UNDOFILE 'filename', a file with the name filename is created in the ndb_nodeid_fs directory within the DataDirectory of each data node in the cluster, where nodeid is the node ID of the data node.
- UNDO_BUFFER_SIZE is limited by the amount of system memory available.
- For more information about the CREATE LOGFILE GROUP statement, see Section 12.1.8, “CREATE LOGFILE GROUP Syntax”. For more information about ALTER LOGFILE GROUP, see Section 12.1.2, “ALTER LOGFILE GROUP Syntax”.
Now we can create a tablespace, which contains files to be used by MySQL Cluster Disk Data tables for storing their data. A tablespace is also associated with a particular log file group. When creating a new tablespace, you must specify the log file group which it is to use for undo logging; you must also specify a data file. You can add more data files to the tablespace after the tablespace is created; it is also possible to drop data files from a tablespace (an example of dropping data files is provided later in this section).
Assume that we wish to create a tablespace named ts_1 which uses lg_1 as its log file group. This tablespace is to contain two data files named data_1.dat and data_2.dat, whose initial sizes are 32 MB and 48 MB, respectively. (The default value for INITIAL_SIZE is 128 MB.) We can do this using two SQL statements, as shown here:
```
CREATE TABLESPACE ts_1
    ADD DATAFILE 'data_1.dat'
    USE LOGFILE GROUP lg_1
    INITIAL_SIZE 32M
    ENGINE NDB;

ALTER TABLESPACE ts_1
    ADD DATAFILE 'data_2.dat'
    INITIAL_SIZE 48M
    ENGINE NDB;
```
The CREATE TABLESPACE statement creates a tablespace ts_1 with the data file data_1.dat, and associates ts_1 with log file group lg_1. The ALTER TABLESPACE adds the second data file (data_2.dat).
Some items of note:
- As is the case with the filenames used here for undo log files, there is no special significance for the .dat file extension; it is used merely for easy recognition.
- All CREATE TABLESPACE and ALTER TABLESPACE statements must contain an ENGINE clause; only tables using the same storage engine as the tablespace can be created in the tablespace. In MySQL 5.1, the only permitted values for this clause are NDB and NDBCLUSTER.
  For more information about the CREATE TABLESPACE and ALTER TABLESPACE statements, see Section 12.1.11, “CREATE TABLESPACE Syntax”, and Section 12.1.5, “ALTER TABLESPACE Syntax”.
Now it is possible to create a table whose non-indexed columns are stored on disk in the tablespace ts_1:
```
CREATE TABLE dt_1 (
    member_id INT UNSIGNED NOT NULL AUTO_INCREMENT PRIMARY KEY,
    last_name VARCHAR(50) NOT NULL,
    first_name VARCHAR(50) NOT NULL,
    dob DATE NOT NULL,
    joined DATE NOT NULL,
    INDEX(last_name, first_name)
    )
    TABLESPACE ts_1 STORAGE DISK
    ENGINE NDB;
```
The TABLESPACE ... STORAGE DISK clause tells the NDBCLUSTER storage engine to use tablespace ts_1 for disk data storage.
Note
Beginning with MySQL Cluster NDB 6.2.5 and MySQL Cluster NDB 6.3.2, it is also possible to specify whether an individual column is stored on disk or in memory by using a STORAGE clause as part of the column's definition in a CREATE TABLE or ALTER TABLE statement. STORAGE DISK causes the column to be stored on disk, and STORAGE MEMORY causes in-memory storage to be used. See Section 12.1.10, “CREATE TABLE Syntax”, for more information.
Once table ts_1 has been created as shown, you can perform INSERT, SELECT, UPDATE, and DELETE statements on it just as you would with any other MySQL table.
For table dt_1 as it has been defined here, only the dob and joined columns are stored on disk. This is because there are indexes on the id, last_name, and first_name columns, and so data belonging to these columns is stored in RAM. In MySQL 5.1, only non-indexed columns can be held on disk; indexes and indexed column data continue to be stored in memory. This tradeoff between the use of indexes and conservation of RAM is something you must keep in mind as you design Disk Data tables.

Performance note. The performance of a cluster using Disk Data storage is greatly improved if Disk Data files are kept on a separate physical disk from the data node filesystem. This must be done for each data node in the cluster to derive any noticeable benefit.

You may use absolute and relative filesystem paths with ADD UNDOFILE and ADD DATAFILE. Relative paths are calculated relative to the data node's data directory.

A log file group, a tablespace, and any Disk Data tables using these must be created in a particular order. The same is true for dropping any of these objects:

A log file group cannot be dropped, so long as any tablespaces are using it.
A tablespace cannot be dropped as long as it contains any data files.
You cannot drop any data files from a tablespace as long as there remain any tables which are using the tablespace.
Beginning with MySQL 5.1.12, it is no longer possible to drop files created in association with a different tablespace than the one with which the files were created. (Bug#20053)

For example, to drop all the objects created so far in this section, you would use the following statements:

mysql> DROP TABLE dt_1;

mysql> ALTER TABLESPACE ts_1
    -> DROP DATAFILE 'data_2.dat'
    -> ENGINE NDB;

mysql> ALTER TABLESPACE ts_1
    -> DROP DATAFILE 'data_1.dat'
    -> ENGINE NDB;

mysql> DROP TABLESPACE ts_1
    -> ENGINE NDB;

mysql> DROP LOGFILE GROUP lg_1
    -> ENGINE NDB;

These statements must be performed in the order shown, except that the two ALTER TABLESPACE ... DROP DATAFILE statements may be executed in either order.

You can obtain information about data files used by Disk Data tables by querying the FILES table in the INFORMATION_SCHEMA database. An extra “NULL row” was added to this table in MySQL 5.1.14 for providing additional information about undo log files. For more information and examples of use, see Section 27.21, “The INFORMATION_SCHEMA FILES Table”.

20.12.2. Disk Data Storage Requirements

The following items apply to Disk Data storage requirements:

Variable-length columns of Disk Data tables take up a fixed amount of space. For each row, this is equal to the space required to store the largest possible value for that column.
For general information about calculating these values, see Section 10.5, “Data Type Storage Requirements”.
In a Disk Data table, the first 256 bytes of a TEXT or BLOB column are stored in memory; only the remainder is stored on disk.

Important

Starting the cluster with the --initial option does not remove Disk Data files. You must remove these manually prior to performing an initial restart of the cluster.

20.12.3. Disk Data Configuration Parameters

Configuration parameters affecting Disk Data behaviour include the following:

DiskPageBufferMemory
This determines the amount of space used for caching pages on disk, and is set in the [ndbd] or [ndbd default] section of the config.ini file. It is measured in bytes. Each page takes up 32 KB. This means that Cluster Disk Data storage always uses N * 32 KB memory where N is some non-negative integer.
SharedGlobalMemory
This determines the amount of memory that is used for log buffers, disk operations (such as page requests and wait queues), and metadata for tablespaces, log file groups, UNDO files, and data files. It can be set in the [ndbd] or [ndbd default] section of the config.ini configuration file, and is measured in bytes.
The default value is 20M.

These parameters were added in MySQL 5.1.6

Note

The OPTIMIZE TABLE statement does not have any effect on Disk Data tables.

20.13. Using High-Speed Interconnects with MySQL Cluster

20.13.1. Configuring MySQL Cluster to use SCI Sockets
20.13.2. MySQL Cluster Interconnects and Performance

Even before design of NDBCLUSTER began in 1996, it was evident that one of the major problems to be encountered in building parallel databases would be communication between the nodes in the network. For this reason, NDBCLUSTER was designed from the very beginning to allow for the use of a number of different data transport mechanisms. In this Manual, we use the term transporter for these.

The MySQL Cluster codebase includes support for four different transporters:

TCP/IP using 100 Mbps or gigabit Ethernet, as discussed in Section 20.3.4.7, “Cluster TCP/IP Connections”.
Direct (machine-to-machine) TCP/IP; although this transporter uses the same TCP/IP protocol as mentioned in the previous item, it requires setting up the hardware differently and is configured differently as well. For this reason, it is considered a separate transport mechanism for MySQL Cluster. See Section 20.3.4.8, “TCP/IP Connections Using Direct Connections”, for details.
Shared memory (SHM). For more information about SHM, see Section 20.3.4.9, “Shared-Memory Connections”.
Scalable Coherent Interface (SCI), as described in the next section of this chapter, Section 20.3.4.10, “SCI Transport Connections”.

Most users today employ TCP/IP over Ethernet because it is ubiquitous. TCP/IP is also by far the best-tested transporter for use with MySQL Cluster.

We are working to make sure that communication with the ndbd process is made in “chunks” that are as large as possible because this benefits all types of data transmission.

For users who desire it, it is also possible to use cluster interconnects to enhance performance even further. There are two ways to achieve this: Either a custom transporter can be designed to handle this case, or you can use socket implementations that bypass the TCP/IP stack to one extent or another. We have experimented with both of these techniques using the SCI (Scalable Coherent Interface) technology developed by Dolphin.

20.13.1. Configuring MySQL Cluster to use SCI Sockets

In this section, we show how to adapt a cluster configured for normal TCP/IP communication to use SCI Sockets instead. This documentation is based on SCI Sockets version 2.3.0 as of 01 October 2004.

Prerequisites. Any machines with which you wish to use SCI Sockets must be equipped with SCI cards.

No special builds (other than the -max builds) are needed for SCI Sockets because it uses normal TCP/IP socket calls which are already available in MySQL Cluster. However, SCI Sockets are currently supported only on the Linux 2.4 and 2.6 kernels. For other operating systems, you can use SCI Transporters, but this requires that the server be built using --with-ndb-sci=/opt/DIS.

Prior to MySQL 5.1.20, there were issues with building MySQL Cluster with SCI support (see Bug#25470), but these have been resolved due to work contributed by Dolphin International. SCI Sockets are now correctly supported for MySQL Cluster using the -max builds, and versions of MySQL Cluster with SCI Transporter support can be built using either of compile-amd64-max-sci or compile-pentium64-max-sci. Both of these build scripts can be found in the BUILD directory of the MySQL 5.1 source; it should not be difficult to adapt them for other platforms.

There are essentially four requirements for SCI Sockets:

Building the SCI Socket libraries.
Installation of the SCI Socket kernel libraries.
Installation of one or two configuration files.
The SCI Socket kernel library must be enabled either for the entire machine or for the shell where the MySQL Cluster processes are started.

This process needs to be repeated for each machine in the cluster where you plan to use SCI Sockets for inter-node communication.

Two packages need to be retrieved to get SCI Sockets working:

The source code package containing the DIS support libraries for the SCI Sockets libraries.
The source code package for the SCI Socket libraries themselves.

Currently, these are available only in source code format. The latest versions of these packages at the time of this writing were available as (respectively) DIS_GPL_2_5_0_SEP_10_2004.tar.gz and SCI_SOCKET_2_3_0_OKT_01_2004.tar.gz. You should be able to find these (or possibly newer versions) at http://www.dolphinics.com/support/downloads.html.

Package Installation. Once you have obtained the library packages, the next step is to unpack them into appropriate directories, with the SCI Sockets library unpacked into a directory below the DIS code. Next, you need to build the libraries. This example shows the commands used on Linux/x86 to perform this task:

shell> tar xzf DIS_GPL_2_5_0_SEP_10_2004.tar.gz
shell> cd DIS_GPL_2_5_0_SEP_10_2004/src/
shell> tar xzf ../../SCI_SOCKET_2_3_0_OKT_01_2004.tar.gz
shell> cd ../adm/bin/Linux_pkgs
shell> ./make_PSB_66_release

It is possible to build these libraries for some 64-bit processors. To build the libraries for Opteron CPUs using the 64-bit extensions, run make_PSB_66_X86_64_release rather than make_PSB_66_release. If the build is made on an Itanium machine, you should use make_PSB_66_IA64_release. The X86-64 variant should work for Intel EM64T architectures but this has not yet (to our knowledge) been tested.

Once the build process is complete, the compiled libraries will be found in a zipped tar file with a name along the lines of DIS-<operating-system>-time-date. It is now time to install the package in the proper place. In this example we will place the installation in /opt/DIS. You most likely need to run the following as the system root user.

shell> cp DIS_Linux_2.4.20-8_181004.tar.gz /opt/
shell> cd /opt
shell> tar xzf DIS_Linux_2.4.20-8_181004.tar.gz
shell> mv DIS_Linux_2.4.20-8_181004 DIS

Network configuration. Now that all the libraries and binaries are in their proper place, we need to ensure that the SCI cards have proper node IDs within the SCI address space.

It is also necessary to decide on the network structure before proceeding. There are three types of network structures which can be used in this context:

A simple one-dimensional ring
One or more SCI switches with one ring per switch port
A two- or three-dimensional torus.

Each of these topologies has its own method for providing node IDs. We discuss each of them in brief.

A simple ring uses node IDs which are non-zero multiples of 4: 4, 8, 12,...

The next possibility uses SCI switches. An SCI switch has 8 ports, each of which can support a ring. It is necessary to make sure that different rings use different node ID spaces. In a typical configuration, the first port uses node IDs below 64 (4 – 60), the next 64 node IDs (68 – 124) are assigned to the next port, and so on, with node IDs 452 – 508 being assigned to the eighth port.

Two- and three-dimensional torus network structures take into account where each node is located in each dimension, incrementing by 4 for each node in the first dimension, by 64 in the second dimension, and (where applicable) by 1024 in the third dimension. See Dolphin's Web site for more thorough documentation.

In our testing we have used switches, although most large cluster installations use 2- or 3-dimensional torus structures. The advantage provided by switches is that, with dual SCI cards and dual switches, it is possible to build with relative ease a redundant network where the average failover time on the SCI network is on the order of 100 microseconds. This is supported by the SCI transporter in MySQL Cluster and is also under development for the SCI Socket implementation.

Failover for the 2D/3D torus is also possible but requires sending out new routing indexes to all nodes. However, this requires only 100 milliseconds or so to complete and should be acceptable for most high-availability cases.

By placing cluster data nodes properly within the switched architecture, it is possible to use 2 switches to build a structure whereby 16 computers can be interconnected and no single failure can hinder more than one of them. With 32 computers and 2 switches it is possible to configure the cluster in such a manner that no single failure can cause the loss of more than two nodes; in this case, it is also possible to know which pair of nodes is affected. Thus, by placing the two nodes in separate node groups, it is possible to build a “safe” MySQL Cluster installation.

To set the node ID for an SCI card use the following command in the /opt/DIS/sbin directory. In this example, -c 1 refers to the number of the SCI card (this is always 1 if there is only 1 card in the machine); -a 0 refers to adapter 0; and 68 is the node ID:

shell> ./sciconfig -c 1 -a 0 -n 68

If you have multiple SCI cards in the same machine, you can determine which card has which slot by issuing the following command (again we assume that the current working directory is /opt/DIS/sbin):

shell> ./sciconfig -c 1 -gsn

This will give you the SCI card's serial number. Then repeat this procedure with -c 2, and so on, for each card in the machine. Once you have matched each card with a slot, you can set node IDs for all cards.

After the necessary libraries and binaries are installed, and the SCI node IDs are set, the next step is to set up the mapping from hostnames (or IP addresses) to SCI node IDs. This is done in the SCI sockets configuration file, which should be saved as /etc/sci/scisock.conf. In this file, each SCI node ID is mapped through the proper SCI card to the hostname or IP address that it is to communicate with. Here is a very simple example of such a configuration file:

#host           #nodeId
alpha           8
beta            12
192.168.10.20   16

It is also possible to limit the configuration so that it applies only to a subset of the available ports for these hosts. An additional configuration file /etc/sci/scisock_opt.conf can be used to accomplish this, as shown here:

#-key                        -type        -values
EnablePortsByDefault                yes
EnablePort                  tcp           2200
DisablePort                 tcp           2201
EnablePortRange             tcp           2202 2219
DisablePortRange            tcp           2220 2231

Driver installation. With the configuration files in place, the drivers can be installed.

First, the low-level drivers and then the SCI socket driver need to be installed:

shell> cd DIS/sbin/
shell> ./drv-install add PSB66
shell> ./scisocket-install add

If desired, the installation can be checked by invoking a script which verifies that all nodes in the SCI socket configuration files are accessible:

shell> cd /opt/DIS/sbin/
shell> ./status.sh

If you discover an error and need to change the SCI socket configuration, it is necessary to use ksocketconfig to accomplish this task:

shell> cd /opt/DIS/util
shell> ./ksocketconfig -f

For more information about ksocketconfig, consult the documentation available from http://www.dolphinics.com/support/documentation.html.

Testing the setup. To ensure that SCI sockets are actually being used, you can employ the latency_bench test program. Using this utility's server component, clients can connect to the server to test the latency of the connection. Determining whether SCI is enabled should be fairly simple from observing the latency.

Note

Before using latency_bench, it is necessary to set the LD_PRELOAD environment variable as shown later in this section.

To set up a server, use the following:

shell> cd /opt/DIS/bin/socket
shell> ./latency_bench -server

To run a client, use latency_bench again, except this time with the -client option:

shell> cd /opt/DIS/bin/socket
shell> ./latency_bench -client server_hostname

SCI socket configuration should now be complete and MySQL Cluster ready to use both SCI Sockets and the SCI transporter (see Section 20.3.4.10, “SCI Transport Connections”).

Starting the cluster. The next step in the process is to start MySQL Cluster. To enable usage of SCI Sockets it is necessary to set the environment variable LD_PRELOAD before starting ndbd, mysqld, and ndb_mgmd. This variable should point to the kernel library for SCI Sockets.

To start ndbd in a bash shell, do the following:

bash-shell> export LD_PRELOAD=/opt/DIS/lib/libkscisock.so
bash-shell> ndbd

In a tcsh environment the same thing can be accomplished with:

tcsh-shell> setenv LD_PRELOAD=/opt/DIS/lib/libkscisock.so
tcsh-shell> ndbd

Note

MySQL Cluster can use only the kernel variant of SCI Sockets.

20.13.2. MySQL Cluster Interconnects and Performance

The ndbd process has a number of simple constructs which are used to access the data in a MySQL Cluster. We have created a very simple benchmark to check the performance of each of these and the effects which various interconnects have on their performance.

There are four access methods:

Primary key access. This is access of a record through its primary key. In the simplest case, only one record is accessed at a time, which means that the full cost of setting up a number of TCP/IP messages and a number of costs for context switching are borne by this single request. In the case where multiple primary key accesses are sent in one batch, those accesses share the cost of setting up the necessary TCP/IP messages and context switches. If the TCP/IP messages are for different destinations, additional TCP/IP messages need to be set up.
Unique key access. Unique key accesses are similar to primary key accesses, except that a unique key access is executed as a read on an index table followed by a primary key access on the table. However, only one request is sent from the MySQL Server, and the read of the index table is handled by ndbd. Such requests also benefit from batching.
Full table scan. When no indexes exist for a lookup on a table, a full table scan is performed. This is sent as a single request to the ndbd process, which then divides the table scan into a set of parallel scans on all cluster ndbd processes. In future versions of MySQL Cluster, an SQL node will be able to filter some of these scans.
Range scan using ordered index
When an ordered index is used, it performs a scan in the same manner as the full table scan, except that it scans only those records which are in the range used by the query transmitted by the MySQL server (SQL node). All partitions are scanned in parallel when all bound index attributes include all attributes in the partitioning key.

With benchmarks developed internally by MySQL for testing simple and batched primary and unique key accesses, we have found that using SCI sockets improves performance by approximately 100% over TCP/IP, except in rare instances when communication performance is not an issue. This can occur when scan filters make up most of processing time or when very large batches of primary key accesses are achieved. In that case, the CPU processing in the ndbd processes becomes a fairly large part of the overhead.

Using the SCI transporter instead of SCI Sockets is only of interest in communicating between ndbd processes. Using the SCI transporter is also only of interest if a CPU can be dedicated to the ndbd process because the SCI transporter ensures that this process will never go to sleep. It is also important to ensure that the ndbd process priority is set in such a way that the process does not lose priority due to running for an extended period of time, as can be done by locking processes to CPUs in Linux 2.6. If such a configuration is possible, the ndbd process will benefit by 10–70% as compared with using SCI sockets. (The larger figures will be seen when performing updates and probably on parallel scan operations as well.)

There are several other optimized socket implementations for computer clusters, including Myrinet, Gigabit Ethernet, Infiniband and the VIA interface. However, we have tested MySQL Cluster so far only with SCI sockets. See Section 20.13.1, “Configuring MySQL Cluster to use SCI Sockets”, for information on how to set up SCI sockets using ordinary TCP/IP for MySQL Cluster.

20.14. Known Limitations of MySQL Cluster

20.14.1. Non-Compliance In SQL Syntax
20.14.2. Limits and Differences from Standard MySQL Limits
20.14.3. Limits Relating to Transaction Handling
20.14.4. Error Handling
20.14.5. Limits Associated with Database Objects
20.14.6. Unsupported Or Missing Features
20.14.7. Limitations Relating to Performance
20.14.8. Issues Exclusive to MySQL Cluster
20.14.9. Limitations Relating to Disk Data Storage
20.14.10. Limitations Relating to Multiple Cluster Nodes
20.14.11. Previous MySQL Cluster Issues Resolved in MySQL 5.1

In the sections that follow, we discuss known limitations in current releases pf MySQL Cluster as compared with the features available when using the MyISAM and InnoDB storage engines. If you check the “Cluster” category in the MySQL bugs database at http://bugs.mysql.com, you can find known bugs in the following categories under “MySQL Server:” in the MySQL bugs database at http://bugs.mysql.com, which we intend to correct in upcoming releases of MySQL Cluster:

Cluster
Cluster Direct API (NDBAPI)
Cluster Disk Data
Cluster Replication

This information is intended to be complete with respect to the conditions just set forth. You can report any discrepancies that you encounter to the MySQL bugs database using the instructions given in Section 1.7, “How to Report Bugs or Problems”. If we do not plan to fix the problem in MySQL 5.1, we will add it to the list.

See Section 20.14.11, “Previous MySQL Cluster Issues Resolved in MySQL 5.1” for a list of issues in MySQL Cluster in MySQL 5.0 that have been resolved in the current version.

Note

Limitations and other issues specific to MySQL Cluster Replication are described in Section 20.11.3, “Known Issues in MySQL Cluster Replication”.

20.14.1. Non-Compliance In SQL Syntax

Some SQL statements relating to certain MySQL features produce errors when used with NDB tables, as described in the following list:

Temporary tables. Temporary tables are not supported. Trying either to create a temporary table that uses the NDB storage engine or to alter an existing temporary table to use NDB fails with the error Table storage engine 'ndbcluster' does not support the create option 'TEMPORARY'.
Indexes and keys in NDB tables. Keys and indexes on MySQL Cluster tables are subject to the following limitations:
- TEXT and BLOB columns. You cannot create indexes on NDB table columns that use any of the TEXT or BLOB data types.
- FULLTEXT indexes. The NDB storage engine does not support FULLTEXT indexes, which are possible for MyISAM tables only.
  However, you can create indexes on VARCHAR columns of NDB tables.
- BIT columns. A BIT column cannot be a primary key, unique key, or index, nor can it be part of a composite primary key, unique key, or index.
- AUTO_INCREMENT columns. Like other MySQL storage engines, the NDB storage engine can handle a maximum of one AUTO_INCREMENT column per table. However, in the case of a Cluster table with no explicit primary key, an AUTO_INCREMENT column is automatically defined and used as a “hidden” primary key. For this reason, you cannot define a table that has an explicit AUTO_INCREMENT column unless that column is also declared using the PRIMARY KEY option. Attempting to create a table with an AUTO_INCREMENT column that is not the table's primary key, and using the NDB storage engine, fails with an error.
MySQL Cluster and geometry data types. Geometry datatypes (WKT and WKB) are supported in NDB tables in MySQL 5.1. However, spatial indexes are not supported.
Creating NDB tables with user-defined partitioning. Support for user-defined partitioning for MySQL Cluster in MySQL 5.1 is restricted to [LINEAR] KEY partitioning. Beginning with MySQL 5.1.12, using any other partitioning type with ENGINE=NDB or ENGINE=NDBCLUSTER in a CREATE TABLE statement results in an error.
Default partitioning scheme. As of MySQL 5.1.6, all Cluster tables are by default partitioned by KEY using the table's primary key as the partitioning key. If no primary key is explicitly set for the table, the “hidden” primary key automatically created by the NDB storage engine is used instead. For additional discussion of these and related issues, see Section 21.2.4, “KEY Partitioning”.
DROP PARTITION not supported. It is not possible to drop partitions from NDB tables using ALTER TABLE ... DROP PARTITION. The other partitioning extensions to ALTER TABLE — ADD PARTITION, REORGANIZE PARTITION, and COALESCE PARTITION — are supported for Cluster tables, but use copying and so are not optimised. See Section 21.3.1, “Management of RANGE and LIST Partitions” and Section 12.1.4, “ALTER TABLE Syntax”.
Row-based replication. When using row-based replication with MySQL Cluster, binary logging cannot be disabled. That is, the NDB storage engine ignores the value of SQL_LOG_BIN. (Bug#16680)

20.14.2. Limits and Differences from Standard MySQL Limits

In this section, we list limits found in MySQL Cluster that either differ from limits found in, or that are not found in, standard MySQL.

Memory usage and recovery. Memory consumed when data is inserted into an NDB table is not automatically recovered when deleted, as it is with other storage engines. Instead, the following rules hold true:
- A DELETE statement on an NDB table makes the memory formerly used by the deleted rows available for re-use by inserts on the same table only. This memory cannot be used by other NDB tables.
- A DROP TABLE or TRUNCATE operation on an NDB table frees the memory that was used by this table for re-use by any NDB table, either by the same table or by another NDB table.
  Note
  Recall that TRUNCATE drops and re-creates the table. See Section 12.2.9, “TRUNCATE Syntax”.
  Memory freed by DELETE operations but still allocated to a specific table can also be made available for general re-use by performing a rolling restart of the cluster. See Section 20.5.1, “Performing a Rolling Restart of the Cluster”.
  Beginning with MySQL Cluster NDB 6.3.7, this limitation can be overcome using OPTIMIZE TABLE. See Section 20.14.11, “Previous MySQL Cluster Issues Resolved in MySQL 5.1”, for more information.
- Limits imposed by the cluster's configuration. A number of hard limits exist which are configurable, but available main memory in the cluster sets limits. See the complete list of configuration parameters in Section 20.3.4, “Configuration File”. Most configuration parameters can be upgraded online. These hard limits include:
  - Database memory size and index memory size (DataMemory and IndexMemory, respectively).
    DataMemory is allocated as 32KB pages. As each DataMemory page is used, it is assigned to a specific table; once allocated, this memory cannot be freed except by dropping the table.
    See Section 20.3.4.5, “Defining Data Nodes”, for further information about DataMemory and IndexMemory.
  - The maximum number of operations that can be performed per transaction is set using the configuration parameters MaxNoOfConcurrentOperations and MaxNoOfLocalOperations.
    Note
    Bulk loading, TRUNCATE TABLE, and ALTER TABLE are handled as special cases by running multiple transactions, and so are not subject to this limitation.
  - Different limits related to tables and indexes. For example, the maximum number of ordered indexes per table is determined by MaxNoOfOrderedIndexes.
- Node and data object maximums. The following limits apply to numbers of cluster nodes and metadata objects:
  - The maximum number of data nodes is 48.
    A data node must have a node ID in the range of 1‐49, inclusive. (Management and API nodes may use any integer in the range of 1‐63 inclusive as a node ID.)
  - The total maximum number of nodes in a MySQL Cluster is 63. This number includes all SQL nodes (MySQL Servers), API nodes (applications accessing the cluster other than MySQL servers), data nodes, and management servers.
  - The maximum number of metadata objects in current versions of MySQL Cluster. This limit is hard-coded.
  See Section 20.14.11, “Previous MySQL Cluster Issues Resolved in MySQL 5.1”, for more information.

20.14.3. Limits Relating to Transaction Handling

A number of limitations exist in MySQL Cluster with regard to the handling of transactions. These include the following:

Transaction isolation level. The NDBCLUSTER storage engine supports only the READ COMMITTED transaction isolation level. (InnoDB, for example, supports READ COMMITTED, READ UNCOMMITTED, REPEATABLE READ, and SERIALIZABLE.) See Section 20.9.5, “Backup Troubleshooting”, for information on how this can affect backing up and restoring Cluster databases.)
Important
If a SELECT from a Cluster table includes a BLOB or TEXT column, the READ COMMITTED transaction isolation level is converted to a read with read lock. This is done to guarantee consistency, due to the fact that parts of the values stored in columns of these types are actually read from a separate table.
Rollbacks. There is no partial rollback of transactions. A duplicate key or similar error rolls back the entire transaction.
Transactions and memory usage. As noted elsewhere in this chapter, MySQL Cluster does not handle large transactions well; it is better to perform a number of small transactions with a few operations each than to attempt a single large transaction containing a great many operations. Among other considerations, large transactions require very large amounts of memory. Because of this, the transactional behaviour of a number of MySQL statements is effected as described in the following list:
- TRUNCATE is not transactional when used on NDB tables. If a TRUNCATE fails to empty the table, then it must be re-run until it is successful.
- DELETE FROM (even with no WHERE clause) is transactional. For tables containing a great many rows, you may find that performance is improved by using several DELETE FROM ... LIMIT ... statements to “chunk” the delete operation. If your objective is to empty the table, then you may wish to use TRUNCATE instead.
- LOAD DATA statements. LOAD DATA INFILE is not transactional when used on NDB tables.
  Important
  When executing a LOAD DATA INFILE statement, the NDB engine performs commits at irregular intervals that enable better utilization of the communication network. It is not possible to know ahead of time when such commits take place.
  LOAD DATA FROM MASTER is not supported in MySQL Cluster.
- ALTER TABLE and transactions. When copying an NDB table as part of an ALTER TABLE, the creation of the copy is non-transactional. (In any case, this operation is rolled back when the copy is deleted.)
Transactions and the COUNT() function. When using MySQL Cluster Replication, it is not possible to guarantee the transactional consistency of the COUNT() function on the slave. In other words, when performing on the master a series of statements (INSERT, DELETE, or both) that changes the number of rows in a table within a single transaction, executing SELECT COUNT(*) FROM table queries on the slave may yield intermediate results. This is due to the fact that SELECT COUNT(...) may perform dirty reads, and is not a bug in the NDB storage engine. (See Bug#31321 for more information.)

20.14.4. Error Handling

Starting, stopping, or restarting a node may give rise to temporary errors causing some transactions to fail. These include the following cases:

Temporary errors. When first starting a node, it is possible that you may see Error 1204 Temporary failure, distribution changed and similar temporary errors.
Errors due to node failure. The stopping or failure of any data node can result in a number of different node failure errors. (However, there should be no aborted transactions when performing a planned shutdown of the cluster.)

In either of these cases, any errors that are generated must be handled within the application. This should be done by retrying the transaction.

20.14.5. Limits Associated with Database Objects

Some database objects such as tables and indexes have different limitations when using the NDBCLUSTER storage engine:

Identifiers. Database names, table names and attribute names cannot be as long in NDB tables as when using other table handlers. Attribute names are truncated to 31 characters, and if not unique after truncation give rise to errors. Database names and table names can total a maximum of 122 characters. In other words, the maximum length for an NDB table name is 122 characters, less the number of characters in the name of the database of which that table is a part.
Table names containing special characters. NDB tables whose names contain characters other than letters, numbers, dashes, and underscores and which are created on one SQL node were not always discovered correctly by other SQL nodes. (Bug#31470)
Note
This issue was fixed in MySQL 5.1.23, MySQL Cluster NDB 6.2.7, and MySQL Cluster NDB 6.3.4.
Number of tables. The maximum number of NDB tables is limited to 20320.
Attributes per table. The maximum number of attributes (that is, columns and indexes) per table is limited to 128.
Attributes per key. The maximum number of attributes per key is 32.
Row size. The maximum permitted size of any one row is 8KB. Note that each BLOB or TEXT column contributes 256 + 8 = 264 bytes towards this total.

20.14.6. Unsupported Or Missing Features

A number of features supported by other storage engines are not supported for NDB tables. Trying to use any of these features in MySQL Cluster does not cause errors in or of itself; however, errors may occur in applications that expects the features to be supported or enforced:

Foreign key constraints. The foreign key construct is ignored, just as it is in MyISAM tables.
OPTIMIZE operations. OPTIMIZE operations are not supported.
Beginning with MySQL Cluster NDB 6.3.7, this limitation has been lifted. See Section 20.14.11, “Previous MySQL Cluster Issues Resolved in MySQL 5.1”, for more information.
LOAD TABLE ... FROM MASTER. LOAD TABLE ... FROM MASTER is not supported.
Savepoints and rollbacks. Savepoints and rollbacks to savepoints are ignored as in MyISAM.
Durability of commits. There are no durable commits on disk. Commits are replicated, but there is no guarantee that logs are flushed to disk on commit.
Replication. Statement-based replication is not supported. Use --binlog-format=ROW (or --binlog-format=MIXED) when setting up cluster replication. See Section 20.11, “MySQL Cluster Replication”, for more information.

Note

See Section 20.14.3, “Limits Relating to Transaction Handling”, for more information relating to limitations on transaction handling in NDB.

20.14.7. Limitations Relating to Performance

The following performance issues are specific to or especially pronounced in MySQL Cluster:

Range scans. There are query performance issues due to sequential access to the NDB storage engine; it is also relatively more expensive to do many range scans than it is with either MyISAM or InnoDB.
Reliability of Records in range. The Records in range statistic is available but is not completely tested or officially supported. This may result in non-optimal query plans in some cases. If necessary, you can employ USE INDEX or FORCE INDEX to alter the execution plan. See Section 12.2.7.2, “Index Hint Syntax”, for more information on how to do this.
Unique hash indexes. Unique hash indexes created with USING HASH cannot be used for accessing a table if NULL is given as part of the key.

20.14.8. Issues Exclusive to MySQL Cluster

The following are limitations specific to the NDBCLUSTER storage engine:

Machine architecture. All machines used in the cluster must have the same architecture. That is, all machines hosting nodes must be either big-endian or little-endian, and you cannot use a mixture of both. For example, you cannot have a management node running on a PowerPC which directs a data node that is running on an x86 machine. This restriction does not apply to machines simply running mysql or other clients that may be accessing the cluster's SQL nodes.
Adding and dropping of data nodes. Online adding or dropping of data nodes is not currently possible. In such cases, the entire cluster must be restarted.
Binary logging. MySQL Cluster has the following limitations or restrictions with regard to binary logging:
- SQL_LOG_BIN has no effect on data operations; however, it is supported for schema operations.
- MySQL Cluster cannot produce a binlog for tables having BLOB columns but no primary key.
- Only the following schema operations are logged in a cluster binlog which is not on the mysqld executing the statement:
  - CREATE TABLE
  - ALTER TABLE
  - DROP TABLE
  - CREATE DATABASE / CREATE SCHEMA
  - DROP DATABASE / DROP SCHEMA
  - CREATE TABLESPACE
  - ALTER TABLESPACE
  - DROP TABLESPACE
  - CREATE LOGFILE GROUP
  - ALTER LOGFILE GROUP
  - DROP LOGFILE GROUP

20.14.9. Limitations Relating to Disk Data Storage

Disk data objects are subject to the following maximums:
- Maximum number of tablespaces: 2^32 (4294967296)
- Maximum number of data files per tablespace: 2^16 (65535)
- The theoretical maximum number of extents per tablespace data file is 2^16 (65525); however, for practical purposes, the recommended maximum number of extents per data file is 2^8 (32768).
- Maximum data file size: The theoretical limit is 64G; however, in MySQL 5.1, the practical upper limit is 32G. This is equivalent to 32768 extents of 1M each.
  The minimum and maximum possible sizes of extents for tablespace data files are 32K and 2G, respectively. See Section 12.1.11, “CREATE TABLESPACE Syntax”, for more information.
Use of Disk Data tables is not supported when running the cluster in diskless mode. Beginning with MySQL 5.1.12, it is disallowed altogether. (Bug#20008)

20.14.10. Limitations Relating to Multiple Cluster Nodes

Multiple SQL nodes. The following are issues relating to the use of multiple MySQL servers as MySQL Cluster SQL nodes, and are specific to the NDBCLUSTER storage engine:

No distributed table locks. A LOCK TABLES works only for the SQL node on which the lock is issued; no other SQL node in the cluster “sees” this lock. This is also true for a lock issued by any statement that locks tables as part of its operations. (See next item for an example.)
ALTER TABLE operations. ALTER TABLE is not fully locking when running multiple MySQL servers (SQL nodes). (As discussed in the previous item, MySQL Cluster does not support distributed table locks.)
DDL operations. DDL operations (such as CREATE TABLE or ALTER TABLE) are not safe from data node failures. If a data node fails while trying to peform one of these, the data dictionary is locked and no further DDL statements can be executed without restarting the cluster.

Multiple management nodes. When using multiple management servers:

You must give nodes explicit IDs in connectstrings because automatic allocation of node IDs does not work across multiple management servers.
You must take extreme care to have the same configurations for all management servers. No special checks for this are performed by the cluster.

Multiple data node processes. While it is possible to run multiple cluster processes concurrently on a single host, it is not always advisable to do so for reasons of performance and high availability, as well as other considerations. In particular, in MySQL 5.1 and MySQL Cluster NDB 6.x, we do not support for production use any MySQL Cluster deployment in which more than one ndbd process is run on a single physical machine.

Note

We may support multiple data nodes per host in a future MySQL release, following additional testing. However, in current versions of MySQL Cluster, such configurations can be considered experimental only.

Multiple network addresses. Multiple network addresses per data node are not supported. Use of these is liable to cause problems: In the event of a data node failure, an SQL node waits for confirmation that the data node went down but never receives it because another route to that data node remains open. This can effectively make the cluster inoperable.

Note

It is possible to use multiple network hardware interfaces (such as Ethernet cards) for a single data node, but these must be bound to the same address. This also means that it not possible to use more than one [tcp] section per connection in the config.ini file. See Section 20.3.4.7, “Cluster TCP/IP Connections”, for more information.

20.14.11. Previous MySQL Cluster Issues Resolved in MySQL 5.1

A number of limitations and related issues existing in earlier versions of MySQL Cluster have been resolved:

Variable-length column support. The NDBCLUSTER storage engine now supports variable-length column types for in-memory tables.
Previously, for example, any Cluster table having one or more VARCHAR fields which contained only relatively small values, much more memory and disk space were required when using the NDBCLUSTER storage engine than would have been the case for the same table and data using the MyISAM engine. In other words, in the case of a VARCHAR column, such a column required the same amount of storage as a CHAR column of the same size. In MySQL 5.1, this is no longer the case for in-memory tables, where storage requirements for variable-length column types such as VARCHAR and BINARY are comparable to those for these column types when used in MyISAM tables (see Section 10.5, “Data Type Storage Requirements”).
Important
For MySQL Cluster Disk Data tables, the fixed-width limitation continues to apply. See Section 20.12, “MySQL Cluster Disk Data Tables”.
Replication with MySQL Cluster. It is now possible to use MySQL replication with Cluster databases. For details, see Section 20.11, “MySQL Cluster Replication”.
Circular Replication. Circular replication is supported for MySQL Cluster beginning with MySQL 5.1.18. See Section 20.11.3, “Known Issues in MySQL Cluster Replication”.
auto_increment_increment and auto_increment_offset. The auto_increment_increment and auto_increment_offset server system variables are supported for Cluster replication beginning with MySQL 5.1.20.
Database autodiscovery and online schema changes. Autodiscovery of databases is now supported for multiple MySQL servers accessing the same MySQL Cluster, provided that a given mysqld is already running and is connected to the cluster at the time that the database is created on a different mysqld.
What this means is that if a mysqld process first connects to the cluster after a database named db_name has been created, you should issue a CREATE SCHEMA db_name statement on the “new” MySQL server when it first accesses that MySQL Cluster. Once this has been done, the “new” mysqld should be able to detect any tables in that database tables without errors.
This also means that online schema changes in NDB tables are now possible. That is, the result of operations such as ALTER TABLE and CREATE INDEX performed on one SQL node in the cluster are now visible to the cluster's other SQL nodes without any additional action being taken.
Backup and restore between architectures. Beginning with MySQL 5.1.10, it is possible to perform a Cluster backup and restore between different architectures. Previously — for example — you could not back up a cluster running on a big-endian platform and then restore from that backup to a cluster running on a little-endian system. (Bug#19255)
Character set directory. Beginning with MySQL 5.1.10, it is possible to install MySQL with Cluster support to a non-default location and change the search path for font description files using either the --basedir or --character-sets-dir options. (Previously, ndbd in MySQL 5.1 searched only the default path — typically /usr/local/mysql/share/mysql/charsets — for character sets.)
In MySQL 5.1, it is no longer necessary, when running multiple management servers, to restart all the cluster's data nodes to enable the management nodes to see one another.
Length of CREATE TABLE statements. CREATE TABLE statements may be no more than 4096 characters in length. This limitation affects MySQL 5.1.6, 5.1.7, and 5.1.8 only. (See Bug#17813)
IGNORE and REPLACE functionality. In MySQL 5.1.7 and earlier, INSERT IGNORE, UPDATE IGNORE, and REPLACE were supported only for primary keys, but not for unique keys. It was possible to work around this issue by removing the constraint, then dropping the unique index, performing any inserts, and then adding the unique index again.
This limitation was removed for INSERT IGNORE and REPLACE in MySQL 5.1.8. (See Bug#17431.)
AUTO_INCREMENT columns. In MySQL 5.1.10 and earlier versions, the maximum number of tables having AUTO_INCREMENT columns — including those belonging to hidden primary keys — was 2048.
This limitation was lifted in MySQL 5.1.11.
Maximum number of cluster nodes. Prior to MySQL Cluster NDB 6.1.1, the total maximum number of nodes in a MySQL Cluster was 63, including all SQL nodes (MySQL Servers), API nodes (applications accessing the cluster other than MySQL servers), data nodes, and management servers.
Starting with MySQL Cluster NDB 6.1.1, up to 255 API nodes (including MySQL servers acting as cluster SQL nodes) are supported by a single cluster. The total number of data nodes and management nodes beginning with this version is 63, of which up to 48 can be data nodes.
Note
The limitation that a data node cannot have a node ID greater than 49 continues to apply.
Recovery of memory from deleted rows. Beginning with MySQL Cluster NDB 6.3.7, memory can be reclaimed from an NDB table for reuse with any NDB table by employing OPTIMIZE TABLE, subject to the following limitations:
- Only in-memory tables are supported; the OPTIMIZE TABLE statement still has no effect on MySQL Cluster Disk Data tables.
- Only variable-length columns (such as those declared as VARCHAR, TEXT, or BLOB) are supported.
  However, you can force columns defined using fixed-length data types (such as CHAR) to be dynamic using the ROW_FORMAT or COLUMN_FORMAT option with a CREATE TABLE or ALTER TABLE statement.
  See Section 12.1.10, “CREATE TABLE Syntax”, and Section 12.1.4, “ALTER TABLE Syntax”, for information on these options.
You can regulate the effects of OPTIMIZE on performance by adjusting the value of the global system variable ndb_optimization_delay, which sets the number of milliseconds to wait between batches of rows being processed by OPTIMIZE. The default value is 10 milliseconds. It is possible to set a lower value (to a minimum of 0), but not recommended. The maximum is 100000 milliseconds (that is, 100 seconds).

20.15. MySQL Cluster Development Roadmap

20.15.1. Features Added in MySQL 5.1 Cluster
20.15.2. Features Added in MySQL Cluster NDB 6.1
20.15.3. Features Added in MySQL Cluster NDB 6.2
20.15.4. Features Added in MySQL Cluster NDB 6.3

In this section, we discuss changes in the implementation of MySQL Cluster in MySQL 5.1 and MySQL Cluster NDB as compared to MySQL 5.0.

There are a number of significant changes in the implementation of the NDBCLUSTER storage engine in mainline MySQL 5.1 releases up to and including MySQL 5.1.23 as compared to that in MySQL 5.0; MySQL Cluster NDB makes further changes and improvements in MySQL Cluster in addition to these. The changes and features most likely to be of interest are shown in the following table:

MySQL 5.1 (through 5.1.23)
MySQL Cluster Replication
Disk Data storage
Variable-size columns
User-defined partitioning
Autodiscovery of table schema changes
Online adding and dropping of indexes

MySQL Cluster NDB 6.1
Greater number of cluster nodes
Disabling of arbitration
Additional `DUMP` commands
Faster Disk Data backups
Batched slave updates

MySQL Cluster NDB 6.2
Improved backup status reporting (`BackupReportFrequency`, `REPORT BackupStatus`)
Multiple connections per SQL node
Data access with `NdbRecord` (NDB API)
`REPORT MemoryUsage` command
Memory allocation improvements
Management client connection control
Micro-GCPs
Online `ADD COLUMN`; improved online index creation

MySQL Cluster NDB 6.3
Conflict detection and resolution for multi-master replication
Compressed backups and local checkpoints
Support for `OPTIMIZE TABLE`
Parallel data node recovery
Enhanced transaction coordinator selection
Improved SQL statement performance metrics
Transaction batching
ndb_restore attribute promotion
Support for `epoll` (Linux only)
Distribution awareness
`NDB` thread locks; realtime extensions for multiple CPUs

20.15.1. Features Added in MySQL 5.1 Cluster

A number of new features for MySQL Cluster have been implemented in MySQL 5.1 through MySQL 5.1.23, when support for MySQL Cluster was moved to MySQL Cluster NDB. All of the features in the following list are also available in all MySQL Cluster NDB (6.1 and later) releases.

Integration of MySQL Cluster into MySQL Replication. MySQL Cluster Replication makes it possible to replicate from one MySQL Cluster to another. Updates on any SQL node (MySQL server) in the cluster acting as the master are replicated to the slave cluster; the state of the slave side remains consistent with the cluster acting as the master. This is sometimes referred to as asynchronous replication between clusters, providing geographic redundancy. It is also possible to replicate from a MySQL Cluster acting as the master to a standalone MySQL server acting as the slave, or from a standalone MySQL master server to to a slave cluster; in either of these cases, the standalone MySQL server uses a storage engine other than NDBCLUSTER. Multi-master replication setups such as circular replication are also supported.
See Section 20.11, “MySQL Cluster Replication”.
Support for storage of rows on disk. Storage of NDBCLUSTER table data on disk is now supported. Indexed columns, including the primary key hash index, must still be stored in RAM; however, all other columns can be stored on disk.
See Section 20.12, “MySQL Cluster Disk Data Tables”.
Variable-size columns. In MySQL 5.0, an NDBCLUSTER table column defined as VARCHAR(255) used 260 bytes of storage independent of what was stored in any particular record. In MySQL 5.1 Cluster tables, only the portion of the column actually taken up by the record is stored. This makes possible a significant reduction in space requirements for such columns as compared to previous release series — by a factor of up to 5 in many cases.
User-defined partitioning. Users can define partitions based on columns that are part of the primary key. It is possible to partition NDB tables based on KEY and LINEAR KEY schemes. This feature is also available for many other MySQL storage engines, which support additional partitioning types that are not available with NDBCLUSTER tables.
For additional general information about user-defined partitioning in MySQL 5.1, see Chapter 21, Partitioning. Specifics of partitioning types are discussed in Section 21.2, “Partition Types”.
The MySQL Server can also determine whether it is possible to “prune away” some of the partitions from the WHERE clause, which can greatly speed up some queries. See Section 21.4, “Partition Pruning”, for information about designing tables and queries to take advantage of partition pruning.
Autodiscovery of table schema changes. In MySQL 5.0, it was necessary to issue a FLUSH TABLES statement or a “dummy” SELECT in order for new NDBCLUSTER tables or changes made to schemas of existing NDBCLUSTER tables on one SQL node to be visible on the cluster's other SQL nodes. In MySQL 5.1, this is no longer necessary; new Cluster tables and changes in the definitions of existing NDBCLUSTER tables made on one SQL node are immediately visible to all SQL nodes connected to the cluster.
Note
When creating a new database, it is still necessary in MySQL 5.1 to issue a CREATE DATABASE or CREATE SCHEMA statement on each SQL node in the cluster.
Distribution awareness (NDB API). Distribution awareness is a mechanism by which the best data node is automatically selected to be queried for information. (Conceptually, it is similar in some ways to partition pruning (see Section 21.4, “Partition Pruning”). To take advantage of distribution awareness, you should do the following:
1. Determine which table column is most likely to be used for finding matching records.
2. Make this column part of the table's primary key.
3. Explicitly partition the table by KEY, using this column as the table' partitioning key.
Following these steps causes records with the same value for the partitioning column to be stored on the same partition (that is, in the same node group). When reading data, transactions are begun on the data node actually having the desired rows instead of this node being determined by the usual round-robin mechanism.
Important
In order to see a measureable impact on performance, the cluster must have at least four data nodes, since, with only two data nodes, both data nodes have exactly the same data.
Using distribution awareness can yield performance increase of as great as 45% when using four data nodes, and possibly more when using a greater number of data nodes.
Note
In mainline MySQL 5.1 releases, distribution awareness was supported only when using the NDB API; support was added for SQL and API nodes in MySQL Cluster NDB 6.3 (see Section 20.15.4, “Features Added in MySQL Cluster NDB 6.3”, which includes an example showing how to create a table in order to take advantage of distribution awareness).

See Section 20.14.11, “Previous MySQL Cluster Issues Resolved in MySQL 5.1”, for more information.

20.15.2. Features Added in MySQL Cluster NDB 6.1

The following list provides an overview of significant feature additions and changes made in MySQL Cluster NDB 6.1. All of the changes in this list are also available in MySQL Cluster NDB 6.2 and 6.3 releases. For detailed information about all changes made in MySQL Cluster NDB 6.1, see Section 20.17.3, “Changes in MySQL Cluster NDB 6.1”.

Increased number of cluster nodes. The maximum number of all nodes in a MySQL Cluster has been increased to 255. For more information, see Section 20.14.10, “Limitations Relating to Multiple Cluster Nodes”.
Disabling arbitration. It is now possible to disable arbitration by setting ArbitrationRank=0 on all cluster management and SQL nodes. For more information, see Defining the Management Server: ArbitrationRank and Defining SQL and Other API Nodes: ArbitrationRank.
Additional DUMP commands. New management client DUMP commands provide help with tracking transactions, scan operations, and locks. See Section 20.7.2, “Commands in the MySQL Cluster Management Client”, and DUMP Commands, for more information.
Faster Disk Data backups. Improvements in backups of Disk Data tables can yield a 10 to 15% increase in backup speed of Disk Data tables.
Batched slave updates. Batching of updates on cluster replication slaves, enabled using the --slave-allow-batching option for mysqld, increases replication efficiency. For more information, see Section 20.11.6, “Starting Replication (Single Replication Channel)”.

20.15.3. Features Added in MySQL Cluster NDB 6.2

The following list provides an overview of significant feature additions and changes made in MySQL Cluster NDB 6.2. All of the changes in this list are also available in MySQL Cluster NDB 6.3 . For more detailed information about all feature changes and bugfixes made in MySQL Cluster NDB 6.2, see Section 20.17.2, “Changes in MySQL Cluster NDB 6.2”.

Enhanced backup status reporting. Backup status reporting has been improved, aided in part by the introduction of a BackupReportFrequency configuration parameter; see Defining Data Nodes: BackupReportFrequency, for more information.
Multiple cluster connections per SQL node. A single MySQL server acting as a MySQL Cluster SQL node can employ multiple connections to the cluster using the --ndb-cluster-connection-pool startup option for mysqld. This option is described in MySQL Cluster-Related Command Options for mysqld: --ndb-cluster-connection-pool option.
New data access interface. The NdbRecord interface provides a new and simplified data handler for use in NDB API applications. See The NdbRecord Interface, for more information.
New reporting commands. The new management client REPORT BackupStatus and REPORT MemoryUsage commands provide better access to information about the status of MySQL Cluster backups and how much memory is being used by MySQL Cluster for data and index storage. See Section 20.7.2, “Commands in the MySQL Cluster Management Client”, for more information about the REPORT commands. In addition, in-progress status reporting is provided by the ndb_restore utility; see Section 20.9.3, “ndb_restore — Restore a Cluster Backup”.
Improved memory allocation and configuration. Memory is now allocated by the NDB kernel to tables on a page-by-page basis, which significantly reduces the memory overhead required for maintaining NDBCLUSTER tables. In addition, the MaxAllocate configuration parameter now makes it possible to set the maximum size of the allocation unit used for table memory; for more information about this configuration parameter, see Defining Data Nodes: MaxAllocate.
Choice of fixed-width or variable-width columns. You can control whether fixed-width or variable-width storage is used for a given column of an NDB table by employing of the COLUMN_FORMAT specifier as part of the column's definition in a CREATE TABLE or ALTER TABLE statement. In addition, the ability to control whether a given column of an NDB table is stored in memory or on disk, using the STORAGE specifier as part of the column's definition in a CREATE TABLE or ALTER TABLE statement. For more information, see Section 12.1.10, “CREATE TABLE Syntax”, and Section 12.1.4, “ALTER TABLE Syntax”.
Controlling management client connections. The --bind-address cluster management server startup option makes it possible to restrict management client connections to ndb_mgmd to a single host (IP address or hostname) and port, which can make MySQL Cluster management operations more secure. For more information about this option, see Section 20.6.5.2, “Command Options for ndb_mgmd”.
Micro-GCPs. Due to a change in the protocol for handling of global checkpoints (GCPs handled in this manner sometimes being referred to as “micro-GCPs”), it is now possible to control how often the GCI number is updated, and how often global checkpoints are written to disk, using the TimeBetweenEpochs configuration parameter. This improves the reliability and performance of MySQL Cluster Replication. For more information, see Defining Data Nodes: TimeBetweenEpochs and Defining Data Nodes: TimeBetweenEpochsTimeout.
Core online schema change support. Support for the online ALTER TABLE operations ADD COLUMN, ADD INDEX, and DROP INDEX is available. When the ONLINE keyword is used, the ALTER TABLE is non-copying, which means that indexes do not have to be re-created, which provides these benefits:
- Single user mode is no longer required for ALTER TABLE operations that can be performed online.
- Transactions can continue during ALTER TABLE operations that can be performed online.
- Tables being altered online are not locked.
Online CREATE INDEX and DROP INDEX statements are also supported. Online changes can be suppressed using the OFFLINE key word. See Section 12.1.4, “ALTER TABLE Syntax”, Section 12.1.7, “CREATE INDEX Syntax”, and Section 12.1.13, “DROP INDEX Syntax”, for more detailed information.
mysql.ndb_binlog_index improvements. More information has been added to the mysql.ndb_binlog_index table so that it is possible to determine which originating epochs have been applied inside an epoch. This is particularly useful for 3-way replication. See Section 20.11.4, “Cluster Replication Schema and Tables”, for more information.
Epoch lag control. The MaxBufferedEpochs data node configuration parameter provides a means to control the maximum number of unprocessed epochs by which a subscribing node can lag. Subscribers which exceed this number are disconnected and forced to reconnect. For a discussion of this configuration parameter, see Defining Data Nodes: MaxBufferedEpochs.

20.15.4. Features Added in MySQL Cluster NDB 6.3

The following list provides an overview of significant feature additions and changes first made in MySQL Cluster NDB 6.3. For more detailed information about all feature changes and bugfixes made in MySQL Cluster NDB 6.3, see Section 20.17.1, “Changes in MySQL Cluster NDB 6.3”.

Conflict detection and resolution. It is now possible to detect and resolve conflicts that arise in multi-master replication scenarios, such as circular replication, when different masters may try to update the same row on the slave with different data. Both “greatest timestamp wins” and “same timestamp wins” scenarios are supported. For more information, see Section 20.11.10, “MySQL Cluster Replication Conflict Resolution”.
Recovery of “one master, many slaves” replication setups. Recovery of multi-way replication setups (“one master, many slaves”) is now supported via the --ndb-log-orig server option and changes in the mysql.ndb_binlog_index table. See Section 20.11.4, “Cluster Replication Schema and Tables”, for more information.
Enhanced selection options for transaction coordinator. New values and behaviors are introduced for --ndb_optimized_node_selection allowing for greater flexibility when an SQL node chooses a transaction coordinator. See MySQL Cluster System Variables: ndb_optimized_node_selection, for more information.
Replication heartbeats. Replication heartbeats facilitate the task of monitoring and detecting failures in master-slave connections in real time. This feature is implemented via a new MASTER_HEARTBEAT_PERIOD = value clause for the CHANGE MASTER TO statement and the addition of two status variables Slave_heartbeat_period and Slave_received_heartbeats. For more information, see Section 12.6.2.1, “CHANGE MASTER TO Syntax”.
NDB thread locks. It is possible to lock NDB execution threads and maintenance threads (such as filesystem and other operating system threads) to specific CPUs on multiprocessor data node hosts, and to leverage real-time scheduling.
Improved performance of updates using primary keys or unique keys. The number of unnecessary reads when performing a primary key or unique key update has been greatly reduced. Since it is seldom necessary to read a record prior to an update, this can yield a considerable improvement in performance. In addition, primary key columns are no longer written to when not needed during update operations.
Batching improvements. Support of batched DELETE and UPDATE operations has been significantly improved. Batching of UPDATE WHERE... and multiple DELETE operations is also now implemented.
Improved SQL statement performance metrics. The Ndb_execute_count system status variable measures the number of round trips made by SQL statements to the NDB kernel, providing an improved metric for determining efficiency with which statements are excuted. For more information, see MySQL Cluster Status Variables: Ndb_execute_count.
Compressed LCPs and backups. Compressed local checkpoints and backups can save 50% or more of the disk space used by uncompressed LCPs and backups. These can be enabled using the two new data node configuration parameters CompressedLCP and CompressedBackup, respectively. See MySQL Cluster Status Variables: CompressedBackup, and MySQL Cluster Status Variables: CompressedLCP, for more information about these parameters.
OPTIMIZE TABLE support with NDBCLUSTER tables. OPTIMIZE TABLE is supported for dynamic columns of in-memory NDB tables. In such cases, it is no longer necessary to drop (and possibly to re-create) a table, or to perform a rolling restart, in order to recover memory from deleted rows for general re-use by Cluster. The performance of OPTIMIZE on Cluster tables can be tuned by adjusting the value of the ndb_optimization_delay system variable, which controls the number of milliseconds to wait between processing batches of rows by OPTIMIZE TABLE. In addition, OPTIMIZE TABLE on an NDBCLUSTER table can be interrupted by, for example, killing the SQL thread performing the OPTIMIZE operation.
Batching of transactions. It is possible to cause statements occurring within the same transaction to be run as a batch by setting the session variable transaction_allow_batching to 1 or ON. To use this feature, AUTOCOMMIT must be set to 0 or OFF. Batch sizes can be controlled using the --ndb-batch-size option for mysqld. For more information, see Section 20.4.2, “MySQL Cluster-Related Command Options for mysqld”, and Section 20.4.3, “MySQL Cluster System Variables”.
Attribute promotion with ndb_restore. It is possible using ndb_restore to restore data reliably from a column of a given type to a column that uses a “larger” type. This is sometimes referred to as attribute promotion. For example, MySQL Cluster backup data that originated in a SMALLINT column can be restored to a MEDIUMINT, INT, or BIGINT column. See Section 20.9.3, “ndb_restore — Restore a Cluster Backup”, for more information.
Parallel data node recovery. Recovery of multiple data nodes can now be done in parallel, rather than sequentially. In other words, several data nodes can be restored concurrently, which can often result in much faster recovery times than when they are restored one at a time.
Increased local checkpoint efficiency. Only 2 local checkpoints are stored, rather than 3, lowering disk space requirements and the size and number of redo log files.
NDBCLUSTER table persistence control. Persistence of NDB tables can be controlled using the session variables ndb_table_temporary and ndb_table_no_logging. ndb_table_no_logging causes NDB tables not to be checkpointed to disk; ndb_table_temporary does the same, and in addition, no schema files are created. See Section 20.4.1, “MySQL Cluster Server Option and Variable Reference”.
Epoll support (Linux only). Epoll is an improved method for handling file descriptors, which is more efficient than scanning to determine whether a file descriptor has data to be read. (The term epoll is specific to Linux and equivalent functionality is known by other names on other platforms such as Solaris and FreeBSD.) Currently, MySQL Cluster supports this functionality on Linux only.
Distribution awareness (SQL nodes). In MySQL Cluster NDB 6.3, SQL nodes can take advantage of distribution awareness. Here we provide a brief example showing how to design a table to make a given class of queries distrubtion-aware. Suppose an NDBCLUSTER table t1 has the following schema:
```
CREATE TABLE t1 (
    userid INT NOT NULL,
    serviceid INT NOT NULL AUTO_INCREMENT PRIMARY KEY,
    data VARCHAR(255)
)   ENGINE=NDBCLUSTER;
```
Suppose further that most of the queries to be used in our application test values of the userid column of this table. The form of such a query looks something like this:
```
SELECT columns FROM t1 
    WHERE userid relation value;  
```
In this query, relation represents some relational operator, such as =, <, >, and so on. Queries using IN and a list of values can also be used:
```
SELECT columns FROM t1 
    WHERE userid IN value_list;  
```
In order to make use of distribution awareness, we need to make the userid column part of the table's primary key, then explicitly partition the table with this column being used as the partitioning key. (Recall that for a partitioned table having one or more unique keys, all columns of the table's partitioning key must also be part of all of the unique keys — for more information and examples, see Section 21.5.1, “Partitioning Keys, Primary Keys, and Unique Keys”.) In other words, the table schema should be equivalent to the following CREATE TABLE statement:
```
CREATE TABLE t1 (
    userid INT NOT NULL,
    serviceid INT NOT NULL AUTO_INCREMENT,
    data VARCHAR(255),
    PRIMARY KEY p (userid,serviceid)
)   ENGINE=NDBCLUSTER
    PARTITION BY KEY(userid);
```
When the table is partitioned in this way, all rows having the same userid value are found on the same node group, and the MySQL Server can immediately select the optimal node to use as the transaction coordinator.
Realtime extensions for multiple CPUs. When running MySQL Cluster data nodes on hosts with multiple processors, the realtime extensions make it possible to give priority to the data node process and control on which CPU cores it should operate. This can be done using the data node configuration parameters RealtimeScheduler, SchedulerExecutionTimer and SchedulerSpinTimer. Doing so properly can significantly lower response times and make them much more predictable response. For more information about using these parameters, see Defining Data Nodes: Realtime Performance Parameters

20.16. MySQL Cluster Glossary

The following terms are useful to an understanding of MySQL Cluster or have specialized meanings when used in relation to it.

Cluster. In its generic sense, a cluster is a set of computers functioning as a unit and working together to accomplish a single task.
NDBCLUSTER. This is the storage engine used in MySQL to implement data storage, retrieval, and management distributed among several computers.
MySQL Cluster. This refers to a group of computers working together using the NDB storage engine to support a distributed MySQL database in a shared-nothing architecture using in-memory storage.
Configuration files. Text files containing directives and information regarding the cluster, its hosts, and its nodes. These are read by the cluster's management nodes when the cluster is started. See Section 20.3.4, “Configuration File”, for details.
Backup. A complete copy of all cluster data, transactions and logs, saved to disk or other long-term storage.
Restore. Returning the cluster to a previous state, as stored in a backup.
Checkpoint. Generally speaking, when data is saved to disk, it is said that a checkpoint has been reached. More specific to Cluster, it is a point in time where all committed transactions are stored on disk. With regard to the NDB storage engine, there are two types of checkpoints which work together to ensure that a consistent view of the cluster's data is maintained:
- Local Checkpoint (LCP). This is a checkpoint that is specific to a single node; however, LCP's take place for all nodes in the cluster more or less concurrently. An LCP involves saving all of a node's data to disk, and so usually occurs every few minutes. The precise interval varies, and depends upon the amount of data stored by the node, the level of cluster activity, and other factors.
- Global Checkpoint (GCP). A GCP occurs every few seconds, when transactions for all nodes are synchronized and the redo-log is flushed to disk.
Cluster host. A computer making up part of a MySQL Cluster. A cluster has both a physical structure and a logical structure. Physically, the cluster consists of a number of computers, known as cluster hosts (or more simply as hosts. See also Node and Node group below.
Node. This refers to a logical or functional unit of MySQL Cluster, and is sometimes also referred to as a cluster node. In the context of MySQL Cluster, we use the term “node” to indicate a process rather than a physical component of the cluster. There are three node types required to implement a working MySQL Cluster:
- Management nodes. Manages the other nodes within the MySQL Cluster. It provides configuration data to the other nodes; starts and stops nodes; handles network partitioning; creates backups and restores from them, and so forth.
- SQL nodes. Instances of MySQL Server which serve as front ends to data kept in the cluster's data nodes. Clients desiring to store, retrieve, or update data can access an SQL node just as they would any other MySQL Server, employing the usual MySQL authentication methods and APIs; the underlying distribution of data between node groups is transparent to users and applications. SQL nodes access the cluster's databases as a whole without regard to the data's distribution across different data nodes or cluster hosts.
- Data nodes. These nodes store the actual data. Table data fragments are stored in a set of node groups; each node group stores a different subset of the table data. Each of the nodes making up a node group stores a replica of the fragment for which that node group is responsible. Currently, a single cluster can support up to 48 data nodes total.
It is possible for more than one node to co-exist on a single machine. (In fact, it is even possible to set up a complete cluster on one machine, although one would almost certainly not want to do this in a production environment.) It may be helpful to remember that, when working with MySQL Cluster, the term host refers to a physical component of the cluster whereas a node is a logical or functional component (that is, a process).
Note Regarding Terms. In older versions of the MySQL Cluster documentation, data nodes were sometimes referred to as “database nodes”. The term “storage nodes” has also been used. In addition, SQL nodes were sometimes known as “client nodes”. This older terminology has been deprecated to minimize confusion, and for this reason should be avoided. They are also often referred to as “API nodes” — an SQL node is actually an API node that provides an SQL interface to the cluster.
Node group. A set of data nodes. All data nodes in a node group contain the same data (fragments), and all nodes in a single group should reside on different hosts. It is possible to control which nodes belong to which node groups.
For more information, see Section 20.1.2, “MySQL Cluster Nodes, Node Groups, Replicas, and Partitions”.
Node failure. MySQL Cluster is not solely dependent upon the functioning of any single node making up the cluster; the cluster can continue to run if one or more nodes fail. The precise number of node failures that a given cluster can tolerate depends upon the number of nodes and the cluster's configuration.
Node restart. The process of restarting a failed cluster node.
Initial node restart. The process of starting a cluster node with its filesystem removed. This is sometimes used in the course of software upgrades and in other special circumstances.
System crash (or system failure). This can occur when so many cluster nodes have failed that the cluster's state can no longer be guaranteed.
System restart. The process of restarting the cluster and reinitializing its state from disk logs and checkpoints. This is required after either a planned or an unplanned shutdown of the cluster.
Fragment. A portion of a database table; in the NDB storage engine, a table is broken up into and stored as a number of fragments. A fragment is sometimes also called a “partition”; however, “fragment” is the preferred term. Tables are fragmented in MySQL Cluster in order to facilitate load balancing between machines and nodes.
Replica. Under the NDB storage engine, each table fragment has number of replicas stored on other data nodes in order to provide redundancy. Currently, there may be up four replicas per fragment.
Transporter. A protocol providing data transfer between nodes. MySQL Cluster currently supports four different types of transporter connections:
- TCP/IP. This is, of course, the familiar network protocol that underlies HTTP, FTP (and so on) on the Internet. TCP/IP can be used for both local and remote connections.
- SCI. Scalable Coherent Interface is a high-speed protocol used in building multiprocessor systems and parallel-processing applications. Use of SCI with MySQL Cluster requires specialized hardware, as discussed in Section 20.13.1, “Configuring MySQL Cluster to use SCI Sockets”. For a basic introduction to SCI, see this essay at dolphinics.com.
- SHM. Unix-style shared memory segments. Where supported, SHM is used automatically to connect nodes running on the same host. The Unix man page for shmop(2) is a good place to begin obtaining additional information about this topic.
Note
The cluster transporter is internal to the cluster. Applications using MySQL Cluster communicate with SQL nodes just as they do with any other version of MySQL Server (via TCP/IP, or through the use of Unix socket files or Windows named pipes). Queries can be sent and results retrieved using the standard MySQL client APIs.
NDB. This stands for Network Database, and refers to the storage engine used to enable MySQL Cluster. The NDB storage engine supports all the usual MySQL data types and SQL statements, and is ACID-compliant. This engine also provides full support for transactions (commits and rollbacks).
Shared-nothing architecture. The ideal architecture for a MySQL Cluster. In a true shared-nothing setup, each node runs on a separate host. The advantage such an arrangement is that there no single host or node can act as single point of failure or as a performance bottle neck for the system as a whole.
In-memory storage. All data stored in each data node is kept in memory on the node's host computer. For each data node in the cluster, you must have available an amount of RAM equal to the size of the database times the number of replicas, divided by the number of data nodes. Thus, if the database takes up 1GB of memory, and you want to set up the cluster with four replicas and eight data nodes, a minimum of 500MB memory will be required per node. Note that this is in addition to any requirements for the operating system and any other applications that might be running on the host.
In MySQL 5.1, it is also possible to create Disk Data tables where non-indexed columns are stored on disk, thus reducing the memory footprint required by the cluster. Note that indexes and indexed column data are still stored in RAM. See Section 20.12, “MySQL Cluster Disk Data Tables”.
Table. As is usual in the context of a relational database, the term “table” denotes a set of identically structured records. In MySQL Cluster, a database table is stored in a data node as a set of fragments, each of which is replicated on additional data nodes. The set of data nodes replicating the same fragment or set of fragments is referred to as a node group.
Cluster programs. These are command-line programs used in running, configuring, and administering MySQL Cluster. They include both server daemons:
- ndbd:
  The data node daemon (runs a data node process)
- ndb_mgmd:
  The management server daemon (runs a management server process)
and client programs:
- ndb_mgm:
  The management client (provides an interface for executing management commands)
- ndb_waiter:
  Used to verify status of all nodes in a cluster
- ndb_restore:
  Restores cluster data from backup
For more about these programs and their uses, see Section 20.6, “Process Management in MySQL Cluster”.
Event log. MySQL Cluster logs events by category (startup, shutdown, errors, checkpoints, and so on), priority, and severity. A complete listing of all reportable events may be found in Section 20.7.3, “Event Reports Generated in MySQL Cluster”. Event logs are of two types:
- Cluster log. Keeps a record of all desired reportable events for the cluster as a whole.
- Node log. A separate log which is also kept for each individual node.
Under normal circumstances, it is necessary and sufficient to keep and examine only the cluster log. The node logs need be consulted only for application development and debugging purposes.
Angel process. When a data node is started, ndbd actually starts two processes. One of these is known as the “angel” process; its purpose is to check to make sure that the main ndbd process continues to run, and to restart the main process if it should stop for any reason.
Watchdog thread. Each ndbd process has an internal watchdog thread which monitors the main worker thread, ensuring forward progress and a timely response to cluster protocols such as the cluster heartbeat. If the ndbd process is not being woken up promptly by the operating system when its sleep time expires, INFO and WARNING events, which are identifiable because they contain “Watchdog:...”, are written to the cluster log. Such messages are usually a symptom of an overloaded system; you should see what else is running on the system, and whether the ndbd process is being swapped out to disk. If ndbd cannot wake up regularly then it cannot respond to heartbeat messages on time, and other nodes eventually consider it “dead” due to the missed heartbeats, causing it to be excluded from the cluster.

20.17. Changes in MySQL Cluster 6.x

20.17.1. Changes in MySQL Cluster NDB 6.3
20.17.2. Changes in MySQL Cluster NDB 6.2
20.17.3. Changes in MySQL Cluster NDB 6.1

This section contains changelog information for MySQL Cluster that use version 6.1, 6.2, and 6.3 of the NDBCLUSTER storage engine.

Each MySQL Cluster release is based on a mainline MySQL 5.1 release and a particular version of the NDBCLUSTER storage engine, as shown in the version string returned by executing SELECT VERSION() in the mysql client, or by executing the ndb_mgm client SHOW or STATUS command; for more information, see Chapter 20, MySQL Cluster.

For general information about features added in MySQL Cluster, see Section 20.15, “MySQL Cluster Development Roadmap”. For a complete list of all bugfixes and feature changes in MySQL Cluster, please refer to the changelog section for each individual MySQL Cluster release.

An overview of features added in MySQL 5.1 not specific to MySQL Cluster can be found here: Section 1.5.1, “What's New in MySQL 5.1”. For a complete list of all bugfixes and features changes made in MySQL 5.1 that are not specific to MySQL Cluster, see Section C.1, “Changes in release 5.1.x (Development)”.

20.17.1. Changes in MySQL Cluster NDB 6.3

20.17.1.1. Changes in MySQL Cluster NDB 6.3.16 (5.1.24-ndb-6.3.16) (27 June 2008)
20.17.1.2. Changes in MySQL Cluster NDB 6.3.15 (5.1.24-ndb-6.3.15) (30 May 2008)
20.17.1.3. Changes in MySQL Cluster NDB 6.3.14 (5.1.24-ndb-6.3.14) (11 May 2008)
20.17.1.4. Changes in MySQL Cluster NDB 6.3.13 (5.1.24-ndb-6.3.13) (10 April 2008)
20.17.1.5. Changes in MySQL Cluster NDB 6.3.12 (5.1.23-ndb-6.3.12) (05 April 2008)
20.17.1.6. Changes in MySQL Cluster NDB 6.3.11 (5.1.23-ndb-6.3.11) (28 March 2008)
20.17.1.7. Changes in MySQL Cluster NDB 6.3.10 (5.1.23-ndb-6.3.10) (17 February 2008)
20.17.1.8. Changes in MySQL Cluster NDB 6.3.9 (5.1.23-ndb-6.3.9) (12 February 2008)
20.17.1.9. Changes in MySQL Cluster NDB 6.3.8 (5.1.23-ndb-6.3.8) (29 January 2008)
20.17.1.10. Changes in MySQL Cluster NDB 6.3.7 (5.1.23-ndb-6.3.7) (19 December 2007)
20.17.1.11. Changes in MySQL Cluster NDB 6.3.6 (5.1.22-ndb-6.3.6) (08 November 2007)
20.17.1.12. Changes in MySQL Cluster NDB 6.3.5 (5.1.22-ndb-6.3.5) (17 October 2007)
20.17.1.13. Changes in MySQL Cluster NDB 6.3.4 (5.1.22-ndb-6.3.4) (15 October 2007)
20.17.1.14. Changes in MySQL Cluster NDB 6.3.3 (5.1.22-ndb-6.3.3) (20 September 2007)
20.17.1.15. Changes in MySQL Cluster NDB 6.3.2 (5.1.22-ndb-6.3.2) (07 September 2007)
20.17.1.16. Changes in MySQL Cluster NDB 6.3.1 (5.1.19-ndb-6.3.1) (04 July 2007)
20.17.1.17. Changes in MySQL Cluster NDB 6.3.0 (5.1.19-ndb-6.3.0) (02 July 2007)

This section contains change history information for MySQL Cluster releases based on version 6.3 of the NDBCLUSTER storage engine.

For an overview of new features added in MySQL Cluster NDB 6.3, see Section 20.15.4, “Features Added in MySQL Cluster NDB 6.3”.

20.17.1.1. Changes in MySQL Cluster NDB 6.3.16 (5.1.24-ndb-6.3.16) (27 June 2008)

This is a new source release, fixing recently discovered bugs in previous MySQL Cluster NDB 6.3 releases.

Obtaining MySQL Cluster NDB 6.3. MySQL Cluster NDB 6.3 releases are source-only releases which you must compile and install using the instructions found in Section 2.9, “MySQL Installation Using a Source Distribution”, and in Section 20.3.1, “Building MySQL Cluster from Source Code”. You can download the GPL source tarball from the MySQL FTP site at ftp://ftp.mysql.com/pub/mysql/download/cluster_telco/.

This release incorporates all bugfixes and changes made in the previous MySQL Cluster NDB 6.3 release, as well as all bugfixes and feature changes which were added in mainline MySQL 5.1 through MySQL 5.1.24 (see Section C.1.5, “Changes in MySQL 5.1.24 (08 April 2008)”).

Note

Please refer to our bug database at http://bugs.mysql.com/ for more details about the individual bugs fixed in this version.

Functionality added or changed:

MySQL Cluster: Event buffer lag reports are now written to the cluster log. (Bug#37427)
MySQL Cluster: Added the --no-binlog option for ndb_restore. When used, this option prevents information being written to SQL node binary logs from the restoration of a cluster backup. (Bug#30452)

Bugs fixed:

Cluster API: MySQL Cluster: Changing the system time on data nodes could cause MGM API applications to hang and the data nodes to crash. (Bug#35607)
MySQL Cluster: Failure of a data node could sometimes cause mysqld to crash. (Bug#37628)
MySQL Cluster: DELETE ... WHERE unique_index_column=value deleted the wrong row from the table. (Bug#37516)
MySQL Cluster: If subscription was terminated while a node was down, the epoch was not properly acknowledged by that node. (Bug#37442)
MySQL Cluster: libmysqld failed to wait for the cluster binlog thread to terminate before exiting. (Bug#37429)
MySQL Cluster: In rare circumstances, a connection followed by a disconnection could give rise to a “stale” connection where the connection still existed but was not seen by the transporter. (Bug#37338)
MySQL Cluster: Queries against NDBCLUSTER tables were cached only if AUTOCOMMIT was in use. (Bug#36692)
Cluster Replication: Data was written to the binlog with --log-slave-updates disabled. (Bug#37472)
Cluster API: Creating a table on an SQL node, then starting an NDB API application that listened for events from this table, then dropping the table from an SQL node, prevented data node restarts. (Bug#37279)
Cluster API: When some operations succeeded and some failed following a call to NdbTransaction::execute(Commit, AO_IgnoreOnError), a race condition could cause spurious occurrences of NDB API Error 4011 (Internal error). (Bug#37158)
Cluster API: A buffer overrun in NdbBlob::setValue() caused erroneous results on Mac OS X. (Bug#31284)

20.17.1.2. Changes in MySQL Cluster NDB 6.3.15 (5.1.24-ndb-6.3.15) (30 May 2008)

This is a new source release, fixing recently discovered bugs in previous MySQL Cluster NDB 6.3 releases.

Note

Please refer to our bug database at http://bugs.mysql.com/ for more details about the individual bugs fixed in this version.

Bugs fixed:

MySQL Cluster: In certain rare situations, ndb_size.pl could fail with the error Can't use string ("value") as a HASH ref while "strict refs" in use. (Bug#43022)
MySQL Cluster: Under some circumstances, a failed CREATE TABLE could mean that subsequent CREATE TABLE statements caused node failures. (Bug#37092)
MySQL Cluster: A fail attempt to create an NDB table could in some cases lead to resource leaks or cluster failures. (Bug#37072)
MySQL Cluster: Attempting to create a native backup of NDB tables having a large number of NULL columns and data could lead to node failures. (Bug#37039)
MySQL Cluster: Checking of API node connections was not efficiently handled. (Bug#36843)
MySQL Cluster: Attempting to delete a non-existent row from a table containing a TEXT or BLOB column within a transaction caused the transaction to fail. (Bug#36756)
See also Bug#36851
MySQL Cluster: If the combined total of tables and indexes in the cluster was greater than 4096, issuing START BACKUP caused data nodes to fail. (Bug#36044)
MySQL Cluster: Where column values to be compared in a query were of the VARCHAR or VARBINARY types, NDBCLUSTER passed a value padded to the full size of the column, which caused unnecessary data to be sent to the data nodes. This also had the effect of wasting CPU and network bandwidth, and causing condition pushdown to be disabled where it could (and should) otherwise have been applied. (Bug#35393)
MySQL Cluster: When dropping a table failed for any reason (such as when in single user mode) then the corresponding .ndb file was still removed.
Replication: When flushing tables, there were a slight chance that the flush occurred between the processing of one table map event and the next. Since the tables were opened one by one, subsequent locking of tables would cause the slave to crash. This problem was observed when replicating NDBCLUSTER or InnoDB tables, when executing multi-table updates, and when a trigger or a stored routine performed an (additional) insert on a table so that two tables were effectively being inserted into in the same statement. (Bug#36197)
Cluster API: Ordered index scans were not pruned correctly where a partitioning key was specified with an EQ-bound. (Bug#36950)
Cluster API: When an insert operation involving BLOB data was attempted on a row which already existed, no duplicate key error was correctly reported and the transaction is incorrectly aborted. In some cases, the existing row could also become corrupted. (Bug#36851)
See also Bug#26756
Cluster API: NdbApi.hpp depended on ndb_global.h, which was not actually installed, causing the compilation of programs that used NdbApi.hpp to fail. (Bug#35853)

20.17.1.3. Changes in MySQL Cluster NDB 6.3.14 (5.1.24-ndb-6.3.14) (11 May 2008)

This is a new source release, fixing recently discovered bugs in previous MySQL Cluster NDB 6.3 releases.

Note

Please refer to our bug database at http://bugs.mysql.com/ for more details about the individual bugs fixed in this version.

Bugs fixed:

MySQL Cluster: SET GLOBAL ndb_extra_logging caused mysqld to crash. (Bug#36547)
MySQL Cluster: A race condition caused by a failure in epoll handling could cause data nodes to fail. (Bug#36537)
MySQL Cluster: Under certain rare circumstances, the failure of the new master node while attempting a node takeover would cause takeover errors to repeat without being resolved. (Bug#36199, Bug#36246, Bug#36247, Bug#36276)
MySQL Cluster: When more than one SQL node connected to the cluster at the same time, creation of the mysql.ndb_schema table failed on one of them with an explicit Table exists error, which was not necessary. (Bug#35943)
MySQL Cluster: mysqld failed to start after running mysql_upgrade. (Bug#35708)
MySQL Cluster: Notification of a cascading master node failures could sometimes not be transmitted correctly (that is, transmission of the NF_COMPLETEREP signal could fail), leading to transactions hanging and timing out (NDB error 4012), scans hanging, and failure of the management server process. (Bug#32645)
MySQL Cluster: If an API node disconnected and then reconnected during Start Phase 8, then the connection could be “blocked” — that is, the QMGR kernel block failed to detect that the API node was in fact connected to the cluster, causing issues with the NDB Subscription Manager (SUMA).
MySQL Cluster: NDB error 1427 (Api node died, when SUB_START_REQ reached node) was incorrectly classified as a schema error rather than a temporary error.
Cluster Replication: Performing SELECT ... FROM mysql.ndb_apply_status before the mysqld process had connected to the cluster failed, and caused this table never to be created. (Bug#36123)
Cluster API: Accesing the debug version of libndbclient via dlopen() resulted in a segmentation fault. (Bug#35927)
Cluster API: Attempting to pass a nonexistent column name to the equal() and setValue() methods of NdbOperation caused NDB API applications to crash. Now the column name is checked, and an error is returned in the event that the column is not found. (Bug#33747)
Cluster API: Relocation errors were encountered when trying to compile NDB API applications on a number of platforms, including 64-bit Linux. As a result, libmysys, libmystrings, and libdbug have been changed from normal libraries to “noinst” libtool helper libraries. They are no longer installed as separate libraries; instead, all necessary symbols from these are added directly to libndbclient. This means that NDB API programs now need to be linked only using -lndbclient. (Bug#29791)

20.17.1.4. Changes in MySQL Cluster NDB 6.3.13 (5.1.24-ndb-6.3.13) (10 April 2008)

This is a new source release, fixing recently discovered bugs in previous MySQL Cluster NDB 6.3 releases.

Note

Please refer to our bug database at http://bugs.mysql.com/ for more details about the individual bugs fixed in this version.

Functionality added or changed:

The ndbd and ndb_mgmd manpages have been reclassified from volume 1 to volume 8. (Bug#34642)

Bugs fixed:

Important Change: MySQL Cluster: mysqld_safe now traps Signal 13 (SIGPIPE) so that this signal no longer kills the MySQL server process. (Bug#33984)
MySQL Cluster: Node or system restarts could fail due an unitialized variable in the DTUP kernel block. This issue was found in MySQL Cluster NDB 6.3.11. (Bug#35797)
MySQL Cluster: If an error occured while executing a statement involving a BLOB or TEXT column of an NDB table, a memory leak could result. (Bug#35593)
MySQL Cluster: It was not possible to determine the value used for the --ndb-cluster-connection-pool option in the mysql client. Now this value is reported as a system status variable. (Bug#35573)
MySQL Cluster: The ndb_waiter utility wrongly calculated timeouts. (Bug#35435)
MySQL Cluster: A SELECT on a table with a non-indexed, large VARCHAR column which resulted in condition pushdown on this column could cause mysqld to crash. (Bug#35413)
MySQL Cluster: ndb_restore incorrectly handled some datatypes when applying log files from backups. (Bug#35343)
MySQL Cluster: In some circumstances, a stopped data node was handled incorrectly, leading to redo log space being exhausted following an initial restart of the node, or an initial or partial restart of the cluster (the wrong CGI might be used in such cases). This could happen, for example, when a node was stopped following the creation of a new table, but before a new LCP could be executed. (Bug#35241)
MySQL Cluster: SELECT ... LIKE ... queries yielded incorrect results when used on NDB tables. As part of this fix, condition pushdown of such queries has been disabled; re-enabling it is expected to be done as part of a later, permanent fix for this issue. (Bug#35185)
MySQL Cluster: ndb_mgmd reported errors to STDOUT rather than to STDERR. (Bug#35169)
MySQL Cluster: Nested multi-range reads failed when the second multi-range read released the first read's unprocessed operations, sometimes leading to a SQL node crash. (Bug#35137)
MySQL Cluster: In some situations, a problem with synchronizing checkpoints between nodes could cause a system restart or a node restart to fail with Error 630 during restore of TX. (Bug#34756)
MySQL Cluster: A node failure during an initial node restart followed by another node start could cause the master data node to fail, because it incorrectly gave the node permission to start even if the invalidated node's LCP was still running. (Bug#34702)
MySQL Cluster: When a secondary index on a DECIMAL column was used to retrieve data from an NDB table, no results were returned even if the target table had a matched value in the column that was defined with the secondary index. (Bug#34515)
MySQL Cluster: An UPDATE on an NDB table that set a new value for a unique key column could cause subsequent queries to fail. (Bug#34208)
MySQL Cluster: If a data node in one node group was placed in the “not started” state (using node_id RESTART -n), it was not possible to stop a data node in a different node group. (Bug#34201)
MySQL Cluster: Numerous NDBCLUSTER test failures occurred in builds compiled using icc on IA64 platforms. (Bug#31239)
MySQL Cluster: If a START BACKUP command was issued while ndb_restore was running, the backup being restored could be overwritten. (Bug#26498)
MySQL Cluster: REPLACE statements did not work correctly with NDBCLUSTER tables when all columns were not explicitly listed. (Bug#22045)
MySQL Cluster: CREATE TABLE and ALTER TABLE statements using ENGINE=NDB or ENGINE=NDBCLUSTER caused mysqld to fail on Solaris 10 for x86 platforms. (Bug#19911)
Replication: A CHANGE MASTER statement with no MASTER_HEARTBEAT_PERIOD option failed to reset the heartbeat period to its default value. (Bug#34686)
Cluster Replication: In some cases, when updating only one or some columns in a table, the complete row was written to the binary log instead of only the updated column or columns, even when ndb_log_updated_only was set to 1. (Bug#35208)
Cluster Replication: The --ndb-wait-connected option caused the server to wait for a partial connection plus an additional 3 seconds for a complete connection to the cluster. This could lead to issues with setting up the binary log. (Bug#34757)
Cluster API: Closing a scan before it was executed caused the application to segfault. (Bug#36375)
Cluster API: Using NDB API applications from older MySQL Cluster versions with libndbclient from newer ones caused the cluster to fail. (Bug#36124)
Cluster API: Some ordered index scans could return tuples out of order. (Bug#35908)
Cluster API: Scans having no bounds set were handled incorrectly. (Bug#35876)
Cluster API: NdbScanFilter::getNdbOperation(), which was inadvertently removed in MySQL Cluster NDB 6.3.11, has been restored. (Bug#35854)
Enabling the read_only system variable while autocommit mode was enabled caused SELECT statements for transactional storage engines to fail. (Bug#35732)
Executing a FLUSH PRIVILEGES statement after creating a temporary table in the mysql database with the same name as one of the MySQL system tables caused the server to crash.
Note
While it is possible to shadow a system table in this way, the temporary table exists only for the current user and connection, and does not effect any user privileges.
(Bug#33275)

20.17.1.5. Changes in MySQL Cluster NDB 6.3.12 (5.1.23-ndb-6.3.12) (05 April 2008)

MySQL Cluster NDB 6.3.12 was pulled due to issues discovered shortly after its release, and is no longer available. Users of MySQL Cluster NDB 6.3.10 and earlier MySQL Cluster NDB 6.3 releases should upgrade to MySQL Cluster NDB 6.3.13 or later.

For information about bugfixes and feature enhancements that were originally scheduled to appear for the first time in this release, see Section 20.17.1.4, “Changes in MySQL Cluster NDB 6.3.13 (5.1.24-ndb-6.3.13) (10 April 2008)”.

20.17.1.6. Changes in MySQL Cluster NDB 6.3.11 (5.1.23-ndb-6.3.11) (28 March 2008)

This release was pulled due to issues discovered shortly after its release, and is no longer available. Users of MySQL Cluster NDB 6.3.10 and earlier MySQL Cluster NDB 6.3 releases should upgrade to MySQL Cluster NDB 6.3.13 or later.

20.17.1.7. Changes in MySQL Cluster NDB 6.3.10 (5.1.23-ndb-6.3.10) (17 February 2008)

This is a new Beta development release, fixing recently discovered bugs in previous MySQL Cluster NDB 6.3 releases.

This Beta release incorporates all bugfixes and changes made in the previous MySQL Cluster NDB 6.3 release, as well as all bugfixes and feature changes which were added in mainline MySQL 5.1 through MySQL 5.1.23 (see Section C.1.6, “Changes in MySQL 5.1.23 (29 January 2008)”).

Note

Please refer to our bug database at http://bugs.mysql.com/ for more details about the individual bugs fixed in this version.

Bugs fixed:

MySQL Cluster: Due to the reduction of the number of local checkpoints from 3 to 2 in MySQL Cluster NDB 6.3.8, a data node using ndbd from MySQL Cluster NDB 6.3.8 or later started using a filesystem from an earlier version could incorrectly invalidate local checkpoints too early during the startup process, causing the node to fail. (Bug#34596)

20.17.1.8. Changes in MySQL Cluster NDB 6.3.9 (5.1.23-ndb-6.3.9) (12 February 2008)

This is a new Beta development release, fixing recently discovered bugs in previous MySQL Cluster NDB 6.3 releases.

Note

Please refer to our bug database at http://bugs.mysql.com/ for more details about the individual bugs fixed in this version.

Functionality added or changed:

Beginning with this version, MySQL Cluster NDB 6.3.x releases once again include the InnoDB storage engine. In order to enable InnoDB, you must configure the build using --with-innodb.

Bugs fixed:

MySQL Cluster: Cluster failures could sometimes occur when performing more than three parallel takeovers during node restarts or system restarts. This affected MySQL Cluster NDB 6.3.x releases only. (Bug#34445)
MySQL Cluster: Upgrades of a cluster using while a DataMemory setting in excess of 16 GB caused data nodes to fail. (Bug#34378)
MySQL Cluster: Performing many SQL statements on NDB tables while in AUTOCOMMIT mode caused a memory leak in mysqld. (Bug#34275)
MySQL Cluster: In certain rare circumstances, a race condition could occur between an aborted insert and a delete leading a data node crash. (Bug#34260)
MySQL Cluster: Multi-table updates using ordered indexes during handling of node failures could cause other data nodes to fail. (Bug#34216)
MySQL Cluster: When configured with NDB support, MySQL failed to compile using gcc 4.3 on 64bit FreeBSD systems. (Bug#34169)
MySQL Cluster: The failure of a DDL statement could sometimes lead to node failures when attempting to execute subsequent DDL statements. (Bug#34160)
MySQL Cluster: Extremely long SELECT statements (where the text of the statement was in excess of 50000 characters) against NDB tables returned empty results. (Bug#34107)
MySQL Cluster: When configured with NDB support, MySQL failed to compile on 64bit FreeBSD systems. (Bug#34046)
MySQL Cluster: Statements executing multiple inserts performed poorly on NDB tables having AUTO_INCREMENT columns. (Bug#33534)
MySQL Cluster: The ndb_waiter utility polled ndb_mgmd excessively when obtaining the status of cluster data nodes. (Bug#32025)
See also Bug#32023
MySQL Cluster: Transaction atomicity was sometimes not preserved between reads and inserts under high loads. (Bug#31477)
MySQL Cluster: Having tables with a great many columns could cause Cluster backups to fail. (Bug#30172)
Cluster Replication: Disk Data: Statements violating unique keys on Disk Data tables (such as attempting to insert NULL into a NOT NULL column) could cause data nodes to fail. When the statement was executed from the binlog, this could also result in failure of the slave cluster. (Bug#34118)
Disk Data: Updating in-memory columns of one or more rows of Disk Data table, followed by deletion of these rows and re-insertion of them, caused data node failures. (Bug#33619)
Cluster Replication: Setting --replicate-ignore-db=mysql caused the mysql.ndb_apply_status table not to be replicated, breaking Cluster Replication. (Bug#28170)

20.17.1.9. Changes in MySQL Cluster NDB 6.3.8 (5.1.23-ndb-6.3.8) (29 January 2008)

This is a new Beta development release, fixing recently discovered bugs in previous MySQL Cluster NDB 6.3 releases.

Note

Please refer to our bug database at http://bugs.mysql.com/ for more details about the individual bugs fixed in this version.

Functionality added or changed:

Cluster API: Important Change: Because NDB_LE_MemoryUsage.page_size_kb shows memory page sizes in bytes rather than kilobytes, it has been renamed to page_size_bytes. The name page_size_kb is now deprecated and thus subject to removal in a future release, although it currently remains supported for reasons of backward compatibility. See The Ndb_logevent_type Type, for more information about NDB_LE_MemoryUsage. (Bug#30271)
MySQL Cluster: ndb_restore now supports basic attribute promotion; that is, data from a column of a given type can be restored to a column using a “larger” type. For example, Cluster backup data taken from a SMALLINT column can be restored to a MEDIUMINT, INT, or BIGINT column.
For more information, see Section 20.9.3, “ndb_restore — Restore a Cluster Backup”.
MySQL Cluster: Now only 2 local checkpoints are stored, rather than 3 as in previous MySQL Cluster versions. This lowers disk space requirements and reduces the size and number of redo log files needed.
MySQL Cluster: The mysqld option --ndb-batch-size has been added. This allows for controlling the size of batches used for running transactions.
MySQL Cluster: Node recovery can now be done in parallel, rather than sequentially, which can result in much faster recovery times.
MySQL Cluster: Persistence of NDB tables can now be controlled using the session variables ndb_table_temporary and ndb_table_no_logging. ndb_table_no_logging causes NDB tables not to be checkpointed to disk; ndb_table_temporary does the same, and in addition, no schema files are created.
MySQL Cluster: OPTIMIZE TABLE can now be interrupted. This can be done, for example, by killing the SQL thread performing the OPTIMIZE operation.

Bugs fixed:

Disk Data: Important Change: It is no longer possible on 32-bit systems to issue statements appearing to create Disk Data log files or data files greater than 4 GB in size. (Trying to create log files or data files larger than 4 GB on 32-bit systems led to unrecoverable data node failures; such statements now fail with NDB error 1515.) (Bug#29186)
Replication: MySQL Cluster: The code implementing heartbeats did not check for possible errors in some circumstances; this kept the dump thread hanging while waiting for heartbeats loop even though the slave was no longer connected. (Bug#33332)
MySQL Cluster: High numbers of insert operations, delete operations, or both could cause NDB error 899 (Rowid already allocated) to occur unnecessarily. (Bug#34033)
MySQL Cluster: A periodic failure to flush the send buffer by the NDB TCP transporter could cause a unnecessary delay of 10 ms between operations. (Bug#34005)
MySQL Cluster: DROP TABLE did not free all data memory. This bug was observed in MySQL Cluster NDB 6.3.7 only. (Bug#33802)
MySQL Cluster: A race condition could occur (very rarely) when the release of a GCI was followed by a data node failure. (Bug#33793)
MySQL Cluster: Some tuple scans caused the wrong memory page to be accessed, leading to invalid results. This issue could affect both in-memory and Disk Data tables. (Bug#33739)
MySQL Cluster: A failure to initialize an internal variable led to sporadic crashes during cluster testing. (Bug#33715)
MySQL Cluster: The server failed to reject properly the creation of an NDB table having an unindexed AUTO_INCREMENT column. (Bug#30417)
MySQL Cluster: Issuing an INSERT ... ON DUPLICATE KEY UPDATE concurrently with or following a TRUNCATE statement on an NDB table failed with NDB error 4350 Transaction already aborted. (Bug#29851)
MySQL Cluster: The Cluster backup process could not detect when there was no more disk space and instead continued to run until killed manually. Now the backup fails with an appropriate error when disk space is exhausted. (Bug#28647)
MySQL Cluster: It was possible in config.ini to define cluster nodes having node IDs greater than the maximum allowed value. (Bug#28298)
MySQL Cluster: Under some circumstances, a recovering data node did not use its own data, instead copying data from another node even when this was not required. This in effect bypassed the optimized node recovery protocol and caused recovery times to be unnecessarily long. (Bug#26913)
Cluster Replication: Consecutive DDL statements involving tables (CREATE TABLE, ALTER TABLE, and DROP TABLE) could be executed so quickly that previous DDL statements upon which they depended were not yet written in the binary log.
For example, if DROP TABLE foo was issued immediately following CREATE TABLE foo, the DROP statement could fail because the CREATE had not yet been recorded. (Bug#34006)
Cluster Replication: ndb_restore -e restored excessively large values to the ndb_apply_status table's epoch column when restoring to a MySQL Cluster version supporting Micro-GCPs from an older version that did not support these.
A workaround when restoring to MySQL Cluster releases supporting micro-GCPs previous to MySQL Cluster NDB 6.3.8 is to perform a 32-bit shift on the epoch column values to reduce them to their proper size. (Bug#33406)
Cluster API: Transactions containing inserts or reads would hang during NdbTransaction::execute() calls made from NDB API applications built against a MySQL Cluster version that did not support micro-GCPs accessing a later version that supported micro-GCPs. This issue was observed while upgrading from MySQL Cluster NDB 6.1.23 to MySQL Cluster NDB 6.2.10 when the API application built against the earlier version attempted to access a data node already running the later version, even after disabling micro-GCPs by setting TimeBetweenEpochs equal to 0. (Bug#33895)
Cluster API: When reading a BIT(64) value using NdbOperation:getValue(), 12 bytes were written to the buffer rather than the expected 8 bytes. (Bug#33750)

20.17.1.10. Changes in MySQL Cluster NDB 6.3.7 (5.1.23-ndb-6.3.7) (19 December 2007)

This is a new Beta development release, fixing recently discovered bugs in previous MySQL Cluster NDB 6.3 releases.

Note

Please refer to our bug database at http://bugs.mysql.com/ for more details about the individual bugs fixed in this version.

Functionality added or changed:

MySQL Cluster: Compressed local checkpoints and backups are now supported, resulting in a space savings of 50% or more over uncompressed LCPs and backups. Cmpression of these can be enabled in the config.ini file using the two new data node configuration parameters CompressedLCP and CompressedBackup, respectively.
MySQL Cluster: OPTIMIZE TABLE is now supported for NDB tables, subject to the following limitations:
- Only in-memory tables are supported. OPTIMIZE still has no effect on Disk Data tables.
- Only variable-length columns are supported. However, you can force columns defined using fixed-length data types to be dynamic using the ROW_FORMAT or COLUMN_FORMAT option with a CREATE TABLE or ALTER TABLE statement.
Memory reclaimed from an NDB table using OPTIMIZE is generally available to the cluster, and not confined to the table from which it was recovered, unlike the case with memory freed using DELETE.
The performance of OPTIMIZE on NDB tables can be regulated by adjusting the value of the ndb_optimization_delay system variable.
MySQL Cluster: It is now possible to cause statements occurring within the same transaction to be run as a batch by setting the session variable transaction_allow_batching to 1 or ON.
Note
To use this feature, autocommit must be disabled.

Bugs fixed:

Partitioning: MySQL Cluster: When partition pruning on an NDB table resulted in an ordered index scan spanning only one partition, any descending flag for the scan was wrongly discarded, causing ORDER BY DESC to be treated as ORDER BY ASC, MAX() to be handled incorrectly, and similar problems. (Bug#33061)
MySQL Cluster: When all data and SQL nodes in the cluster were shut down abnormally (that is, other than by using STOP in the cluster management client), ndb_mgm used excessive amounts of CPU. (Bug#33237)
MySQL Cluster: When using micro-GCPs, if a node failed while preparing for a global checkpoint, the master node would use the wrong GCI. (Bug#32922)
MySQL Cluster: Under some conditions, performing an ALTER TABLE on an NDBCLUSTER table failed with a Table is full error, even when only 25% of DataMemory was in use and the result should have been a table using less memory (for example, changing a VARCHAR(100) column to VARCHAR(80)). (Bug#32670)
Cluster Replication: Creating the mysql.ndb_replication table with the wrong number of columns for the primary key caused mysqld to crash. Now a CREATE TABLE [mysql.]ndb_replication statement that is invalid for this reason fails with the error Bad schema for mysql.ndb_replication table. Message: Wrong number of primary keys, expected number. (Bug#33159)
Cluster Replication: Where a table being replicated had a TEXT or BLOB column, an UPDATE on the master that did not refer explicitly to this column in the WHERE clause stopped the SQL thread on the slave with Error in Write_rows event: row application failed. Got error 4288 'Blob handle for column not available' from NDBCLUSTER. (Bug#30674)

20.17.1.11. Changes in MySQL Cluster NDB 6.3.6 (5.1.22-ndb-6.3.6) (08 November 2007)

This is a new Beta development release, fixing recently discovered bugs in previous MySQL Cluster NDB 6.3 releases.

This Beta release incorporates all bugfixes and changes made in the previous MySQL Cluster NDB 6.3 release, as well as all bugfixes and feature changes which were added in mainline MySQL 5.1 through MySQL 5.1.22 (see Section C.1.7, “Changes in MySQL 5.1.22 (24 September 2007: Release Candidate)”).

Note

Please refer to our bug database at http://bugs.mysql.com/ for more details about the individual bugs fixed in this version.

Functionality added or changed:

Important Note: MySQL Cluster NDB 6.2 and 6.3 source archives are now available in separate commercial and GPL versions. Due to licensing concerns, previous MySQL Cluster NDB 6.2 and 6.3 source archives were removed from the FTP site.
MySQL Cluster: Unnecessary reads when performing a primary key or unique key update have been reduced, and in some cases, eliminated. (It is almost never necessary to read a record prior to an update, the lone exception to this being when a primary key is updated, since this requires a delete followed by an insert, which must be prepared by reading the record.) Depending on the number of primary key and unique key lookups that are performed per transaction, this can yield a considerable improvement in performance.
MySQL Cluster: Batched operations are now better supported for DELETE and UPDATE. (UPDATE WHERE... and muliple DELETE.)
MySQL Cluster: Introduced the Ndb_execute_count status variable, which measures the number of round trips made by queries to the NDB kernel.

Bugs fixed:

MySQL Cluster: An insert or update with combined range and equality constraints failed when run against an NDB table with the error Got unknown error from NDB. An example of such a statement would be UPDATE t1 SET b = 5 WHERE a IN (7,8) OR a >= 10;. (Bug#31874)
MySQL Cluster: An error with an if statement in sql/ha_ndbcluster.cc could potentially lead to an infinite loop in case of failure when working with AUTO_INCREMENT columns in NDB tables. (Bug#31810)
MySQL Cluster: The NDB storage engine code was not safe for strict-alias optimization in gcc 4.2.1. (Bug#31761)
MySQL Cluster: ndb_restore displayed incorrect backup file version information. This meant (for example) that, when attempting to restore a backup made from a MySQL 5.1.22 cluster to a MySQL Cluster NDB 6.3.3 cluster, the restore process failed with the error Restore program older than backup version. Not supported. Use new restore program. (Bug#31723)
MySQL Cluster: Following an upgrade, ndb_mgmd would fail with an ArbitrationError. (Bug#31690)
MySQL Cluster: The NDB management client command node_id REPORT MEMORY provided no output when node_id was the node ID of a management or API node. Now, when this occurs, the management client responds with Node node_id: is not a data node. (Bug#29485)
MySQL Cluster: Performing DELETE operations after a data node had been shut down could lead to inconsistent data following a restart of the node. (Bug#26450)
MySQL Cluster: UPDATE IGNORE could sometimes fail on NDB tables due to the use of unitialized data when checking for duplicate keys to be ignored. (Bug#25817)
Cluster Replication: Updates performed unnecessary writes to the primary keys of the rows being updated. (Bug#31841)
Cluster Replication: Slave batching did not work correctly with UPDATE statements. (Bug#31787)
Cluster Replication: A node failure during replication could lead to buckets out of order; now active subscribers are checked for, rather than empty buckets. (Bug#31701)
Cluster Replication: When the master mysqld crashed or was restarted, no LOST_EVENTS entry was made in the binlog. (Bug#31484)
See also Bug#21494

20.17.1.12. Changes in MySQL Cluster NDB 6.3.5 (5.1.22-ndb-6.3.5) (17 October 2007)

This is a new Beta development release, fixing recently discovered bugs in previous MySQL Cluster NDB 6.3 releases.

This Beta release incorporates all bugfixes and changes made in the previous MySQL Cluster NDB 6.3 release, as well as all bugfixes and feature changes which were added in mainline MySQL 5.1 through MySQL 5.1.22 (see Section C.1.7, “Changes in MySQL 5.1.22 (24 September 2007: Release Candidate)”).

Note

Please refer to our bug database at http://bugs.mysql.com/ for more details about the individual bugs fixed in this version.

Bugs fixed:

MySQL Cluster: A query against a table with TEXT or BLOB columns that would return more than a certain amount of data failed with Got error 4350 'Transaction already aborted' from NDBCLUSTER. (Bug#31482)
This regression was introduced by Bug#29102
Cluster Replication: In some cases, not all tables were properly initialized before the binlog thread was started. (Bug#31618)

20.17.1.13. Changes in MySQL Cluster NDB 6.3.4 (5.1.22-ndb-6.3.4) (15 October 2007)

This is a new Beta development release, fixing recently discovered bugs in previous MySQL Cluster NDB 6.3 releases.

This Beta release incorporates all bugfixes and changes made in the previous MySQL Cluster NDB 6.3 release, as well as all bugfixes and feature changes which were added in mainline MySQL 5.1 through MySQL 5.1.22 (see Section C.1.7, “Changes in MySQL 5.1.22 (24 September 2007: Release Candidate)”).

Note

Please refer to our bug database at http://bugs.mysql.com/ for more details about the individual bugs fixed in this version.

Functionality added or changed:

MySQL Cluster: The --ndb_optimized_node_selection startup option for mysqld now allows a wider range of values and corresponding behaviors for SQL nodes when selecting a transaction coordinator. See Section 5.1.3, “System Variables”, for more information.
Cluster Replication: Replication: A replication heartbeat mechanism has been added to facilitate monitoring. This provides an alternative to checking log files, making it possible to detect in real time when a slave has failed.
Configuration of heartbeats is done via a new MASTER_HEARTBEAT_PERIOD = interval clause for the CHANGE MASTER TO statement (see Section 12.6.2.1, “CHANGE MASTER TO Syntax”); monitoring can be done by checking the values of the status variables Slave_heartbeat_period and Slave_received_heartbeats (see Section 5.1.6, “Status Variables”).
The addition of replication heartbeats addresses a number of issues:
- Relay logs were rotated every slave_net_timeout seconds even if no statements were being replicated.
- SHOW SLAVE STATUS displayed an incorrect value for seconds_behind_master following a FLUSH LOGS statement.
- Replication master-slave connections used slave_net_timeout for connection timeouts.
(Bug#20435, Bug#29309, Bug#30932)
Cluster Replication: A new configuration parameter TimeBetweenEpochsTimeout allows a timeout to be set for time between epochs. For more information, see Section 20.3.4.5, “Defining Data Nodes”. (Bug#31276)
Cluster Replication: Support for a new conflict resolution function NDB$OLD() has been added for handling simultaneous updates in multi-master and circular replication setups. A new status variable Ndb_conflict_fn_old tracks the number of times that updates are prevented from being applied due to this type of conflict resolution. See Section 20.11.10, “MySQL Cluster Replication Conflict Resolution”, for more information.
On MySQL replication slaves having multiple network interfaces, it is now possible to set which interface to use for connecting to the master. This can be done by using either the mysqld startup option --master-bind or the MASTER_BIND='interface' clause in a CHANGE MASTER statement. (Bug#25939)
Additional checks were implemented to catch unsupported online ALTER TABLE operations. Currently it is not possible to reorder columns or to change the storage engine used for a table via online ALTER TABLE.
Some redundant checks made during online creation of indexes were removed.
A --bind-address option has been added to a number of MySQL client programs: mysql, mysqldump, mysqladmin, mysqlbinlog, mysqlcheck, mysqlimport, and mysqlshow. This is for use on a computer having multiple network interfaces, and allows you to choose which interface is used to connect to the MySQL server.

Bugs fixed:

MySQL Cluster: It was possible in some cases for a node group to be “lost” due to missed local checkpoints following a system restart. (Bug#31525)
MySQL Cluster: NDB tables having names containing non-alphanumeric characters (such as “ $ ”) were not discovered correctly. (Bug#31470)
MySQL Cluster: A node failure during a local checkpoint could lead to a subsequent failure of the cluster during a system restart. (Bug#31257)
MySQL Cluster: A cluster restart could sometimes fail due to an issue with table IDs. (Bug#30975)
MySQL Cluster: Transaction timeouts were not handled well in some circumstances, leading to excessive number of transactions being aborted unnecessarily. (Bug#30379)
MySQL Cluster: In some cases, the cluster managment server logged entries multiple times following a restart of mgmd. (Bug#29565)
MySQL Cluster: ndb_mgm --help did not display any information about the -a option. (Bug#29509)
MySQL Cluster: An interpreted program of sufficient size and complexity could cause all cluster data nodes to shut down due to buffer overruns. (Bug#29390)
MySQL Cluster: The cluster log was formatted inconsistently and contained extraneous newline characters. (Bug#25064)
A transaction was not aborted following the failure of statement. (Bug#31320)
Online ALTER operations involving a column whose data type has an implicit default value left behind temporary .FRM files, causing subsequent DROP DATABASE statements to fail. (Bug#31097)
Errors could sometimes occur during an online ADD COLUMN under load. (Bug#31082)
Transactions were committed prematurely when LOCK TABLE and SET AUTOCOMMIT=OFF were used together. (Bug#30996)
The mysqld_safe script contained a syntax error. (Bug#30624)

20.17.1.14. Changes in MySQL Cluster NDB 6.3.3 (5.1.22-ndb-6.3.3) (20 September 2007)

This is a new Beta development release, fixing recently discovered bugs in previous MySQL Cluster NDB 6.3 releases.

This Beta release incorporates all bugfixes and changes made in the previous MySQL Cluster NDB 6.3 release, as well as all bugfixes and feature changes which were added in mainline MySQL 5.1 through MySQL 5.1.22 (see Section C.1.7, “Changes in MySQL 5.1.22 (24 September 2007: Release Candidate)”).

Note

Please refer to our bug database at http://bugs.mysql.com/ for more details about the individual bugs fixed in this version.

Functionality added or changed:

MySQL Cluster: Mapping of NDB error codes to MySQL storage engine error codes has been improved. (Bug#28423)
Cluster Replication: A server status variable ndb_conflict_fn_max now provides a count of the number of times that conflict resolution for MySQL Cluster Replication has been applied.
See Section 20.11.10, “MySQL Cluster Replication Conflict Resolution”, for more information.

Bugs fixed:

Partitioning: MySQL Cluster: EXPLAIN PARTITIONS reported partition usage by queries on NDB tables according to the standard MySQL hash function than the hash function used in the NDB storage engine. (Bug#29550)
MySQL Cluster: Attempting to restore a backup made on a cluster host using one endian to a machine using the other endian could cause the cluster to fail. (Bug#29674)
MySQL Cluster: The description of the --print option provided in the output from ndb_restore --help was incorrect. (Bug#27683)
MySQL Cluster: Restoring a backup made on a cluster host using one endian to a machine using the other endian failed for BLOB and DATETIME columns. (Bug#27543, Bug#30024)
Errors could sometimes occur during an online ADD COLUMN under load. (Bug#31082)

20.17.1.15. Changes in MySQL Cluster NDB 6.3.2 (5.1.22-ndb-6.3.2) (07 September 2007)

This is a new Beta development release, fixing recently discovered bugs in previous MySQL Cluster NDB 6.3 releases.

This Beta release incorporates all bugfixes and changes made in the previous MySQL Cluster NDB 6.3 release, as well as all bugfixes and feature changes which were added in mainline MySQL 5.1 through MySQL 5.1.22 (see Section C.1.7, “Changes in MySQL 5.1.22 (24 September 2007: Release Candidate)”).

Note

Please refer to our bug database at http://bugs.mysql.com/ for more details about the individual bugs fixed in this version.

Functionality added or changed:

MySQL Cluster: Online ADD COLUMN, ADD INDEX, and DROP INDEX operations can now be performed explicitly for NDB tables, as well as online renaming of tables and columns for NDB and MyISAM tables — that is, without copying or locking of the affected tables — using ALTER ONLINE TABLE.
Indexes can also be created and dropped online using CREATE INDEX and DROP INDEX, respectively, using the ONLINE keyword.
You can force operations that would otherwise be performed online to be done offline using the OFFLINE keyword.
See Section 12.1.4, “ALTER TABLE Syntax”, Section 12.1.7, “CREATE INDEX Syntax”, and Section 12.1.13, “DROP INDEX Syntax”, for more information.
MySQL Cluster: It is now possible to control whether fixed-width or variable-width storage is used for a given column of an NDB table by means of the COLUMN_FORMAT specifier as part of the column's definition in a CREATE TABLE or ALTER TABLE statement.
It is also possible to control whether a given column of an NDB table is stored in memory or on disk, using the STORAGE specifier as part of the column's definition in a CREATE TABLE or ALTER TABLE statement.
For permitted values and other information about COLUMN_FORMAT and STORAGE, see Section 12.1.10, “CREATE TABLE Syntax”.
MySQL Cluster: A new cluster management server startup option --bind-address makes it possible to restrict management client connections to ndb_mgmd to a single host and port. For more information, see Section 20.6.5.2, “Command Options for ndb_mgmd”.
Cluster Replication: Multi-way replication failover and recovery for NDB is facilitated with the introduction of the --ndb-log-orig option. When mysqld is started with this option, the originating server ID and epoch of each binlog event is recorded in the mysql.ndb_binlog_index table, which now contains two additional columns orig_server_id and orig_epoch for storing this information. In such cases, a single epoch on a slave may be represented by multiple rows in the slave's ndb_binlog_index table, one for each epoch as it originated from a master.
Cluster Replication: The protocol for handling global checkpoints has been changed. It is now possible to control how often the GCI number is updated, and how often global checkpoints are written to disk, using the TimeBetweenEpochs configuration parameter. This improves the reliability and performance of MySQL Cluster Replication.
GCPs handled using the new protocol are sometimes referred to as “micro-GCPs”.
For more information, see TimeBetweenEpochs.

Bugs fixed:

MySQL Cluster: When an NDB event was left behind but the corresponding table was later recreated and received a new table ID, the event could not be dropped. (Bug#30877)
MySQL Cluster: An insufficiently descriptive and potentially misleading Error 4006 (Connect failure - out of connection objects...) was produced when either of the following two conditions occurred:
1. There were no more transaction records in the transaction coordinator
2. an Ndb object in the NDB API was initialized with insufficient parallellism
Separate error messages are now generated for each of these two cases. (Bug#11313)
Cluster API: An Ndb object in the NDB API was initialized with insufficient parallellism.
For micro-GCPs, fixed the assignment of “fake” CGI events so that they do not cause buckets to be sent out of order. Now, when assigning a GCI to a non-GCI event (that is, creating a pseudo-GCI or “fake” CGI), the GCI that is to arrive is always initiated, even if no known GCI exists, which could occur in the event of a node failure. (Bug#30884)
There were no more transaction records in the transaction coordinator

20.17.1.16. Changes in MySQL Cluster NDB 6.3.1 (5.1.19-ndb-6.3.1) (04 July 2007)

This is a new Beta development release, fixing recently discovered bugs in the previous MySQL Cluster NDB 6.3 release.

This Beta release incorporates all bugfixes and changes made in the previous MySQL Cluster NDB 6.3 release, as well as all bugfixes and feature changes which were added in mainline MySQL 5.1 through MySQL 5.1.19 (see Section C.1.10, “Changes in MySQL 5.1.19 (25 May 2007)”).

Note

Please refer to our bug database at http://bugs.mysql.com/ for more details about the individual bugs fixed in this version.

Bugs fixed:

Batching of transactions was not handled correctly in some cases. (Bug#29525)

20.17.1.17. Changes in MySQL Cluster NDB 6.3.0 (5.1.19-ndb-6.3.0) (02 July 2007)

This is the first development release for MySQL Cluster NDB 6.3, based on version 6.3 of the NDBCLUSTER storage engine.

This Beta release incorporates all bugfixes and feature changes made in previous MySQL Cluster releases, as well as all bugfixes and feature changes which were added in mainline MySQL 5.1 through MySQL 5.1.19 (see Section C.1.10, “Changes in MySQL 5.1.19 (25 May 2007)”).

Note

Please refer to our bug database at http://bugs.mysql.com/ for more details about the individual bugs fixed in this version.

Functionality added or changed:

MySQL Cluster: Reporting functionality has been significantly enhanced in this release:
- A new configuration parameter BackupReportFrequency now makes it possible to cause the management client to provide status reports at regular intervals as well as for such reports to be written to the cluster log (depending on cluster event logging levels). See Section 20.3.4.5, “Defining Data Nodes”, for more information about this parameter.
- A new REPORT command has been added in the cluster management client. REPORT BackupStatus allows you to obtain a backup status report at any time during a backup. REPORT MemoryUsage reports the current data memory and index memory used by each data node. For more about the REPORT command, see Section 20.7.2, “Commands in the MySQL Cluster Management Client”.
- ndb_restore now provides running reports of its progress when restoring a backup. In addition, a complete report status report on the backup is written to the cluster log.
MySQL Cluster: A new configuration parameter ODirect causes NDB to attempt using O_DIRECT writes for LCP, backups, and redo logs, often lowering CPU usage.
Cluster Replication: This release implements conflict resolution, which makes it possible to determine on a per-table basis whether or not an update to a given row on the master should be applied on the slave. For more information, see Section 20.11.10, “MySQL Cluster Replication Conflict Resolution”.

20.17.2. Changes in MySQL Cluster NDB 6.2

20.17.2.1. Changes in MySQL Cluster NDB 6.2.16 (5.1.24-ndb-6.2.16) (Not yet released)
20.17.2.2. Changes in MySQL Cluster NDB 6.2.15 (5.1.23-ndb-6.2.15) (22 May 2008)
20.17.2.3. Changes in MySQL Cluster NDB 6.2.14 (5.1.23-ndb-6.2.14) (05 March 2008)
20.17.2.4. Changes in MySQL Cluster NDB 6.2.13 (5.1.23-ndb-6.2.13) (22 February 2008)
20.17.2.5. Changes in MySQL Cluster NDB 6.2.12 (5.1.23-ndb-6.2.12) (12 February 2008)
20.17.2.6. Changes in MySQL Cluster NDB 6.2.11 (5.1.23-ndb-6.2.11) (28 January 2008)
20.17.2.7. Changes in MySQL Cluster NDB 6.2.10 (5.1.23-ndb-6.2.10) (19 December 2007)
20.17.2.8. Changes in MySQL Cluster NDB 6.2.9 (5.1.22-ndb-6.2.9) (22 November 2007)
20.17.2.9. Changes in MySQL Cluster NDB 6.2.8 (5.1.22-ndb-6.2.8) (08 November 2007)
20.17.2.10. Changes in MySQL Cluster NDB 6.2.7 (5.1.22-ndb-6.2.7) (10 October 2008)
20.17.2.11. Changes in MySQL Cluster NDB 6.2.6 (5.1.22-ndb-6.2.6) (20 September 2007)
20.17.2.12. Changes in MySQL Cluster NDB 6.2.5 (5.1.22-ndb-6.2.5) (06 September 2007)
20.17.2.13. Changes in MySQL Cluster NDB 6.2.4 (5.1.19-ndb-6.2.4) (04 July 2007)
20.17.2.14. Changes in MySQL Cluster NDB 6.2.3 (5.1.19-ndb-6.2.3) (02 July 2007)
20.17.2.15. Changes in MySQL Cluster NDB 6.2.2 (5.1.18-ndb-6.2.2) (07 May 2007)
20.17.2.16. Changes in MySQL Cluster NDB 6.2.1 (5.1.18-ndb-6.2.1) (30 April 2007)
20.17.2.17. Changes in MySQL Cluster NDB 6.2.0 (5.1.16-ndb-6.2.0) (03 March 2007)

This section contains change history information for MySQL Cluster releases based on version 6.2 of the NDBCLUSTER storage engine.

For an overview of new features added in MySQL Cluster NDB 6.2, see Section 20.15.3, “Features Added in MySQL Cluster NDB 6.2”.

20.17.2.1. Changes in MySQL Cluster NDB 6.2.16 (5.1.24-ndb-6.2.16) (Not yet released)

This is a bugfix release, fixing recently discovered bugs in the previous MySQL Cluster NDB 6.2 release.

Obtaining MySQL Cluster NDB 6.2. You can download the latest MySQL Cluster NDB 6.2 source code and binaries for supported platforms from http://dev.mysql.com/downloads/cluster.

This Beta release incorporates all bugfixes and changes made in previous MySQL Cluster NDB 6.2 releases, as well as all bugfixes and feature changes which were added in mainline MySQL 5.1 through MySQL 5.1.24 (see Section C.1.5, “Changes in MySQL 5.1.24 (08 April 2008)”).

Note

Please refer to our bug database at http://bugs.mysql.com/ for more details about the individual bugs fixed in this version.

Functionality added or changed:

MySQL Cluster: Event buffer lag reports are now written to the cluster log. (Bug#37427)
MySQL Cluster: Added the --no-binlog option for ndb_restore. When used, this option prevents information being written to SQL node binary logs from the restoration of a cluster backup. (Bug#30452)
The ndbd and ndb_mgmd manpages have been reclassified from volume 1 to volume 8. (Bug#34642)

Bugs fixed:

Disk Data: Important Change: It is no longer possible on 32-bit systems to issue statements appearing to create Disk Data log files or data files greater than 4 GB in size. (Trying to create log files or data files larger than 4 GB on 32-bit systems led to unrecoverable data node failures; such statements now fail with NDB error 1515.) (Bug#29186)
Cluster API: MySQL Cluster: Changing the system time on data nodes could cause MGM API applications to hang and the data nodes to crash. (Bug#35607)
MySQL Cluster: In certain rare situations, ndb_size.pl could fail with the error Can't use string ("value") as a HASH ref while "strict refs" in use. (Bug#43022)
MySQL Cluster: Failure of a data node could sometimes cause mysqld to crash. (Bug#37628)
MySQL Cluster: If subscription was terminated while a node was down, the epoch was not properly acknowledged by that node. (Bug#37442)
MySQL Cluster: In rare circumstances, a connection followed by a disconnection could give rise to a “stale” connection where the connection still existed but was not seen by the transporter. (Bug#37338)
MySQL Cluster: Under some circumstances, a failed CREATE TABLE could mean that subsequent CREATE TABLE statements caused node failures. (Bug#37092)
MySQL Cluster: A fail attempt to create an NDB table could in some cases lead to resource leaks or cluster failures. (Bug#37072)
MySQL Cluster: Checking of API node connections was not efficiently handled. (Bug#36843)
MySQL Cluster: Attempting to delete a non-existent row from a table containing a TEXT or BLOB column within a transaction caused the transaction to fail. (Bug#36756)
See also Bug#36851
MySQL Cluster: Queries against NDBCLUSTER tables were cached only if AUTOCOMMIT was in use. (Bug#36692)
MySQL Cluster: SET GLOBAL ndb_extra_logging caused mysqld to crash. (Bug#36547)
MySQL Cluster: If the combined total of tables and indexes in the cluster was greater than 4096, issuing START BACKUP caused data nodes to fail. (Bug#36044)
MySQL Cluster: When more than one SQL node connected to the cluster at the same time, creation of the mysql.ndb_schema table failed on one of them with an explicit Table exists error, which was not necessary. (Bug#35943)
MySQL Cluster: mysqld failed to start after running mysql_upgrade. (Bug#35708)
MySQL Cluster: If an error occured while executing a statement involving a BLOB or TEXT column of an NDB table, a memory leak could result. (Bug#35593)
MySQL Cluster: It was not possible to determine the value used for the --ndb-cluster-connection-pool option in the mysql client. Now this value is reported as a system status variable. (Bug#35573)
MySQL Cluster: The ndb_waiter utility wrongly calculated timeouts. (Bug#35435)
MySQL Cluster: Where column values to be compared in a query were of the VARCHAR or VARBINARY types, NDBCLUSTER passed a value padded to the full size of the column, which caused unnecessary data to be sent to the data nodes. This also had the effect of wasting CPU and network bandwidth, and causing condition pushdown to be disabled where it could (and should) otherwise have been applied. (Bug#35393)
MySQL Cluster: ndb_restore incorrectly handled some datatypes when applying log files from backups. (Bug#35343)
MySQL Cluster: In some circumstances, a stopped data node was handled incorrectly, leading to redo log space being exhausted following an initial restart of the node, or an initial or partial restart of the cluster (the wrong CGI might be used in such cases). This could happen, for example, when a node was stopped following the creation of a new table, but before a new LCP could be executed. (Bug#35241)
MySQL Cluster: SELECT ... LIKE ... queries yielded incorrect results when used on NDB tables. As part of this fix, condition pushdown of such queries has been disabled; re-enabling it is expected to be done as part of a later, permanent fix for this issue. (Bug#35185)
MySQL Cluster: ndb_mgmd reported errors to STDOUT rather than to STDERR. (Bug#35169)
MySQL Cluster: Nested multi-range reads failed when the second multi-range read released the first read's unprocessed operations, sometimes leading to a SQL node crash. (Bug#35137)
MySQL Cluster: In some situations, a problem with synchronizing checkpoints between nodes could cause a system restart or a node restart to fail with Error 630 during restore of TX. (Bug#34756)
MySQL Cluster: When a secondary index on a DECIMAL column was used to retrieve data from an NDB table, no results were returned even if the target table had a matched value in the column that was defined with the secondary index. (Bug#34515)
MySQL Cluster: An UPDATE on an NDB table that set a new value for a unique key column could cause subsequent queries to fail. (Bug#34208)
MySQL Cluster: If a data node in one node group was placed in the “not started” state (using node_id RESTART -n), it was not possible to stop a data node in a different node group. (Bug#34201)
MySQL Cluster: When configured with NDB support, MySQL failed to compile on 64bit FreeBSD systems. (Bug#34046)
MySQL Cluster: Numerous NDBCLUSTER test failures occurred in builds compiled using icc on IA64 platforms. (Bug#31239)
MySQL Cluster: CREATE TABLE and ALTER TABLE statements using ENGINE=NDB or ENGINE=NDBCLUSTER caused mysqld to fail on Solaris 10 for x86 platforms. (Bug#19911)
MySQL Cluster: If an API node disconnected and then reconnected during Start Phase 8, then the connection could be “blocked” — that is, the QMGR kernel block failed to detect that the API node was in fact connected to the cluster, causing issues with the NDB Subscription Manager (SUMA).
MySQL Cluster: NDB error 1427 (Api node died, when SUB_START_REQ reached node) was incorrectly classified as a schema error rather than a temporary error.
MySQL Cluster: When dropping a table failed for any reason (such as when in single user mode) then the corresponding .ndb file was still removed.
Replication: When flushing tables, there were a slight chance that the flush occurred between the processing of one table map event and the next. Since the tables were opened one by one, subsequent locking of tables would cause the slave to crash. This problem was observed when replicating NDBCLUSTER or InnoDB tables, when executing multi-table updates, and when a trigger or a stored routine performed an (additional) insert on a table so that two tables were effectively being inserted into in the same statement. (Bug#36197)
Cluster Replication: Performing SELECT ... FROM mysql.ndb_apply_status before the mysqld process had connected to the cluster failed, and caused this table never to be created. (Bug#36123)
Cluster Replication: In some cases, when updating only one or some columns in a table, the complete row was written to the binary log instead of only the updated column or columns, even when ndb_log_updated_only was set to 1. (Bug#35208)
Cluster API: Creating a table on an SQL node, then starting an NDB API application that listened for events from this table, then dropping the table from an SQL node, prevented data node restarts. (Bug#37279)
Cluster API: When some operations succeeded and some failed following a call to NdbTransaction::execute(Commit, AO_IgnoreOnError), a race condition could cause spurious occurrences of NDB API Error 4011 (Internal error). (Bug#37158)
Cluster API: Ordered index scans were not pruned correctly where a partitioning key was specified with an EQ-bound. (Bug#36950)
Cluster API: When an insert operation involving BLOB data was attempted on a row which already existed, no duplicate key error was correctly reported and the transaction is incorrectly aborted. In some cases, the existing row could also become corrupted. (Bug#36851)
See also Bug#26756
Cluster API: NdbApi.hpp depended on ndb_global.h, which was not actually installed, causing the compilation of programs that used NdbApi.hpp to fail. (Bug#35853)
Cluster API: Attempting to pass a nonexistent column name to the equal() and setValue() methods of NdbOperation caused NDB API applications to crash. Now the column name is checked, and an error is returned in the event that the column is not found. (Bug#33747)
Cluster API: A buffer overrun in NdbBlob::setValue() caused erroneous results on Mac OS X. (Bug#31284)
Cluster API: Relocation errors were encountered when trying to compile NDB API applications on a number of platforms, including 64-bit Linux. As a result, libmysys, libmystrings, and libdbug have been changed from normal libraries to “noinst” libtool helper libraries. They are no longer installed as separate libraries; instead, all necessary symbols from these are added directly to libndbclient. This means that NDB API programs now need to be linked only using -lndbclient. (Bug#29791)

20.17.2.2. Changes in MySQL Cluster NDB 6.2.15 (5.1.23-ndb-6.2.15) (22 May 2008)

This is re-release of MySQL Cluster NDB 6.2.14 providing binaries for supported platforms. For more information, see Section 20.17.2.3, “Changes in MySQL Cluster NDB 6.2.14 (5.1.23-ndb-6.2.14) (05 March 2008)”.

Obtaining MySQL Cluster NDB 6.2. You can download the latest MySQL Cluster NDB 6.2 source code and binaries for supported platforms from http://dev.mysql.com/downloads/cluster.

20.17.2.3. Changes in MySQL Cluster NDB 6.2.14 (5.1.23-ndb-6.2.14) (05 March 2008)

This is a bugfix release, fixing recently discovered bugs in the previous MySQL Cluster NDB 6.2 release.

Obtaining MySQL Cluster NDB 6.2. You can download the latest MySQL Cluster NDB 6.2 source code and binaries for supported platforms from http://dev.mysql.com/downloads/cluster.

This Beta release incorporates all bugfixes and changes made in previous MySQL Cluster NDB 6.2 releases, as well as all bugfixes and feature changes which were added in mainline MySQL 5.1 through MySQL 5.1.23 (see Section C.1.6, “Changes in MySQL 5.1.23 (29 January 2008)”).

Note

Please refer to our bug database at http://bugs.mysql.com/ for more details about the individual bugs fixed in this version.

Functionality added or changed:

MySQL Cluster: Added the MaxBufferedEpochs data node configuration parameter, which controls the maximum number of unprocessed epochs by which a subscribing node can lag. Subscribers which exceed this number are disconnected and forced to reconnect.
See Section 20.3.4.5, “Defining Data Nodes”, for more information.
Replication: Introduced the slave_exec_mode system variable to control whether idempotent or strict mode is used for replication conflict resolution. Idempotent mode suppresses duplicate-key, no-key-found, and some other errors, and is needed for circular replication, multi-master replication, and some other complex replication setups when using MySQL Cluster. Strict mode is the default. (Bug#31609)
Cluster Replication: RESET MASTER now uses TRUNCATE rather than DELETE to clear the mysql.ndb_binlog_index table. This improves the performance of the statement and is less likely to leave the table in a fragmented state. (Bug#34356)
Formerly, when the MySQL server crashed, the generated stack dump was numeric and required external tools to properly resolve the names of functions. This is not very helpful to users having a limited knowledge of debugging techniques. In addition, the generated stack trace contained only the names of functions and was formatted differently for each platform due to different stack layouts.
Now it is possible to take advantage of newer versions of the GNU C Library provide a set of functions to obtain and manipulate stack traces from within the program. On systems that use the ELF binary format, the stack trace contains important information such as the shared object where the call was generated, an offset into the function, and the actual return address. Having the function name also makes possible the name demangling of C++ functions.
The library generates meaningful stack traces on the following platforms: i386, x86_64, PowerPC, IA64, Alpha, and S390. On other platforms, a numeric stack trace is still produced, and the use of the resolve_stack_dump utility is still required. (Bug#31891)
mysqltest now has mkdir and rmdir commands for creating and removing directories. (Bug#31004)
Added the Uptime_since_flush_status status variable, which indicates the number of seconds since the most recent FLUSH STATUS statement. (From Jeremy Cole) (Bug#24822)

Bugs fixed:

Important Change: Replication: When the master crashed during an update on a transactional table while in AUTOCOMMIT mode, the slave failed. This fix causes every transaction (including AUTOCOMMIT transactions) to be recorded in the binlog as starting with a BEGIN and ending with a COMMIT or ROLLBACK. (Bug#26395)
Replication: Important Note: Network timeouts between the master and the slave could result in corruption of the relay log. This fix rectifies a long-standing replication issue when using unreliable networks, including replication over wide area networks such as the Internet. If you experience reliability issues and see many You have an error in your SQL syntax errors on replication slaves, we strongly recommend that you upgrade to a MySQL version which includes this fix. (Bug#26489)
Replication: When the Windows version of mysqlbinlog read 4.1 binlogs containing LOAD DATA INFILE statements, it output backslashes as path separators, causing problems for client programs expecting forward slashes. In such cases, it now converts \\ to / in directory paths. (Bug#34355)
Replication: SHOW SLAVE STATUS failed when slave I/O was about to terminate. (Bug#34305)
Replication: mysqlbinlog from a 5.1 or later MySQL distribution could not read binary logs generated by a 4.1 server when the logs contained LOAD DATA INFILE statements. (Bug#34141)
This regression was introduced by Bug#32407
Replication: A CREATE USER, DROP USER, or RENAME USER statement that fails on the master, or that is a duplicate of any of these statements, is no longer written to the binlog; previously, either of these occurrences could cause the slave to fail.
(Bug#33862)
See also Bug#29749
Replication: mysqlbinlog failed to release all of its memory after terminating abnormally. (Bug#33247)
Replication: The error message generated due to lack of a default value for an extra column was not sufficiently informative. (Bug#32971)
Replication: When a user variable was used inside an INSERT statement, the corresponding binlog event was not written to the binlog correctly. (Bug#32580)
Replication: When using row-based replication, deletes from a table with a foreign key constraint failed on the slave. (Bug#32468)
Replication: SQL statements containing comments using -- syntax were not replayable by mysqlbinlog, even though such statements replicated correctly. (Bug#32205)
Replication: When using row-based replication from a master running MySQL 5.1.21 or earlier to a slave running 5.1.22 or later, updates of integer columns failed on the slave with Error in Unknown event: row application failed. (Bug#31583)
This regression was introduced by Bug#21842
Replication: Replicating write, update, or delete events from a master running MySQL 5.1.15 or earlier to a slave running 5.1.16 or later caused the slave to crash. (Bug#31581)
Replication: When using row-based replication, the slave stopped when attempting to delete non-existent rows from a slave table without a primary key. In addition, no error was reported when this occurred. (Bug#31552)
Replication: Issuing a DROP VIEW statement caused replication to fail if the view did not actually exist. (Bug#30998)
Replication: Setting server_id did not update its value for the current session. (Bug#28908)
Replication: Some older servers wrote events to the binary log using different numbering from what is currently used, even though the file format number in the file is the same. Slaves running MySQL 5.1.18 and later could not read these binary logs properly. Binary logs from these older versions now are recognized and event numbers are mapped to the current numbering so that they can be interpreted properly. (Bug#27779, Bug#32434)
This regression was introduced by Bug#22583
Cluster Replication: The --ndb-wait-connected option caused the server to wait for a partial connection plus an additional 3 seconds for a complete connection to the cluster. This could lead to issues with setting up the binary log. (Bug#34757)
Cluster API: Closing a scan before it was executed caused the application to segfault. (Bug#36375)
Cluster API: Using NDB API applications from older MySQL Cluster versions with libndbclient from newer ones caused the cluster to fail. (Bug#36124)
Cluster API: Scans having no bounds set were handled incorrectly. (Bug#35876)
Use of stored functions in the WHERE clause for SHOW OPEN TABLES caused a server crash. (Bug#34166)
Large unsigned integers were improperly handled for prepared statements, resulting in truncation or conversion to negative numbers. (Bug#33798)
The server crashed when executing a query that had a subquery containing an equality X=Y where Y referred to a named select list expression from the parent select. The server crashed when trying to use the X=Y equality for ref-based access. (Bug#33794)
The UPDATE statement allowed NULL to be assigned to NOT NULL columns (the default data type value was assigned). An error occurs now. (Bug#33699)
ORDER BY ... DESC sorts could produce misordered results. (Bug#33697)
The server could crash when REPEAT or another control instruction was used in conjunction with labels and a LEAVE instruction. (Bug#33618)
SET GLOBAL myisam_max_sort_file_size=DEFAULT set myisam_max_sort_file_size to an incorrect value. (Bug#33382)
See also Bug#31177
Granting the UPDATE privilege on one column of a view caused the server to crash. (Bug#33201)
For DECIMAL columns used with the ROUND(X,D) or TRUNCATE(X,D) function with a non-constant value of D, adding an ORDER BY for the function result produced misordered output. (Bug#33143)
See also Bug#33402, Bug#30617
The SHOW ENGINE INNODB STATUS and SHOW ENGINE INNODB MUTEX statements incorrectly required the SUPER privilege rather than the PROCESS privilege. (Bug#32710)
Tables in the mysql database that stored the current sql_mode value as part of stored program definitions were not updated with newer mode values (NO_ENGINE_SUBSTITUTION, PAD_CHAR_TO_FULL_LENGTH). This causes various problems defining stored programs if those modes were included in the current sql_mode value. (Bug#32633)
ROUND(X,D) or TRUNCATE(X,D) for non-constant values of D could crash the server if these functions were used in an ORDER BY that was resolved using filesort. (Bug#30889)
Resetting the query cache by issuing a SET GLOBAL query_cache_size=0 statement caused the server to crash if it concurrently was saving a new result set to the query cache. (Bug#30887)
Replication of LOAD DATA INFILE could fail when read_buffer_size was larger than max_allowed_packet. (Bug#30435)
The Table_locks_waited waited variable was not incremented in the cases that a lock had to be waited for but the waiting thread was killed or the request was aborted. (Bug#30331)
The Com_create_function status variable was not incremented properly. (Bug#30252)
mysqld displayed the --enable-pstack option in its help message even if MySQL was configured without --with-pstack. (Bug#29836)
Replication crashed with the NDB storage engine when mysqld was started with --character-set-server=ucs2. (Bug#29562)
Views were treated as insertable even if some base table columns with no default value were omitted from the view definition. (This is contrary to the condition for insertability that a view must contain all columns in the base table that do not have a default value.) (Bug#29477)
Previously, the parser accepted the ODBC { OJ ... LEFT OUTER JOIN ...} syntax for writing left outer joins. The parser now allows { OJ ... } to be used to write other types of joins, such as INNER JOIN or RIGHT OUTER JOIN. This helps with compatibility with some third-party applications, but is not official ODBC syntax. (Bug#28317)
The parser rules for the SHOW PROFILE statement were revised to work with older versions of bison. (Bug#27433)
resolveip failed to produce correct results for hostnames that begin with a digit. (Bug#27427)
mysqlcheck -A -r did not correctly identify all tables that needed repairing. (Bug#25347)
InnoDB exhibited thread thrashing with more than 50 concurrent connections under an update-intensive workload. (Bug#22868)
Warnings for deprecated syntax constructs used in stored routines make sense to report only when the routine is being created, but they were also being reported when the routine was parsed for loading into the execution cache. Now they are reported only at routine creation time. (Bug#21801)
CREATE ... SELECT did not always set DEFAULT column values in the new table. (Bug#21380)
If a SELECT calls a stored function in a transaction, and a statement within the function fails, that statement should roll back. Furthermore, if ROLLBACK is executed after that, the entire transaction should be rolled back. Before this fix, the failed statement did not roll back when it failed (even though it might ultimately get rolled back by a ROLLBACK later that rolls back the entire transaction). (Bug#12713)
See also Bug#34655

20.17.2.4. Changes in MySQL Cluster NDB 6.2.13 (5.1.23-ndb-6.2.13) (22 February 2008)

This is a bugfix release, fixing recently discovered bugs in the previous MySQL Cluster NDB 6.2 release.

Obtaining MySQL Cluster NDB 6.2. You can download the latest MySQL Cluster NDB 6.2 source code and binaries for supported platforms from http://dev.mysql.com/downloads/cluster.

This Beta release incorporates all bugfixes and changes made in previous MySQL Cluster NDB 6.2 releases, as well as all bugfixes and feature changes which were added in mainline MySQL 5.1 through MySQL 5.1.23 (see Section C.1.6, “Changes in MySQL 5.1.23 (29 January 2008)”).

Note

Please refer to our bug database at http://bugs.mysql.com/ for more details about the individual bugs fixed in this version.

Bugs fixed:

MySQL Cluster: A node failure during an initial node restart followed by another node start could cause the master data node to fail, because it incorrectly gave the node permission to start even if the invalidated node's LCP was still running. (Bug#34702)

20.17.2.5. Changes in MySQL Cluster NDB 6.2.12 (5.1.23-ndb-6.2.12) (12 February 2008)

This is a bugfix release, fixing recently discovered bugs in the previous MySQL Cluster NDB 6.2 release.

Obtaining MySQL Cluster NDB 6.2. You can download the latest MySQL Cluster NDB 6.2 source code and binaries for supported platforms from http://dev.mysql.com/downloads/cluster.

This Beta release incorporates all bugfixes and changes made in previous MySQL Cluster NDB 6.2 releases, as well as all bugfixes and feature changes which were added in mainline MySQL 5.1 through MySQL 5.1.23 (see Section C.1.6, “Changes in MySQL 5.1.23 (29 January 2008)”).

Note

Please refer to our bug database at http://bugs.mysql.com/ for more details about the individual bugs fixed in this version.

Functionality added or changed:

Beginning with this version, MySQL Cluster NDB 6.3.x releases once again include the InnoDB storage engine. In order to enable InnoDB, you must configure the build using --with-innodb.

Bugs fixed:

MySQL Cluster: Upgrades of a cluster using while a DataMemory setting in excess of 16 GB caused data nodes to fail. (Bug#34378)
MySQL Cluster: Performing many SQL statements on NDB tables while in AUTOCOMMIT mode caused a memory leak in mysqld. (Bug#34275)
MySQL Cluster: In certain rare circumstances, a race condition could occur between an aborted insert and a delete leading a data node crash. (Bug#34260)
MySQL Cluster: Multi-table updates using ordered indexes during handling of node failures could cause other data nodes to fail. (Bug#34216)
MySQL Cluster: When configured with NDB support, MySQL failed to compile using gcc 4.3 on 64bit FreeBSD systems. (Bug#34169)
MySQL Cluster: The failure of a DDL statement could sometimes lead to node failures when attempting to execute subsequent DDL statements. (Bug#34160)
MySQL Cluster: Extremely long SELECT statements (where the text of the statement was in excess of 50000 characters) against NDB tables returned empty results. (Bug#34107)
MySQL Cluster: Statements executing multiple inserts performed poorly on NDB tables having AUTO_INCREMENT columns. (Bug#33534)
MySQL Cluster: The ndb_waiter utility polled ndb_mgmd excessively when obtaining the status of cluster data nodes. (Bug#32025)
See also Bug#32023
MySQL Cluster: Transaction atomicity was sometimes not preserved between reads and inserts under high loads. (Bug#31477)
MySQL Cluster: Having tables with a great many columns could cause Cluster backups to fail. (Bug#30172)
Cluster Replication: Disk Data: Statements violating unique keys on Disk Data tables (such as attempting to insert NULL into a NOT NULL column) could cause data nodes to fail. When the statement was executed from the binlog, this could also result in failure of the slave cluster. (Bug#34118)
Disk Data: Updating in-memory columns of one or more rows of Disk Data table, followed by deletion of these rows and re-insertion of them, caused data node failures. (Bug#33619)
Cluster Replication: Setting --replicate-ignore-db=mysql caused the mysql.ndb_apply_status table not to be replicated, breaking Cluster Replication. (Bug#28170)

20.17.2.6. Changes in MySQL Cluster NDB 6.2.11 (5.1.23-ndb-6.2.11) (28 January 2008)

This is a bugfix release, fixing recently discovered bugs in the previous MySQL Cluster NDB 6.2 release.

Obtaining MySQL Cluster NDB 6.2. You can download the latest MySQL Cluster NDB 6.2 source code and binaries for supported platforms from http://dev.mysql.com/downloads/cluster.

This Beta release incorporates all bugfixes and changes made in previous MySQL Cluster NDB 6.2 releases, as well as all bugfixes and feature changes which were added in mainline MySQL 5.1 through MySQL 5.1.23 (see Section C.1.6, “Changes in MySQL 5.1.23 (29 January 2008)”).

Note

Please refer to our bug database at http://bugs.mysql.com/ for more details about the individual bugs fixed in this version.

Functionality added or changed:

Cluster API: Important Change: Because NDB_LE_MemoryUsage.page_size_kb shows memory page sizes in bytes rather than kilobytes, it has been renamed to page_size_bytes. The name page_size_kb is now deprecated and thus subject to removal in a future release, although it currently remains supported for reasons of backward compatibility. See The Ndb_logevent_type Type, for more information about NDB_LE_MemoryUsage. (Bug#30271)

Bugs fixed:

MySQL Cluster: High numbers of insert operations, delete operations, or both could cause NDB error 899 (Rowid already allocated) to occur unnecessarily. (Bug#34033)
MySQL Cluster: A periodic failure to flush the send buffer by the NDB TCP transporter could cause a unnecessary delay of 10 ms between operations. (Bug#34005)
MySQL Cluster: A race condition could occur (very rarely) when the release of a GCI was followed by a data node failure. (Bug#33793)
MySQL Cluster: Some tuple scans caused the wrong memory page to be accessed, leading to invalid results. This issue could affect both in-memory and Disk Data tables. (Bug#33739)
MySQL Cluster: The server failed to reject properly the creation of an NDB table having an unindexed AUTO_INCREMENT column. (Bug#30417)
MySQL Cluster: Issuing an INSERT ... ON DUPLICATE KEY UPDATE concurrently with or following a TRUNCATE statement on an NDB table failed with NDB error 4350 Transaction already aborted. (Bug#29851)
MySQL Cluster: The Cluster backup process could not detect when there was no more disk space and instead continued to run until killed manually. Now the backup fails with an appropriate error when disk space is exhausted. (Bug#28647)
MySQL Cluster: It was possible in config.ini to define cluster nodes having node IDs greater than the maximum allowed value. (Bug#28298)
Cluster Replication: ndb_restore -e restored excessively large values to the ndb_apply_status table's epoch column when restoring to a MySQL Cluster version supporting Micro-GCPs from an older version that did not support these.
A workaround when restoring to MySQL Cluster releases supporting micro-GCPs previous to MySQL Cluster NDB 6.3.8 is to perform a 32-bit shift on the epoch column values to reduce them to their proper size. (Bug#33406)
Cluster API: Transactions containing inserts or reads would hang during NdbTransaction::execute() calls made from NDB API applications built against a MySQL Cluster version that did not support micro-GCPs accessing a later version that supported micro-GCPs. This issue was observed while upgrading from MySQL Cluster NDB 6.1.23 to MySQL Cluster NDB 6.2.10 when the API application built against the earlier version attempted to access a data node already running the later version, even after disabling micro-GCPs by setting TimeBetweenEpochs equal to 0. (Bug#33895)
Cluster API: When reading a BIT(64) value using NdbOperation:getValue(), 12 bytes were written to the buffer rather than the expected 8 bytes. (Bug#33750)

20.17.2.7. Changes in MySQL Cluster NDB 6.2.10 (5.1.23-ndb-6.2.10) (19 December 2007)

This is a bugfix release, fixing recently discovered bugs in the previous MySQL Cluster NDB 6.2 release.

Obtaining MySQL Cluster NDB 6.2. You can download the latest MySQL Cluster NDB 6.2 source code and binaries for supported platforms from http://dev.mysql.com/downloads/cluster.

This Beta release incorporates all bugfixes and changes made in previous MySQL Cluster NDB 6.2 releases, as well as all bugfixes and feature changes which were added in mainline MySQL 5.1 through MySQL 5.1.23 (see Section C.1.6, “Changes in MySQL 5.1.23 (29 January 2008)”).

Note

Please refer to our bug database at http://bugs.mysql.com/ for more details about the individual bugs fixed in this version.

Bugs fixed:

Partitioning: MySQL Cluster: When partition pruning on an NDB table resulted in an ordered index scan spanning only one partition, any descending flag for the scan was wrongly discarded, causing ORDER BY DESC to be treated as ORDER BY ASC, MAX() to be handled incorrectly, and similar problems. (Bug#33061)
MySQL Cluster: When all data and SQL nodes in the cluster were shut down abnormally (that is, other than by using STOP in the cluster management client), ndb_mgm used excessive amounts of CPU. (Bug#33237)
MySQL Cluster: When using micro-GCPs, if a node failed while preparing for a global checkpoint, the master node would use the wrong GCI. (Bug#32922)
MySQL Cluster: Under some conditions, performing an ALTER TABLE on an NDBCLUSTER table failed with a Table is full error, even when only 25% of DataMemory was in use and the result should have been a table using less memory (for example, changing a VARCHAR(100) column to VARCHAR(80)). (Bug#32670)

20.17.2.8. Changes in MySQL Cluster NDB 6.2.9 (5.1.22-ndb-6.2.9) (22 November 2007)

This is a bugfix release, fixing recently discovered bugs in the previous MySQL Cluster NDB 6.2 release.

Obtaining MySQL Cluster NDB 6.2. You can download the latest MySQL Cluster NDB 6.2 source code and binaries for supported platforms from http://dev.mysql.com/downloads/cluster.

This Beta release incorporates all bugfixes and changes made in previous MySQL Cluster NDB 6.2 releases, as well as all bugfixes and feature changes which were added in mainline MySQL 5.1 through MySQL 5.1.22 (see Section C.1.7, “Changes in MySQL 5.1.22 (24 September 2007: Release Candidate)”).

Note

Please refer to our bug database at http://bugs.mysql.com/ for more details about the individual bugs fixed in this version.

Functionality added or changed:

MySQL Cluster: Added the ndb_mgm client command DUMP 8011, which dumps all subscribers to the cluster log. See DUMP 8011, for more information.

Bugs fixed:

MySQL Cluster: A local checkpoint could sometimes be started before the previous LCP was restorable from a global checkpoint. (Bug#32519)
MySQL Cluster: High numbers of API nodes on a slow or congested network could cause connection negotiation to time out prematurely, leading to the following issues:
- Excessive retries
- Excessive CPU usage
- Partially connected API nodes
(Bug#32359)
MySQL Cluster: The failure of a master node could lead to subsequent failures in local checkpointing. (Bug#32160)
MySQL Cluster: Adding a new TINYTEXT column to an NDB table which used COLUMN_FORMAT = DYNAMIC, and when binary logging was enabled, caused all cluster mysqld processes to crash. (Bug#30213)
MySQL Cluster: After adding a new column of one of the TEXT or BLOB types to an NDB table which used COLUMN_FORMAT = DYNAMIC, it was no longer possible to access or drop the table using SQL. (Bug#30205)
MySQL Cluster: A restart of the cluster failed when more than 1 REDO phase was in use. (Bug#22696)
Cluster Replication: Under certain conditions, the slave stopped processing relay logs. This resulted in the logs never being cleared and the slave eventually running out of disk space. (Bug#31958)
Cluster Replication: Where a table being replicated had a TEXT or BLOB column, an UPDATE on the master that did not refer explicitly to this column in the WHERE clause stopped the SQL thread on the slave with Error in Write_rows event: row application failed. Got error 4288 'Blob handle for column not available' from NDBCLUSTER. (Bug#30674)

20.17.2.9. Changes in MySQL Cluster NDB 6.2.8 (5.1.22-ndb-6.2.8) (08 November 2007)

This is a bugfix release, fixing recently discovered bugs in the previous MySQL Cluster NDB 6.2 release.

Obtaining MySQL Cluster NDB 6.2. You can download the latest MySQL Cluster NDB 6.2 source code and binaries for supported platforms from http://dev.mysql.com/downloads/cluster.

This Beta release incorporates all bugfixes and changes made in previous MySQL Cluster NDB 6.2 releases, as well as all bugfixes and feature changes which were added in mainline MySQL 5.1 through MySQL 5.1.22 (see Section C.1.7, “Changes in MySQL 5.1.22 (24 September 2007: Release Candidate)”).

Note

Please refer to our bug database at http://bugs.mysql.com/ for more details about the individual bugs fixed in this version.

Functionality added or changed:

Important Note: MySQL Cluster NDB 6.2 and 6.3 source archives are now available in separate commercial and GPL versions. Due to licensing concerns, previous MySQL Cluster NDB 6.2 and 6.3 source archives were removed from the FTP site.

Bugs fixed:

MySQL Cluster: In a cluster running in diskless mode and with arbitration disabled, the failure of a data node during an insert operation caused other data node to fail. (Bug#31980)
MySQL Cluster: An insert or update with combined range and equality constraints failed when run against an NDB table with the error Got unknown error from NDB. An example of such a statement would be UPDATE t1 SET b = 5 WHERE a IN (7,8) OR a >= 10;. (Bug#31874)
MySQL Cluster: An error with an if statement in sql/ha_ndbcluster.cc could potentially lead to an infinite loop in case of failure when working with AUTO_INCREMENT columns in NDB tables. (Bug#31810)
MySQL Cluster: The NDB storage engine code was not safe for strict-alias optimization in gcc 4.2.1. (Bug#31761)
MySQL Cluster: Following an upgrade, ndb_mgmd would fail with an ArbitrationError. (Bug#31690)
MySQL Cluster: The NDB management client command node_id REPORT MEMORY provided no output when node_id was the node ID of a management or API node. Now, when this occurs, the management client responds with Node node_id: is not a data node. (Bug#29485)
MySQL Cluster: Performing DELETE operations after a data node had been shut down could lead to inconsistent data following a restart of the node. (Bug#26450)
MySQL Cluster: UPDATE IGNORE could sometimes fail on NDB tables due to the use of unitialized data when checking for duplicate keys to be ignored. (Bug#25817)
Cluster Replication: A node failure during replication could lead to buckets out of order; now active subscribers are checked for, rather than empty buckets. (Bug#31701)
Cluster Replication: When the master mysqld crashed or was restarted, no LOST_EVENTS entry was made in the binlog. (Bug#31484)
See also Bug#21494

20.17.2.10. Changes in MySQL Cluster NDB 6.2.7 (5.1.22-ndb-6.2.7) (10 October 2008)

This is a bugfix release, fixing recently discovered bugs in the previous MySQL Cluster NDB 6.2 release.

Obtaining MySQL Cluster NDB 6.2. You can download the latest MySQL Cluster NDB 6.2 source code and binaries for supported platforms from http://dev.mysql.com/downloads/cluster.

This Beta release incorporates all bugfixes and changes made in previous MySQL Cluster NDB 6.2 releases, as well as all bugfixes and feature changes which were added in mainline MySQL 5.1 through MySQL 5.1.22 (see Section C.1.7, “Changes in MySQL 5.1.22 (24 September 2007: Release Candidate)”).

Note

Please refer to our bug database at http://bugs.mysql.com/ for more details about the individual bugs fixed in this version.

Functionality added or changed:

Cluster Replication: A new configuration parameter TimeBetweenEpochsTimeout allows a timeout to be set for time between epochs. For more information, see Section 20.3.4.5, “Defining Data Nodes”. (Bug#31276)
Additional checks were implemented to catch unsupported online ALTER TABLE operations. Currently it is not possible to reorder columns or to change the storage engine used for a table via online ALTER TABLE.
Some redundant checks made during online creation of indexes were removed.

Bugs fixed:

MySQL Cluster: It was possible in some cases for a node group to be “lost” due to missed local checkpoints following a system restart. (Bug#31525)
MySQL Cluster: NDB tables having names containing non-alphanumeric characters (such as “ $ ”) were not discovered correctly. (Bug#31470)
MySQL Cluster: A node failure during a local checkpoint could lead to a subsequent failure of the cluster during a system restart. (Bug#31257)
MySQL Cluster: A cluster restart could sometimes fail due to an issue with table IDs. (Bug#30975)
MySQL Cluster: Transaction timeouts were not handled well in some circumstances, leading to excessive number of transactions being aborted unnecessarily. (Bug#30379)
MySQL Cluster: In some cases, the cluster managment server logged entries multiple times following a restart of mgmd. (Bug#29565)
MySQL Cluster: ndb_mgm --help did not display any information about the -a option. (Bug#29509)
MySQL Cluster: The cluster log was formatted inconsistently and contained extraneous newline characters. (Bug#25064)
Online ALTER operations involving a column whose data type has an implicit default value left behind temporary .FRM files, causing subsequent DROP DATABASE statements to fail. (Bug#31097)
Transactions were committed prematurely when LOCK TABLE and SET AUTOCOMMIT=OFF were used together. (Bug#30996)
The mysqld_safe script contained a syntax error. (Bug#30624)

20.17.2.11. Changes in MySQL Cluster NDB 6.2.6 (5.1.22-ndb-6.2.6) (20 September 2007)

This is a bugfix release, fixing recently discovered bugs in the previous MySQL Cluster NDB 6.2 release.

Obtaining MySQL Cluster NDB 6.2. You can download the latest MySQL Cluster NDB 6.2 source code and binaries for supported platforms from http://dev.mysql.com/downloads/cluster.

This Beta release incorporates all bugfixes and changes made in previous MySQL Cluster NDB 6.2 releases, as well as all bugfixes and feature changes which were added in mainline MySQL 5.1 through MySQL 5.1.22 (see Section C.1.7, “Changes in MySQL 5.1.22 (24 September 2007: Release Candidate)”).

Note

Please refer to our bug database at http://bugs.mysql.com/ for more details about the individual bugs fixed in this version.

Functionality added or changed:

MySQL Cluster: Mapping of NDB error codes to MySQL storage engine error codes has been improved. (Bug#28423)

Bugs fixed:

Partitioning: MySQL Cluster: EXPLAIN PARTITIONS reported partition usage by queries on NDB tables according to the standard MySQL hash function than the hash function used in the NDB storage engine. (Bug#29550)
MySQL Cluster: When an NDB event was left behind but the corresponding table was later recreated and received a new table ID, the event could not be dropped. (Bug#30877)
MySQL Cluster: Attempting to restore a backup made on a cluster host using one endian to a machine using the other endian could cause the cluster to fail. (Bug#29674)
MySQL Cluster: The description of the --print option provided in the output from ndb_restore --help was incorrect. (Bug#27683)
MySQL Cluster: Restoring a backup made on a cluster host using one endian to a machine using the other endian failed for BLOB and DATETIME columns. (Bug#27543, Bug#30024)
MySQL Cluster: An insufficiently descriptive and potentially misleading Error 4006 (Connect failure - out of connection objects...) was produced when either of the following two conditions occurred:
1. There were no more transaction records in the transaction coordinator
2. an Ndb object in the NDB API was initialized with insufficient parallellism
Separate error messages are now generated for each of these two cases. (Bug#11313)
Cluster API: An Ndb object in the NDB API was initialized with insufficient parallellism.
For micro-GCPs, fixed the assignment of “fake” CGI events so that they do not cause buckets to be sent out of order. Now, when assigning a GCI to a non-GCI event (that is, creating a pseudo-GCI or “fake” CGI), the GCI that is to arrive is always initiated, even if no known GCI exists, which could occur in the event of a node failure. (Bug#30884)
There were no more transaction records in the transaction coordinator

20.17.2.12. Changes in MySQL Cluster NDB 6.2.5 (5.1.22-ndb-6.2.5) (06 September 2007)

This is a bugfix release, fixing recently discovered bugs in the previous MySQL Cluster NDB 6.2 release.

Obtaining MySQL Cluster NDB 6.2. You can download the latest MySQL Cluster NDB 6.2 source code and binaries for supported platforms from http://dev.mysql.com/downloads/cluster.

This Beta release incorporates all bugfixes and changes made in previous MySQL Cluster NDB 6.2 releases, as well as all bugfixes and feature changes which were added in mainline MySQL 5.1 through MySQL 5.1.22 (see Section C.1.7, “Changes in MySQL 5.1.22 (24 September 2007: Release Candidate)”).

Note

Please refer to our bug database at http://bugs.mysql.com/ for more details about the individual bugs fixed in this version.

Functionality added or changed:

MySQL Cluster: The following improvements have been made in the ndb_size.pl utility:
- The script can now be used with multiple databases; lists of databases and tables can also be excluded from analysis.
- Schema name information has been added to index table calculations.
- The database name is now an optional parameter, the exclusion of which causes all databases to be examined.
- If selecting from INFORMATION_SCHEMA fails, the script now attempts to fall back to SHOW TABLES.
- A --real_table_name option has been added; this designates a table to handle unique index size calculations.
- The report title has been amended to cover cases where more than one database is being analyzed.
Support for a --socket option was also added.
For more information, see Section 20.10.15, “ndb_size.pl — NDBCLUSTER Size Requirement Estimator”. (Bug#28683, Bug#28253)
MySQL Cluster: Online ADD COLUMN, ADD INDEX, and DROP INDEX operations can now be performed explicitly for NDB tables, as well as online renaming of tables and columns for NDB and MyISAM tables — that is, without copying or locking of the affected tables — using ALTER ONLINE TABLE.
Indexes can also be created and dropped online using CREATE INDEX and DROP INDEX, respectively, using the ONLINE keyword.
You can force operations that would otherwise be performed online to be done offline using the OFFLINE keyword.
See Section 12.1.4, “ALTER TABLE Syntax”, Section 12.1.7, “CREATE INDEX Syntax”, and Section 12.1.13, “DROP INDEX Syntax”, for more information.
MySQL Cluster: It is now possible to control whether fixed-width or variable-width storage is used for a given column of an NDB table by means of the COLUMN_FORMAT specifier as part of the column's definition in a CREATE TABLE or ALTER TABLE statement.
It is also possible to control whether a given column of an NDB table is stored in memory or on disk, using the STORAGE specifier as part of the column's definition in a CREATE TABLE or ALTER TABLE statement.
For permitted values and other information about COLUMN_FORMAT and STORAGE, see Section 12.1.10, “CREATE TABLE Syntax”.
MySQL Cluster: A new cluster management server startup option --bind-address makes it possible to restrict management client connections to ndb_mgmd to a single host and port. For more information, see Section 20.6.5.2, “Command Options for ndb_mgmd”.
Cluster Replication: The protocol for handling global checkpoints has been changed. It is now possible to control how often the GCI number is updated, and how often global checkpoints are written to disk, using the TimeBetweenEpochs configuration parameter. This improves the reliability and performance of MySQL Cluster Replication.
GCPs handled using the new protocol are sometimes referred to as “micro-GCPs”.
For more information, see TimeBetweenEpochs.

Bugs fixed:

MySQL Cluster: When handling BLOB columns, the addition of read locks to the lock queue was not handled correctly. (Bug#30764)
MySQL Cluster: Discovery of NDB tables did not work correctly with INFORMATION_SCHEMA. (Bug#30667)
MySQL Cluster: A filesystem close operation could fail during a node or system restart. (Bug#30646)
MySQL Cluster: Using the --ndb-cluster-connection-pool option for mysqld caused DDL statements to be executed twice. (Bug#30598)
MySQL Cluster: ndb_size.pl failed on tables with FLOAT columns whose definitions included commas (for example, FLOAT(6,2)). (Bug#29228)
MySQL Cluster: Reads on BLOB columns were not locked when they needed to be to guarantee consistency. (Bug#29102)
See also Bug#31482
MySQL Cluster: A query using joins between several large tables and requiring unique index lookups failed to complete, eventually returning Uknown Error after a very long period of time. This occurred due to inadequate handling of instances where the Transaction Coordinator ran out of TransactionBufferMemory, when the cluster should have returned NDB error code 4012 (Request ndbd time-out). (Bug#28804)
MySQL Cluster: An attempt to perform a SELECT ... FROM INFORMATION_SCHEMA.TABLES whose result included information about NDB tables for which the user had no privileges crashed the MySQL Server on which the query was performed. (Bug#26793)
Cluster Replication: Cluster replication did not handle large VARCHAR columns correctly. (Bug#29904)
Cluster Replication: An issue with the mysql.ndb_apply_status table could cause NDB schema autodiscovery to fail in certain rare circumstances. (Bug#20872)
Cluster API: A call to CHECK_TIMEDOUT_RET() in mgmapi.cpp should have been a call to DBUG_CHECK_TIMEDOUT_RET(). (Bug#30681)

20.17.2.13. Changes in MySQL Cluster NDB 6.2.4 (5.1.19-ndb-6.2.4) (04 July 2007)

This is a bugfix release, fixing recently discovered bugs in the previous MySQL Cluster NDB 6.2 release.

Obtaining MySQL Cluster NDB 6.2. You can download the latest MySQL Cluster NDB 6.2 source code and binaries for supported platforms from http://dev.mysql.com/downloads/cluster.

This Beta release incorporates all bugfixes and changes made in previous MySQL Cluster NDB 6.2 releases, as well as all bugfixes and feature changes which were added in mainline MySQL 5.1 through MySQL 5.1.19 (see Section C.1.10, “Changes in MySQL 5.1.19 (25 May 2007)”).

Note

Please refer to our bug database at http://bugs.mysql.com/ for more details about the individual bugs fixed in this version.

Bugs fixed:

MySQL Cluster: When restarting a data node, queries could hang during that node's start phase 5, and continue only after the node had entered phase 6. (Bug#29364)
MySQL Cluster: Replica redo logs were inconsistently handled during a system restart. (Bug#29354)
Disk Data: Performing Disk Data schema operations during a node restart could cause forced shutdowns of other data nodes. (Bug#29501)
Disk Data: Disk data meta-information that existed in ndbd might not be visible to mysqld. (Bug#28720)
Disk Data: The number of free extents was incorrectly reported for some tablespaces. (Bug#28642)
Batching of transactions was not handled correctly in some cases. (Bug#29525)

20.17.2.14. Changes in MySQL Cluster NDB 6.2.3 (5.1.19-ndb-6.2.3) (02 July 2007)

This is a bugfix release, fixing recently discovered bugs in the previous MySQL Cluster NDB 6.2 release.

Obtaining MySQL Cluster NDB 6.2. You can download the latest MySQL Cluster NDB 6.2 source code and binaries for supported platforms from http://dev.mysql.com/downloads/cluster.

This Beta release incorporates all bugfixes and changes made in previous MySQL Cluster NDB 6.2 releases, as well as all bugfixes and feature changes which were added in mainline MySQL 5.1 through MySQL 5.1.18 (see Section C.1.11, “Changes in MySQL 5.1.18 (08 May 2007)”).

Note

Please refer to our bug database at http://bugs.mysql.com/ for more details about the individual bugs fixed in this version.

Bugs fixed:

MySQL Cluster: When restarting a data node, queries could hang during that node's start phase 5, and continue only after the node had entered phase 6. (Bug#29364)
MySQL Cluster: Replica redo logs were inconsistently handled during a system restart. (Bug#29354)
Disk Data: Performing Disk Data schema operations during a node restart could cause forced shutdowns of other data nodes. (Bug#29501)
Disk Data: Disk data meta-information that existed in ndbd might not be visible to mysqld. (Bug#28720)
Disk Data: The number of free extents was incorrectly reported for some tablespaces. (Bug#28642)
Batching of transactions was not handled correctly in some cases. (Bug#29525)

20.17.2.15. Changes in MySQL Cluster NDB 6.2.2 (5.1.18-ndb-6.2.2) (07 May 2007)

This is a bugfix release, fixing recently discovered bugs in the previous MySQL Cluster NDB 6.2 release.

Obtaining MySQL Cluster NDB 6.2. You can download the latest MySQL Cluster NDB 6.2 source code and binaries for supported platforms from http://dev.mysql.com/downloads/cluster.

This Beta release incorporates all bugfixes and changes made in previous MySQL Cluster NDB 6.2 releases, as well as all bugfixes and feature changes which were added in mainline MySQL 5.1 through MySQL 5.1.18 (see Section C.1.11, “Changes in MySQL 5.1.18 (08 May 2007)”).

Note

Please refer to our bug database at http://bugs.mysql.com/ for more details about the individual bugs fixed in this version.

Functionality added or changed:

MySQL Cluster: New cluster management client DUMP commands were added to aid in tracking transactions, scan operations, and locks. See DUMP 2350, DUMP 2352, and DUMP 2550, for more information.
MySQL Cluster: Added the mysqld option --ndb-cluster-connection-pool that allows a single MySQL server to use multiple connections to the cluster. This allows for scaling out using multiple MySQL clients per SQL node instead of or in addition to using multiple SQL nodes with the cluster.
For more information about this option, see Section 20.4, “MySQL Cluster Options and Variables”.

20.17.2.16. Changes in MySQL Cluster NDB 6.2.1 (5.1.18-ndb-6.2.1) (30 April 2007)

This is a Beta development release, fixing recently discovered bugs in the previous MySQL Cluster NDB 6.2 release.

Obtaining MySQL Cluster NDB 6.2. You can download the latest MySQL Cluster NDB 6.2 source code and binaries for supported platforms from http://dev.mysql.com/downloads/cluster.

This Beta release incorporates all bugfixes and changes made in the previous MySQL Cluster NDB 6.2 release, as well as all bugfixes and feature changes which were added in mainline MySQL 5.1 through MySQL 5.1.18 (see Section C.1.11, “Changes in MySQL 5.1.18 (08 May 2007)”).

Note

Please refer to our bug database at http://bugs.mysql.com/ for more details about the individual bugs fixed in this version.

Bugs fixed:

MySQL Cluster: Multiple operations involving deletes followed by reads were not handled correctly.
Note
This issue could also affect MySQL Cluster Replication.
(Bug#28276)
Cluster API: Using NdbBlob::writeData() to write data in the middle of an existing blob value (that is, updating the value) could overwrite some data past the end of the data to be changed. (Bug#27018)
Incorrect handling of fragmentation in a node takeover during a restart could cause stale data to be copied to the starting node, leading eventually to failure of the node. (Bug#27434)
An incorrect assertion was made when sending a TCKEYFAILREF or TCKEYCONF message to a failed data node. (Bug#26814)

20.17.2.17. Changes in MySQL Cluster NDB 6.2.0 (5.1.16-ndb-6.2.0) (03 March 2007)

This is the first MySQL Cluster NDB 6.2 development release, based on version 6.2 of the NDBCLUSTER storage engine.

Obtaining MySQL Cluster NDB 6.2. You can download the latest MySQL Cluster NDB 6.2 source code and binaries for supported platforms from http://dev.mysql.com/downloads/cluster.

This Beta release incorporates bugfixes and changes made in previous MySQL Cluster releases, as well as all bugfixes and feature changes which were added in mainline MySQL 5.1 through MySQL 5.1.16 (see Section C.1.13, “Changes in MySQL 5.1.16 (26 February 2007)”).

Note

Please refer to our bug database at http://bugs.mysql.com/ for more details about the individual bugs fixed in this version.

Functionality added or changed:

MySQL Cluster: An --ndb-wait-connected option has been added for mysqld, which causes mysqld to wait for the specified amount of time to connect to the cluster before starting to accept MySQL client connections.
Cluster API: The Ndb::startTransaction() method now provides an alternative interface for starting a transaction. See Ndb::startTransaction(), for more information.
Cluster API: Methods were added to the Ndb_cluster_connection class to faciliate iterating over existing Ndb objects. See ndb_cluster_connection::get_next_ndb_object(), for more information.
It is now possible to disable arbitration by setting ArbitrationRank equal to 0 on all nodes.
A new TcpBind_INADDR_ANY configuration parameter allows data nodes node to bind INADDR_ANY instead of a hostname or IP address in the config.ini file.
Memory allocation has been improved on 32-bit architectures that enables using close to 3GB for DataMemory and IndexMemory combined.

20.17.3. Changes in MySQL Cluster NDB 6.1

20.17.3.1. Changes in MySQL Cluster NDB 6.1.23 (5.1.15-ndb-6.1.23) (20 November 2007)
20.17.3.2. Changes in MySQL Cluster NDB 6.1.22 (5.1.15-ndb-6.1.22) (19 October 2007)
20.17.3.3. Changes in MySQL Cluster NDB 6.1.21 (5.1.15-ndb-6.1.21) (01 October 2007)
20.17.3.4. Changes in MySQL Cluster NDB 6.1.20 (5.1.15-ndb-6.1.20) (14 September 2007)
20.17.3.5. Changes in MySQL Cluster NDB 6.1.19 (5.1.15-ndb-6.1.19) (01 August 2007)
20.17.3.6. Changes in MySQL Cluster NDB 6.1.18 (5.1.15-ndb-6.1.18) (Not released)
20.17.3.7. Changes in MySQL Cluster NDB 6.1.17 (5.1.15-ndb-6.1.17) (03 July 2007)
20.17.3.8. Changes in MySQL Cluster NDB 6.1.16 (5.1.15-ndb-6.1.16) (29 June 2007)
20.17.3.9. Changes in MySQL Cluster NDB 6.1.15 (5.1.15-ndb-6.1.15) (20 June 2007)
20.17.3.10. Changes in MySQL Cluster NDB 6.1.14 (5.1.15-ndb-6.1.14) (19 June 2007)
20.17.3.11. Changes in MySQL Cluster NDB 6.1.13 (5.1.15-ndb-6.1.13) (15 June 2007)
20.17.3.12. Changes in MySQL Cluster NDB 6.1.12 (5.1.15-ndb-6.1.12) (13 June 2007)
20.17.3.13. Changes in MySQL Cluster NDB 6.1.11 (5.1.15-ndb-6.1.11) (06 June 2007)
20.17.3.14. Changes in MySQL Cluster NDB 6.1.10 (5.1.15-ndb-6.1.10) (30 May 2007)
20.17.3.15. Changes in MySQL Cluster NDB 6.1.9 (5.1.15-ndb-6.1.9) (24 May 2007)
20.17.3.16. Changes in MySQL Cluster NDB 6.1.8 (5.1.15-ndb-6.1.8) (05 May 2007)
20.17.3.17. Changes in MySQL Cluster NDB 6.1.7 (5.1.15-ndb-6.1.7) (05 May 2007)
20.17.3.18. Changes in MySQL Cluster NDB 6.1.6 (5.1.15-ndb-6.1.6) (Not released)
20.17.3.19. Changes in MySQL Cluster NDB 6.1.5 (5.1.15-ndb-6.1.5) (15 March 2007)
20.17.3.20. Changes in MySQL Cluster NDB 6.1.4 (5.1.15-ndb-6.1.4) (09 March 2007))
20.17.3.21. Changes in MySQL Cluster NDB 6.1.3 (5.1.15-ndb-6.1.3) (25 February 2007)
20.17.3.22. Changes in MySQL Cluster NDB 6.1.2 (5.1.15-ndb-6.1.2) (07 February 2007)
20.17.3.23. Changes in MySQL Cluster NDB 6.1.1 (5.1.15-ndb-6.1.1) (01 February 2007)
20.17.3.24. Changes in MySQL Cluster NDB 6.1.0 (5.1.14-ndb-6.1.0) (20 December 2006)

This section contains change history information for MySQL Cluster releases based on version 6.1 of the NDBCLUSTER storage engine.

For an overview of new features added in MySQL Cluster NDB 6.1, see Section 20.15.2, “Features Added in MySQL Cluster NDB 6.1”.

20.17.3.1. Changes in MySQL Cluster NDB 6.1.23 (5.1.15-ndb-6.1.23) (20 November 2007)

This is a bugfix release, fixing recently discovered bugs in the previous MySQL Cluster NDB 6.1 release.

MySQL Cluster NDB 6.1 no longer in development. MySQL Cluster NDB 6.1 (formerly known as “MySQL Cluster Carrier Grade Edition 6.1.x”) is no longer being developed or maintained; if you are using a MySQL Cluster NDB 6.1 release, you should consider upgrading to MySQL Cluster NDB 6.2 or 6.3.

This Beta release incorporates all bugfixes and changes made in previous MySQL Cluster NDB 6.1 releases, as well as all bugfixes and feature changes which were added in mainline MySQL 5.1 through MySQL 5.1.15 (see Section C.1.14, “Changes in MySQL 5.1.15 (25 January 2007)”).

Note

Please refer to our bug database at http://bugs.mysql.com/ for more details about the individual bugs fixed in this version.

Bugs fixed:

MySQL Cluster: The NDB storage engine code was not safe for strict-alias optimization in gcc 4.2.1. (Bug#31761)
Cluster Replication: Under certain conditions, the slave stopped processing relay logs. This resulted in the logs never being cleared and the slave eventually running out of disk space. (Bug#31958)

20.17.3.2. Changes in MySQL Cluster NDB 6.1.22 (5.1.15-ndb-6.1.22) (19 October 2007)

This is a bugfix release, fixing recently discovered bugs in the previous MySQL Cluster NDB 6.1 release.

Note

Please refer to our bug database at http://bugs.mysql.com/ for more details about the individual bugs fixed in this version.

Bugs fixed:

MySQL Cluster: It was possible in some cases for a node group to be “lost” due to missed local checkpoints following a system restart. (Bug#31525)
Cluster Replication: A node failure during replication could lead to buckets out of order; now active subscribers are checked for, rather than empty buckets. (Bug#31701)

20.17.3.3. Changes in MySQL Cluster NDB 6.1.21 (5.1.15-ndb-6.1.21) (01 October 2007)

This is a bugfix release, fixing recently discovered bugs in the previous MySQL Cluster NDB 6.1 release.

Note

Please refer to our bug database at http://bugs.mysql.com/ for more details about the individual bugs fixed in this version.

Bugs fixed:

MySQL Cluster: A node failure during a local checkpoint could lead to a subsequent failure of the cluster during a system restart. (Bug#31257)
MySQL Cluster: A cluster restart could sometimes fail due to an issue with table IDs. (Bug#30975)

20.17.3.4. Changes in MySQL Cluster NDB 6.1.20 (5.1.15-ndb-6.1.20) (14 September 2007)

This is a bugfix release, fixing recently discovered bugs in the previous MySQL Cluster NDB 6.1 release.

Note

Please refer to our bug database at http://bugs.mysql.com/ for more details about the individual bugs fixed in this version.

Bugs fixed:

Cluster Replication: Incorrect handling of INSERT plus DELETE operations with regard to local checkpoints caused data node failures in multi-master replication setups. (Bug#30914)

20.17.3.5. Changes in MySQL Cluster NDB 6.1.19 (5.1.15-ndb-6.1.19) (01 August 2007)

This is a bugfix release, fixing recently discovered bugs in the previous MySQL Cluster NDB 6.1 release.

Note

Please refer to our bug database at http://bugs.mysql.com/ for more details about the individual bugs fixed in this version.

Functionality added or changed:

MySQL Cluster: Whenever a TCP send buffer is over 80% full, temporary error 1218 (Send Buffers overloaded in NDB kernel) is now returned. See SendBufferMemory for more information.
An INFO event is now sent if the time between global checkpoints is excessive, or if DUMP 7901 is issued in the management client.

20.17.3.6. Changes in MySQL Cluster NDB 6.1.18 (5.1.15-ndb-6.1.18) (Not released)

This is a bugfix release, fixing recently discovered bugs in the previous MySQL Cluster NDB 6.1 release.

Note

Please refer to our bug database at http://bugs.mysql.com/ for more details about the individual bugs fixed in this version.

Bugs fixed:

MySQL Cluster: When restarting a data node, queries could hang during that node's start phase 5, and continue only after the node had entered phase 6. (Bug#29364)
Disk Data: Disk data meta-information that existed in ndbd might not be visible to mysqld. (Bug#28720)
Disk Data: The number of free extents was incorrectly reported for some tablespaces. (Bug#28642)
Storage engine error conditions in row-based replication were not correctly reported to the user. (Bug#29570)

20.17.3.7. Changes in MySQL Cluster NDB 6.1.17 (5.1.15-ndb-6.1.17) (03 July 2007)

This is a bugfix release, fixing recently discovered bugs in the previous MySQL Cluster NDB 6.1 release.

Note

Please refer to our bug database at http://bugs.mysql.com/ for more details about the individual bugs fixed in this version.

Functionality added or changed:

Cluster Replication: Batching of updates on cluster replication slaves, enabled using the --slave-allow-batching option for mysqld.

Bugs fixed:

MySQL Cluster: Replica redo logs were inconsistently handled during a system restart. (Bug#29354)

20.17.3.8. Changes in MySQL Cluster NDB 6.1.16 (5.1.15-ndb-6.1.16) (29 June 2007)

This is a bugfix release, fixing recently discovered bugs in the previous MySQL Cluster NDB 6.1 release.

Note

Please refer to our bug database at http://bugs.mysql.com/ for more details about the individual bugs fixed in this version.

Bugs fixed:

MySQL Cluster: When a node failed to respond to a COPY_GCI signal as part of a global checkpoint, the master node was killed instead of the node that actually failed. (Bug#29331)
MySQL Cluster: An invalid comparison made during REDO validation that could lead to an Error while reading REDO log condition. (Bug#29118)
MySQL Cluster: The wrong data pages were sometimes invalidated following a global checkpoint. (Bug#29067)
MySQL Cluster: If at least 2 files were involved in REDO invalidation, then file 0 of page 0 was not updated and so pointed to an invalid part of the redo log. (Bug#29057)
Disk Data: When dropping a page, the stack's bottom entry could sometime be left “cold” rather than “hot”, violating the rules for stack pruning. (Bug#29176)

20.17.3.9. Changes in MySQL Cluster NDB 6.1.15 (5.1.15-ndb-6.1.15) (20 June 2007)

This is a bugfix release, fixing recently discovered bugs in the previous MySQL Cluster NDB 6.1 release.

Note

Please refer to our bug database at http://bugs.mysql.com/ for more details about the individual bugs fixed in this version.

Bugs fixed:

MySQL Cluster: Memory corruption could occur due to a problem in the DBTUP kernel block. (Bug#29229)

20.17.3.10. Changes in MySQL Cluster NDB 6.1.14 (5.1.15-ndb-6.1.14) (19 June 2007)

This is a bugfix release, fixing recently discovered bugs in the previous MySQL Cluster NDB 6.1 release.

Note

Please refer to our bug database at http://bugs.mysql.com/ for more details about the individual bugs fixed in this version.

Bugs fixed:

MySQL Cluster: In the event that two data nodes in the same node group and participating in a GCP crashed before they had written their respective P0.sysfile files, QMGR could refuse to start, issuing an invalid Insufficient nodes for restart error instead. (Bug#29167)

20.17.3.11. Changes in MySQL Cluster NDB 6.1.13 (5.1.15-ndb-6.1.13) (15 June 2007)

This is a bugfix release, fixing recently discovered bugs in the previous MySQL Cluster NDB 6.1 release.

Note

Please refer to our bug database at http://bugs.mysql.com/ for more details about the individual bugs fixed in this version.

Functionality added or changed:

Read ahead was implemented for backups of Disk Data tables, resulting in a 10 to 15% increase in backup speed of Disk Data tables. (Bug#29099)

Bugs fixed:

Cluster API: NdbApi.hpp depended on ndb_global.h, which was not actually installed, causing the compilation of programs that used NdbApi.hpp to fail. (Bug#35853)

20.17.3.12. Changes in MySQL Cluster NDB 6.1.12 (5.1.15-ndb-6.1.12) (13 June 2007)

This is a bugfix release, fixing recently discovered bugs in the previous MySQL Cluster NDB 6.1 release.

Note

Please refer to our bug database at http://bugs.mysql.com/ for more details about the individual bugs fixed in this version.

Functionality added or changed:

MySQL Cluster: New cluster management client DUMP commands were added to aid in tracking transactions, scan operations, and locks. See DUMP 2350, DUMP 2352, and DUMP 2550.
Backup dump output was extended to provide more information.

Bugs fixed:

MySQL Cluster: It is now possible to set the maximum size of the allocation unit for table memory using the MaxAllocate configuration parameter. (Bug#29044)

20.17.3.13. Changes in MySQL Cluster NDB 6.1.11 (5.1.15-ndb-6.1.11) (06 June 2007)

This is a bugfix release, fixing recently discovered bugs in the previous MySQL Cluster NDB 6.1 release.

Note

Please refer to our bug database at http://bugs.mysql.com/ for more details about the individual bugs fixed in this version.

Functionality added or changed:

Important Change: MySQL Cluster: The TimeBetweenWatchdogCheckInitial configuration parameter was added to allow setting of a separate watchdog timeout for memory allocation during startup of the data nodes. See Section 20.3.4.5, “Defining Data Nodes”, for more information. (Bug#28899)
MySQL Cluster: A new configuration parameter ODirect causes NDB to attempt using O_DIRECT writes for LCP, backups, and redo logs, often lowering CPU usage.
It is now possible to set the size of redo log files (fragment log files) using the FragmentLogFileSize configuration parameter.

Bugs fixed:

MySQL Cluster: Having large amounts of memory locked caused swapping to disk. (Bug#28751)
MySQL Cluster: LCP files were not removed following an initial system restart. (Bug#28726)
Disk Data: Repeated INSERT and DELETE operations on a Disk Data table having one or more large VARCHAR columns could cause data nodes to fail. (Bug#20612)

20.17.3.14. Changes in MySQL Cluster NDB 6.1.10 (5.1.15-ndb-6.1.10) (30 May 2007)

This is a bugfix release, fixing recently discovered bugs in the previous MySQL Cluster NDB 6.1 release.

Note

Please refer to our bug database at http://bugs.mysql.com/ for more details about the individual bugs fixed in this version.

Functionality added or changed:

MySQL Cluster: A new times printout was added in the ndbd watchdog thread.
MySQL Cluster: Some unneeded printouts in the ndbd out file were removed.
The names of some log and other files were changed to avoid issues with the tar command's 99-character filename limit.

Bugs fixed:

MySQL Cluster: A regression in the heartbeat monitoring code could lead to node failure under high load. This issue affected MySQL 5.1.19 and MySQL Cluster NDB 6.1.10 only. (Bug#28783)
MySQL Cluster: A corrupt schema file could cause a File already open error. (Bug#28770)
MySQL Cluster: Setting InitialNoOpenFiles equal to MaxNoOfOpenFiles caused an error. This was due to the fact that the actual value of MaxNoOfOpenFiles as used by the cluster was offset by 1 from the value set in config.ini. (Bug#28749)
MySQL Cluster: A race condition could result when non-master nodes (in addition to the master node) tried to update active status due to a local checkpoint (that is, between NODE_FAILREP and COPY_GCIREQ events). Now only the master updates the active status. (Bug#28717)
MySQL Cluster: A fast global checkpoint under high load with high usage of the redo buffer caused data nodes to fail. (Bug#28653)
Disk Data: When loading data into a cluster following a version upgrade, the data nodes could forcibly shut down due to page and buffer management failures (that is, ndbrequire failures in PGMAN). (Bug#28525)

20.17.3.15. Changes in MySQL Cluster NDB 6.1.9 (5.1.15-ndb-6.1.9) (24 May 2007)

This is a bugfix release, fixing recently discovered bugs in the previous MySQL Cluster NDB 6.1 release.

Note

Please refer to our bug database at http://bugs.mysql.com/ for more details about the individual bugs fixed in this version.

Bugs fixed:

MySQL Cluster: When an API node sent more than 1024 signals in a single batch, NDB would process only the first 1024 of these, and then hang. (Bug#28443)
Disk Data: The cluster backup process scanned in ACC index order, which had bad effects for disk data. (Bug#28593)

20.17.3.16. Changes in MySQL Cluster NDB 6.1.8 (5.1.15-ndb-6.1.8) (05 May 2007)

This is a bugfix release, fixing recently discovered bugs in the previous MySQL Cluster NDB 6.1 release.

Note

Please refer to our bug database at http://bugs.mysql.com/ for more details about the individual bugs fixed in this version.

Bugs fixed:

MySQL Cluster: Local checkpoint files relating to dropped NDB tables were not removed. (Bug#28348)
MySQL Cluster: Repeated insertion of data generated by mysqldump into NDB tables could eventually lead to failure of the cluster. (Bug#27437)
Disk Data: Extremely large inserts into Disk Data tables could lead to data node failure in some circumstances. (Bug#27942)
Cluster API: In a multi-operation transaction, a delete operation followed by the insertion of an implicit NULL failed to overwrite an existing value. (Bug#20535)
Setting MaxNoOfTables very low and relative to DataMemory caused Out of memory in Ndb Kernel errors when inserting relatively small amounts of data into NDB tables. (Bug#24173)

20.17.3.17. Changes in MySQL Cluster NDB 6.1.7 (5.1.15-ndb-6.1.7) (05 May 2007)

This is a bugfix release, fixing recently discovered bugs in the previous MySQL Cluster NDB 6.1 release.

Note

Please refer to our bug database at http://bugs.mysql.com/ for more details about the individual bugs fixed in this version.

Functionality added or changed:

Cluster Replication: Incompatible Change: The schema for the ndb_apply_status table in the mysql system database has changed. When upgrading to this release from a previous MySQL Cluster NDB 6.x or mainline MySQL 5.1 release, you must drop the mysql.ndb_apply_status table, then restart the server in order for the table to be re-created with the new schema.
See Section 20.11.4, “Cluster Replication Schema and Tables”, for additional information.

Bugs fixed:

MySQL Cluster: The cluster waited 30 seconds instead of 30 milliseconds before reading table statistics. (Bug#28093)
MySQL Cluster: Under certain rare circumstances, ndbd could get caught in an infinite loop when one transaction took a read lock and then a second transaction attempted to obtain a write lock on the same tuple in the lock queue. (Bug#28073)
MySQL Cluster: Under some circumstances, a node restart could fail to update the Global Checkpoint Index (GCI). (Bug#28023)
MySQL Cluster: An INSERT followed by a delete DELETE on the same NDB table caused a memory leak. (Bug#27756)
This regression was introduced by Bug#20612
MySQL Cluster: Under certain rare circumstances performing a DROP TABLE or TRUNCATE on an NDB table could cause a node failure or forced cluster shutdown. (Bug#27581)
MySQL Cluster: Memory usage of a mysqld process grew even while idle. (Bug#27560)
MySQL Cluster: Performing a delete followed by an insert during a local checkpoint could cause a Rowid already allocated error. (Bug#27205)
Cluster Replication: Disk Data: An issue with replication of Disk Data tables could in some cases lead to node failure. (Bug#28161)
Disk Data: Changes to a Disk Data table made as part of a transaction could not be seen by the client performing the changes until the transaction had been committed. (Bug#27757)
Disk Data: When restarting a data node following the creation of a large number of Disk Data objects (approximately 200 such objects), the cluster could not assign a node ID to the restarting node. (Bug#25741)
Disk Data: Changing a column specification or issuing a TRUNCATE statement on a Disk Data table caused the table to become an in-memory table.
This fix supersedes an incomplete fix that was made for this issue in MySQL 5.1.15. (Bug#24667, Bug#25296)
Cluster Replication: Some queries that updated multiple tables were not backed up correctly. (Bug#27748)
Cluster Replication: It was possible for API nodes to begin interacting with the cluster subscription manager before they were fully connected to the cluster. (Bug#27728)
Cluster Replication: Under very high loads, checkpoints could be read or written with checkpoint indexes out of order. (Bug#27651)
Cluster API: An issue with the way in which the NdbDictionary::Dictionary::listEvents() method freed resources could sometimes lead to memory corruption. (Bug#27663)
mysqldump could not dump log tables. (Bug#26121)
The --with-readline option for configure did not work for commercial source packages, but no error message was printed to that effect. Now a message is printed. (Bug#25530)

20.17.3.18. Changes in MySQL Cluster NDB 6.1.6 (5.1.15-ndb-6.1.6) (Not released)

This is a bugfix release, fixing recently discovered bugs in the previous MySQL Cluster NDB 6.1 release.

Note

Please refer to our bug database at http://bugs.mysql.com/ for more details about the individual bugs fixed in this version.

Functionality added or changed:

Cluster Replication: Incompatible Change: The schema for the ndb_apply_status table in the mysql system database has changed. When upgrading to this release from a previous MySQL Cluster NDB 6.x or mainline MySQL 5.1 release, you must drop the mysql.ndb_apply_status table, then restart the server in order for the table to be re-created with the new schema.
See Section 20.11.4, “Cluster Replication Schema and Tables”, for additional information.

Bugs fixed:

MySQL Cluster: A data node failing while another data node was restarting could leave the cluster in an inconsistent state. In certain rare cases, this could lead to a race condition and the eventual forced shutdown of the cluster. (Bug#27466)
MySQL Cluster: It was not possible to set LockPagesInMainMemory equal to 0. (Bug#27291)
MySQL Cluster: A race condition could sometimes occur if the node acting as master failed while node IDs were still being allocated during startup. (Bug#27286)
MySQL Cluster: When a data node was taking over as the master node, a race condition could sometimes occur as the node was assuming responsibility for handling of global checkpoints. (Bug#27283)
MySQL Cluster: mysqld could crash shortly after a data node failure following certain DML operations. (Bug#27169)
MySQL Cluster: The same failed request from an API node could be handled by the cluster multiple times, resulting in reduced performance. (Bug#27087)
MySQL Cluster: The failure of a data node while restarting could cause other data nodes to hang or crash. (Bug#27003)
MySQL Cluster: mysqld processes would sometimes crash under high load.
Note
This fix improves on and replaces a fix for this bug that was made in MySQL Cluster NDB 6.1.5.
(Bug#26825)
Disk Data: DROP INDEX on a Disk Data table did not always move data from memory into the tablespace. (Bug#25877)
Cluster Replication: Trying to replicate a large number of frequent updates with a relatively small relay log (max-relay-log-size set to 1M or less) could cause the slave to crash. (Bug#27529)
Cluster API: An issue with the way in which the NdbDictionary::Dictionary::listEvents() method freed resources could sometimes lead to memory corruption. (Bug#27663)
Cluster API: A delete operation using a scan followed by an insert using a scan could cause a data node to fail. (Bug#27203)

20.17.3.19. Changes in MySQL Cluster NDB 6.1.5 (5.1.15-ndb-6.1.5) (15 March 2007)

This is a bugfix release, fixing recently discovered bugs in the previous MySQL Cluster NDB 6.1 release.

Note

Please refer to our bug database at http://bugs.mysql.com/ for more details about the individual bugs fixed in this version.

Functionality added or changed:

Cluster Replication: Incompatible Change: The schema for the ndb_apply_status table in the mysql system database has changed. When upgrading to this release from a previous MySQL Cluster NDB 6.x or mainline MySQL 5.1 release, you must drop the mysql.ndb_apply_status table, then restart the server in order for the table to be re-created with the new schema.
See Section 20.11.4, “Cluster Replication Schema and Tables”, for additional information.

Bugs fixed:

MySQL Cluster: Creating a table on one SQL node while in single user mode caused other SQL nodes to crash. (Bug#26997)
MySQL Cluster: mysqld processes would sometimes crash under high load.
Note
This fix was reverted in MySQL Cluster NDB 6.1.6.
(Bug#26825)
MySQL Cluster: An infinite loop in an internal logging function could cause trace logs to fill up with Unknown Signal type error messages and thus grow to unreasonable sizes. (Bug#26720)
Disk Data: When creating a log file group, setting INITIAL_SIZE to less than UNDO_BUFFER_SIZE caused data nodes to crash. (Bug#25743)

20.17.3.20. Changes in MySQL Cluster NDB 6.1.4 (5.1.15-ndb-6.1.4) (09 March 2007))

This is a bugfix release, fixing recently discovered bugs in the previous MySQL Cluster NDB 6.1 release.

Note

Please refer to our bug database at http://bugs.mysql.com/ for more details about the individual bugs fixed in this version.

Functionality added or changed:

MySQL Cluster: An --ndb-wait-connected option has been added for mysqld. It causes mysqld wait a specified amount of time to be connected to the cluster before accepting client connections. See here, for more information.
Cluster API: It is now possible to specify the transaction coordinator when starting a transaction. See Ndb::startTransaction(), for more information.
Cluster API: It is now possible to iterate over all existing Ndb objects using three new methods of the Ndb_cluster_connection class:
- lock_ndb_objects()
- get_next_ndb_object()
- unlock_ndb_objects()
For more information about these methods and their use, see ndb_cluster_connection::get_next_ndb_object(), in the MySQL Cluster API Guide.
Data node memory allocation has been improved. On 32-bit platforms, it should now be possible to use close to 3GB RAM for IndexMemory and DataMemory combined.

Bugs fixed:

MySQL Cluster: Using only the --print_data option (and no other options) with ndb_restore caused ndb_restore to fail. (Bug#26741)
This regression was introduced by Bug#14612
MySQL Cluster: An inadvertent use of unaligned data caused ndb_restore to fail on some 64-bit platforms, including Sparc and Itanium-2. (Bug#26739)
Assigning a node ID greater than 63 to an SQL node caused an out of bounds error in mysqld. It should now be possible to assign to SQL nodes node IDs up to 255. (Bug#26663)

20.17.3.21. Changes in MySQL Cluster NDB 6.1.3 (5.1.15-ndb-6.1.3) (25 February 2007)

This is a bugfix release, fixing recently discovered bugs in the previous MySQL Cluster NDB 6.1 release.

Note

Please refer to our bug database at http://bugs.mysql.com/ for more details about the individual bugs fixed in this version.

Functionality added or changed:

MySQL Cluster: The ndbd_redo_log_reader utility is now part of the default build. For more information, see Section 20.10.11, “ndbd_redo_log_reader — Check and Print Content of Cluster Redo Log”.
MySQL Cluster: The ndb_show_tables utility now displays information about table events. See Section 20.10.14, “ndb_show_tables — Display List of NDB Tables”, for more information.
Cluster API: A new listEvents() method has been added to the Dictionary class. See Dictionary::listEvents(), for more information.
It is now possible to disable arbitration by setting ArbitrationRank=0 on all management and SQL nodes.

Bugs fixed:

MySQL Cluster: An invalid pointer was returned following a FSCLOSECONF signal when accessing the REDO logs during a node restart or system restart. (Bug#26515)
MySQL Cluster: The InvalidUndoBufferSize error used the same error code (763) as the IncompatibleVersions error. InvalidUndoBufferSize now uses its own error code (779). (Bug#26490)
MySQL Cluster: The failure of a data node when restarting it with --initial could lead to failures of subsequent data node restarts. (Bug#26481)
MySQL Cluster: Takeover for local checkpointing due to multiple failures of master nodes was sometimes incorrectly handled. (Bug#26457)
MySQL Cluster: The LockPagesInMainMemory parameter was not read until after distributed communication had already started between cluster nodes. When the value of this parameter was 1, this could sometimes result in data node failure due to missed heartbeats. (Bug#26454)
MySQL Cluster: Under some circumstances, following the restart of a management node, all data nodes would connect to it normally, but some of them subsequently failed to log any events to the management node. (Bug#26293)
MySQL Cluster: No appropriate error message was provided when there was insufficient REDO log file space for the cluster to start. (Bug#25801)
MySQL Cluster: A memory allocation failure in SUMA (the cluster Subscription Manager) could cause the cluster to crash. (Bug#25239)
MySQL Cluster: The message Error 0 in readAutoIncrementValue(): no Error was written to the error log whenever SHOW TABLE STATUS was performed on a Cluster table that did not have an AUTO_INCREMENT column.
Note
This improves on and supersedes an earlier fix that was made for this issue in MySQL 5.1.12.
(Bug#21033)
Disk Data: A memory overflow could occur with tables having a large amount of data stored on disk, or with queries using a very high degree of parallelism on Disk Data tables. (Bug#26514)
Disk Data: Use of a tablespace whose INITIAL_SIZE was greater than 1 GB could cause the cluster to crash. (Bug#26487)

20.17.3.22. Changes in MySQL Cluster NDB 6.1.2 (5.1.15-ndb-6.1.2) (07 February 2007)

This is a bugfix release, fixing recently discovered bugs in the previous MySQL Cluster NDB 6.1 release.

Note

Please refer to our bug database at http://bugs.mysql.com/ for more details about the individual bugs fixed in this version.

Bugs fixed:

MySQL Cluster: Using node IDs greater than 48 could sometimes lead to incorrect memory access and a subsequent forced shutdown of the cluster. (Bug#26267)

20.17.3.23. Changes in MySQL Cluster NDB 6.1.1 (5.1.15-ndb-6.1.1) (01 February 2007)

This is a bugfix release, fixing recently discovered bugs in the previous MySQL Cluster NDB 6.1 release.

This Beta release incorporates all bugfixes and changes made in the previous MySQL Cluster NDB 6.1 release, as well as all bugfixes and feature changes which were added in mainline MySQL 5.1 through MySQL 5.1.15 (see Section C.1.14, “Changes in MySQL 5.1.15 (25 January 2007)”).

Note

Please refer to our bug database at http://bugs.mysql.com/ for more details about the individual bugs fixed in this version.

Functionality added or changed:

MySQL Cluster: A single cluster can now support up to 255 API nodes, including MySQL servers acting as SQL nodes. See Section 20.14.8, “Issues Exclusive to MySQL Cluster”, for more information.

Bugs fixed:

MySQL Cluster: A memory leak could cause problems during a node or cluster shutdown or failure. (Bug#25997)
Cluster API: Disk Data: A delete and a read performed in the same operation could cause one or more data nodes to crash. This could occur when the operation affected more than 5 columns concurrently, or when one or more of the columns was of the VARCHAR type and was stored on disk. (Bug#25794)
An element could sometimes be inserted twice into the hash table, causing a data node to crash. (Bug#25286)

20.17.3.24. Changes in MySQL Cluster NDB 6.1.0 (5.1.14-ndb-6.1.0) (20 December 2006)

This is the first MySQL Cluster NDB 6.1 release, incorporating new features and bugfixes made for the NDBCLUSTER storage engine made since branching from MySQL 5.1.14 standard (see Section C.1.15, “Changes in MySQL 5.1.14 (05 December 2006)”).

Note

Please refer to our bug database at http://bugs.mysql.com/ for more details about the individual bugs fixed in this version.

Functionality added or changed:

MySQL Cluster: A new configuration parameter MemReportFrequency allows for additional control of data node memory usage. Previously, only warnings at predetermined percentages of memory allocation were given; setting this parameter allows for that behavior to be overridden. For more information, see Section 20.3.4.5, “Defining Data Nodes”.

Bugs fixed:

MySQL Cluster: When a data node was shut down using the management client STOP command, a connection event (NDB_LE_Connected) was logged instead of a disconnection event (NDB_LE_Disconnected). (Bug#22773)
MySQL Cluster: SELECT statements with a BLOB or TEXT column in the selected column list and a WHERE condition including a primary key lookup on a VARCHAR primary key produced empty result sets. (Bug#19956)
Disk Data: MEDIUMTEXT columns of Disk Data tables were stored in memory rather than on disk, even if the columns were not indexed. (Bug#25001)
Disk Data: Performing a node restart with a newly dropped Disk Data table could lead to failure of the node during the restart. (Bug#24917)
Disk Data: When restoring from backup a cluster containing any Disk Data tables with hidden primary keys, a node failure resulted which could lead to a crash of the cluster. (Bug#24166)
Disk Data: Repeated CREATE, DROP, or TRUNCATE in various combinations with system restarts between these operations could lead to the eventual failure of a system restart. (Bug#21948)
Disk Data: Extents that should have been available for re-use following a DROP TABLE operation were not actually made available again until after the cluster had performed a local checkpoint. (Bug#17605)
Cluster API: Invoking the NdbTransaction::execute() method using execution type Commit and abort option AO_IgnoreError could lead to a crash of the transaction coordinator (DBTC). (Bug#25090)
Cluster API: A unique index lookup on a non-existent tuple could lead to a data node timeout (error 4012). (Bug#25059)
Cluster API: When using the NdbTransaction::execute() method, a very long timeout (greater than 5 minutes) could result if the last data node being polled was disconnected from the cluster. (Bug#24949)
Cluster API: Due to an error in the computation of table fragment arrays, some transactions were not executed from the correct starting point. (Bug#24914)
Under certain rare circumstances, local checkpoints were not performed properly, leading to an inability to restart one or more data nodes. (Bug#24664)

Prev		Next
Chapter 19. Replication	Home	Chapter 21. Partitioning

Node	IP Address
Management (MGM) node	192.168.0.10
MySQL server (SQL) node	192.168.0.20
Data (NDBD) node "A"	192.168.0.30
Data (NDBD) node "B"	192.168.0.40

Version Introduced	5.1.12
Variable Name	`ndb_use_copying_alter_table`
Variable Scope	Both
Dynamic Variable	No