21.5.14.3 Adding NDB Cluster Data Nodes Online: Detailed Example

In this section we provide a detailed example illustrating how
to add new NDB Cluster data nodes online, starting with an NDB
Cluster having 2 data nodes in a single node group and
concluding with a cluster having 4 data nodes in 2 node groups.

Starting configuration.
For purposes of illustration, we assume a minimal
configuration, and that the cluster uses a
config.ini file containing only the
following information:

We have left a gap in the sequence between data node IDs and
other nodes. This make it easier later to assign node IDs that
are not already in use to data nodes which are newly added.

We also assume that you have already started the cluster using
the appropriate command line or my.cnf
options, and that running
SHOW in the management
client produces output similar to what is shown here:

The memory usage and related information shown later in this
section was generated after inserting approximately 50000 rows
into this table.

Note

In this example, we show the single-threaded
ndbd being used for the data node
processes. You can also apply this example, if you are using
the multi-threaded ndbmtd by substituting
ndbmtd for ndbd wherever
it appears in the steps that follow.

Step 1: Update configuration file.
Open the cluster global configuration file in a text editor
and add [ndbd] sections corresponding to
the 2 new data nodes. (We give these data nodes IDs 3 and 4,
and assume that they are to be run on host machines at
addresses 192.168.0.3 and 192.168.0.4, respectively.) After
you have added the new sections, the contents of the
config.ini file should look like what is
shown here, where the additions to the file are shown in bold
type:

Because shutting down the management server causes the
management client to terminate, you must start the
management server from the system shell. For simplicity, we
assume that config.ini is in the same
directory as the management server binary, but in practice,
you must supply the correct path to the configuration file.
You must also supply the
--reload or
--initial option so that
the management server reads the new configuration from the
file rather than its configuration cache. If your
shell's current directory is also the same as the
directory where the management server binary is located,
then you can invoke the management server as shown here:

After issuing each X
RESTART command, wait until the management client
reports Node X: Started
(version ...)before proceeding
any further.

You can verify that all existing data nodes were restarted using
the updated configuration by checking the
ndbinfo.nodes table in the
mysql client.

Step 4: Perform a rolling restart of all cluster API nodes.
Shut down and restart each MySQL server acting as an SQL node
in the cluster using mysqladmin shutdown
followed by mysqld_safe (or another startup
script). This should be similar to what is shown here, where
password is the MySQL
root password for a given MySQL server
instance:

Of course, the exact input and output depend on how and where
MySQL is installed on the system, as well as which options you
choose to start it (and whether or not some or all of these
options are specified in a my.cnf file).

Step 5: Perform an initial start of the new data nodes.
From a system shell on each of the hosts for the new data
nodes, start the data nodes as shown here, using the
--initial option:

shell> ndbd -c 192.168.0.10 --initial

Note

Unlike the case with restarting the existing data nodes, you
can start the new data nodes concurrently; you do not need to
wait for one to finish starting before starting the other.

Wait until both of the new data nodes have started
before proceeding with the next step. Once the new
data nodes have started, you can see in the output of the
management client SHOW
command that they do not yet belong to any node group (as
indicated with bold type here):

Step 6: Create a new node group.
You can do this by issuing a CREATE
NODEGROUP command in the cluster management client.
This command takes as its argument a comma-separated list of
the node IDs of the data nodes to be included in the new node
group, as shown here:

ndb_mgm> CREATE NODEGROUP 3,4
Nodegroup 1 created

By issuing SHOW again, you
can verify that data nodes 3 and 4 have joined the new node
group (again indicated in bold type):

Step 7: Redistribute cluster data.
When a node group is created, existing data and indexes are
not automatically distributed to the new node group's
data nodes, as you can see by issuing the appropriate
REPORT command in the
management client:

By using ndb_desc with the
-p option, which causes the output to include
partitioning information, you can see that the table still uses
only 2 partitions (in the Per partition info
section of the output, shown here in bold text):

ALTER TABLE ... ALGORITHM=INPLACE, REORGANIZE
PARTITION does not work on tables that were created
with the MAX_ROWS option. Instead, use
ALTER TABLE ... ALGORITHM=INPLACE,
MAX_ROWS=... to reorganize such tables.

Keep in mind that using MAX_ROWS to set the
number of partitions per table is deprecated in NDB 7.5.4 and
later, where you should use
PARTITION_BALANCE instead; see
Section 13.1.18.10, “Setting NDB_TABLE Options”, for
more information.

After issuing the statement ALTER TABLE ips
ALGORITHM=INPLACE, REORGANIZE PARTITION, you can see
using ndb_desc that the data for this table
is now stored using 4 partitions, as shown here (with the
relevant portions of the output in bold type):

Normally, ALTER
TABLE table_name
[ALGORITHM=INPLACE,] REORGANIZE PARTITION is used
with a list of partition identifiers and a set of partition
definitions to create a new partitioning scheme for a table
that has already been explicitly partitioned. Its use here to
redistribute data onto a new NDB Cluster node group is an
exception in this regard; when used in this way, no other
keywords or identifiers follow REORGANIZE
PARTITION.

The INFORMATION_SCHEMA.TABLES.ENGINE value
for an NDB Cluster table is always
NDBCLUSTER, regardless of whether
the CREATE TABLE statement used to create
the table (or ALTER TABLE
statement used to convert an existing table from a different
storage engine) used NDB or
NDBCLUSTER in its
ENGINE option.

You can see after performing these statements in the output of
ALL REPORT MEMORY that the
data and indexes are now redistributed between all cluster data
nodes, as shown here:

Alternative procedure, without rolling restart.
It is possible to avoid the need for a rolling restart by
configuring the extra data nodes, but not starting them, when
first starting the cluster. We assume, as before, that you
wish to start with two data nodes—nodes 1 and 2—in
one node group and later to expand the cluster to four data
nodes, by adding a second node group consisting of nodes 3 and
4:

The data nodes to be brought online at a later time (nodes 3 and
4) can be configured with
NodeGroup = 65536, in
which case nodes 1 and 2 can each be started as shown here:

shell> ndbd -c 192.168.0.10 --initial

The data nodes configured with
NodeGroup = 65536 are
treated by the management server as though you had started nodes
1 and 2 using --nowait-nodes=3,4
after waiting for a period of time determined by the setting for
the
StartNoNodeGroupTimeout
data node configuration parameter. By default, this is 15
seconds (15000 milliseconds).

Note

StartNoNodegroupTimeout
must be the same for all data nodes in the cluster; for this
reason, you should always set it in the [ndbd
default] section of the
config.ini file, rather than for
individual data nodes.

When you are ready to add the second node group, you need only
perform the following additional steps:

Start data nodes 3 and 4, invoking the data node process
once for each new node: