WL#3574: Online backup: Service API for Metadata Objects II

Rationale
---------
To make a alpha-quality interface between server kernel and backup kernel.
Stabilizing this interface is WL#4264.
Overview
--------
The online backup system requires server functionality to permit the access to
metadata concerning database objects (e.g., CREATE DATABASE). This
functionality is not unique to online backup and therefore shall be designed to
support access to metadata using a generalized interface.
Background
----------
One of the driving goals of the online backup system is to minimize reliance on
functionality in the MySQL server proper. That is, to abstract the backup code
from the server code as much as possible in order to avoid conditions where
changes in one breaks the other. While a complete abstraction is not possible
(the backup code needs to communicate with the server in a variety of ways), it
is possible to isolate the affects of changes to either the online backup or
server code by using an intermediate layer. The term "service interface" has
been given to this layer to indicate its goal: to provide a buffer between the
services needed by online backup and how those services are provided by the
server.
Requirements
------------
The purpose of this worklog is to establish a service interface that can be
used to gather metadata about database objects, iterate through the various
types of objects supported, and to create database objects on restore. The
deliverable for this worklog is a service interface that implements these
features (at a minimum) for the following database objects.
* Trigger (WL#3582) [40]
* View [37]
* Stored Procedure [38]
* Stored Function [39]
* Event [41]
Note: The [] references a number assigned to each object described in WL#3713.
There is a complete list of objects described in WL#3713.

The service interface shall be designed to allow the following operations
concerning metadata.
* The retrieval of metadata concerning a specific object.
* The recreation of a specific object using the retrieved metadata.
* Enumeration for each object supported and for all objects of a given type.
* Abstractions of the DDL blocker to permit isolation of DDL blocker.
* Dependency checking for objects to discover objects that need to be
created/exist before a given object is created.
The service interface shall be designed using abstractions to represent the
above functionality.
The retrieval of metadata shall be called 'serialize' and the output of the
serialize() method called the serialization of the object.
The serialization of the object can be used to create an instance of the
service interface class and this process shall be called materialize.
The process of using the materialized serialization and creating a new object
on the server shall be referred as executing the serialization.
Thus, an object can be serialized for storage in the online backup and
materialized on restore and the serialization executed to create the database
object on the server.
For example, the serialization, e.g. db->serialize() of a database shall
produce a serialization that can be stored in the online backup image file.
Upon restore, the service interface can be used to materialize this data using
db->materialize(serialized_data). To create the object on the server (to create
the database), the service interface would allow db->execute() to create the
database 'xyz'. All other objects supported in this worklog shall use this
mechanism.
Architecture
------------
The high-level architecture is given in the following diagram.
+-----------+ +---------------+
| <other> | | online backup |
|-----------+ +---------------+
| |
| |
+--------+----------+
|
|
+--------+----------+
| service interface |
+--------+----------+
|
|
+-------+---------+
| server (mysqld) |
+-------+---------+
This diagram illustrates that the service interface provides an abstraction of
server functionality for calling applications such as mysqldump or online
backup. The advantage is that the server interface can be used to isolate
changes to the server from the applications and vice-versa. Furthermore, the
service interface can be improved independently of either caller or callee.
Decision
--------
A decision was made to serialize the object metadata using the currently
available show_create_* methods. This implementation of the serialization shall
produce a string obtained in the same way as in SHOW CREATE statements.
However, there is a problem related to the fact that output of SHOW CRATE
statement depends on the setting of sql_mode variable (see BUG#33570 and
BUG#33571). If the mode is anything else than the "native" mysql mode then
CREATE statements can miss some important clauses (like "ENGINE=xxx" for a
table). Also, when object is re-created using the full CREATE statement for it,
sql_mode must be set to "native" as otherwise not all clauses will be
understood by the server. All this must be taken into account when implementing
serialization via CREATE statements.
Note: In the future, the serialization may not be based on CREATE statements
and therefore the caller of the service interface should make no assumptions
about the content of the serialization other than what has been stated in the
above sections.
Known Limitation
----------------
When objects are described by CREATE statements, they are created by executing
these statements. Currently this is done by passing CREATE statement to
mysql_parse() function. However, this function can not be called from inside a
stored routine - if it happens, an assertion is violated and server crashes
(see BUG#33563). Therefore, it has been decided to not permit the inclusion of
the BACKUP or RESTORE command inside a stored routing. See solution for
BUG#33563 for more details.
Future Work
-----------
It has been suggested that the ownership of the service interface could be the
runtime team. If ownership of this interface is transferred to the runtime
team, it is requested that they refactor the implementation of the service
interface however they see fit. It was suggested that the
serialize/materialize/execute mechanism is sufficient to allow the
implementation to be changed to use something other than CREATE SQL statements.
In such a case, the design shall incorporate a mechanism to allow
identification of versions of the service interface.
It has been decided to leave out the complete implementation of dependency
checking for views and other objects. Additional work is needed to understand
all of the requirements for dependency checking WRT backup and restore. See
WL#4211 and WL#4225.

The service interface shall be implemented in the ./sql source folder and named
si_objects.*. These source files shall comprise the interface specification and
code for the following requirements.
Object Class
------------
The service interface shall implement a general object class named Obj. This
class shall have, at a minimum, the following methods. This shall become an
abstract class from which each database object support (trigger, event, etc.)
shall derive.
class Obj
{
public:
virtual bool serialize(THD *thd, String *serialialization) = 0;
virtual const String *get_name() = 0;
virtual const String *get_db_name() = 0;
virtual bool execute(THD *thd) = 0;
virtual ~Obj() { }
private:
virtual bool materialize(uint serialization_version,
const String *serialialization) = 0;
virtual bool drop(THD *thd) = 0;
};
In addition to the serialize(), materialize(), and execute() methods discussed
in the high level specification, the methods get_name() and get_db_name() are
provided to allow identification of the object instance. The method drop() is
included to permit a general call to destroy an object on the server. For
example to drop a database, a child class could be materialized for the
database then drop() can be called to execute DROP DATABASE ... on the server.
The serialization mechanism shall be implemented to support versioning of the
interface. The initial version shall be 0 and it shall not be considered in
future revisions for backward compatibility. The reason is version 0 implements
serialization using CREATE commands which may not be the best implementation
for future plans for the server. Thus, starting with version 1, the service
interface shall require the version number present when services are requested
and shall provide backward compatibility in future versions of the service
interface.
Iterators
---------
The iterators for the database objects shall be a simple class that permits the
instantiation of the iterator and a forward-only increment mechanism. The
abstract class for the iterator functionality is shown below.
class ObjIterator
{
public:
ObjIterator() { }
virtual Obj *next() = 0;
public:
virtual ~ObjIterator() { }
};
Versioning of Service Interface
-------------------------------
The service interface shall use as its versioning the version number of the
server. When a caller uses the serialization mechanism, she must record the
version number of the server and store it with the serialization data. Upon
materialization, the caller passes the version number in the method call as
shown.
materialize(uint serialization_version,
const String *serialialization);
This way, the service interface shall be capable of supporting services for
multiple releases. Note that the first release of this interface shall use
0 in place of the version number as described above.
Convenience Methods
-------------------
The service interface shall implement convenience methods to reduce the burden
of the caller to create instances of objects, iterators, and materialization.
A method for each database object supported shall be created to allow the
instantiation of a single object shall be provided using the following or
similar method.
Obj *get_database(const String *db_name);
A method for each database object supported shall be created to allow the
creation of an object iterator. For example, to get an iterator for all of the
databases on a server, the service interface shall implement the following or
similar method.
ObjIterator *get_databases(THD *thd);
A method for each database object supported shall be created to allow the
materialization of a single object. The following or similar method shall be
implemented.
Obj *materialize_database(const String *db_name,
uint serialization_version,
const String *serialialization);
DDL Blocker
-----------
The following methods shall be implemented to abstract the DDL blocker methods
already implemented in the server. These methods are necessary to allow the
replacement of the DDL blocker code in the future without affecting the code in
the caller.
bool ddl_blocker_enable(THD *thd);
void ddl_blocker_disable();
void ddl_blocker_exception_on(THD *thd);
void ddl_blocker_exception_off(THD *thd);
Helper Methods
--------------
Aside from the methods listed above, the following helper methods shall be
implemented in the service interface. Their function is shown above each.
/*
Check to see if db_name is an internal database such as mysql
of information_schema.
*/
bool is_internal_db_name(const String *db_name);
/*
Check if the given directory actually exists.
*/
bool check_db_existence(const String *db_name);