Tag: WordPress Plugins

There is a constant need for a tool to replicate the production site with minimum data on a test setup with a completely functional theme. We all have come across a situation where we wanted to test a piece of code on a setup that is exactly like our production environment. And we start by setting up the theme and a test server and then import data from production into it.

Production databases are huge, having 3 – 10GB or more of data for each site, and if we have multiple sites it easily becomes approximately 60–100GB of data that we have to download. If we want to create a setup for each developer we are creating that many environments with such huge data occupying space and increasing redundancy. We always wanted to have a much smaller dataset that represents each type of content from the production server in our test site. Also, if we launch a new section or content type, we create dummy content for testing but most often than not its different than the way editors use that section in the wild to create content and we want to have a piece of that dataset on our local setup to understand the use of that section and test it properly.

Keeping in mind the above scenario, I was asked by Amit Sannad to develop a plugin using the WordPress REST API so that we can import data from production site into any WordPress setup of our choice which would give us a theme and a mini site that is completely functional and mirrors production environment. This concept of having a plugin to populate minimum required data is a good way to fill our test sites with all the features, new content types from custom post types and also it is a great idea that this plugin allows us to easily refresh data on a dev site.

I started with the JSON REST API plugin. It is a very useful plugin and very well documented here. It lets you import most of the built-in features such as posts, page, users, tags, categories etc… It also allows you to extend it and add custom endpoints to fetch any custom data type which is not available by default such as Custom Post Types and Custom Taxonomies. But soon we realised that it is not part of the WordPress.com core yet and is in beta release. It required us to install as a plugin and then develop, customize, and deploy the entire code.

We host our sites on WordPress.com VIP and did not want to develop against something that was not yet part of the WordPress.com core. So I came across WordPress.com’s REST API — it’s different than JSON REST API, it’s public, and allows authentication based retrieval of data. That became an important factor in choosing an API that is well supported by WordPress.com VIP and provides authentication when we want to pull sensitive data.

Using WordPress.com’s REST API, we can fetch posts, tags, categories, attachments, and comments. However we require OAuth 2 Authentication for pulling users and menus. When I got familiar with OAuth2 I learned that it requires 2 step authentication which would provide an access token that can be saved and used subsequently. It is a secure way of trying to connect to a production site and getting data. If we are on a familiar dev/QA environment and know the users that would use this site we can have the access token fetched once and saved in the database and use it in our request header as Authentication Bearer.

Then comes the data import for non built-in types. A key breakthrough here was when I learned from WordPress VIP that we have a filter to whitelist the custom post types and pull them using the post endpoint using the ‘type’ query string. We can whitelist all the custom post types we want to access. The private custom post types will only be available if the user is authenticated. This way we can get access to sensitive information without it being exposed to the general public while the public custom post types are exposed out without authentication. Here is a snippet that gets the allowed post types for import.

But there was another challenge. The API does not allow us to extend its functionality. That is all it could provide us for the moment and I was blocked when I learned that custom taxonomies, sidebar widgets, and admin settings had no endpoints through this API.

We needed a way to get these things to complete the theme setup. Amit Sannad suggested we use the WordPress.com’s XMl-RPC API to get those values imported. WordPress.com’s XML-RPC API provides default methods to fetch taxonomies and terms. But for sidebar settings and widgets I had to write a custom XML-RPC method by extending the wp_xmlrpc_server class to get the required dataset based on authentication.

One more feature of this plugin is the support for WP-CLI so that we can run the imports from command line and get the data imported on any start up scripts to setup or update the dev environment. This allows us to automate the import process. We can have credentials saved to a json file ( format of the json file is comitted to the github repository ) and pass the path to the file as an argument for authentication only when we run the CLI for first time on a site and the access token will be saved to database for any subsequent use. To import using WP-CLI we can use :

/*Import all data from production server for a given URL*/
wp --url=dev.local pmc-import-live import-all --file=/path/to/auth.json
/*Import only specific required data from production server for given URL*/
wp --url=dev.local pmc-import-live import-routes --routes=users,menus
--post-type=post,gallery,page --xmlrpc=taxonomies,options

This process completed the basic entities that were required for importing content in order to have a fully functional replica of a production site with minimum amount of data. And we could create as many setups and pull just the required data on a click and also refresh existing test site with new data as and when required without having to actually touch the production site. The PMC Theme Unit Test plugin would perform all that for us.

PMC Theme Unit Test has been a very challenging and satisfying plugin to develop. Thank you Gabriel Koenand Amit Sannad for conceptualising this wonderful plugin and letting me work on this. Thanks Corey Gilmore, Amit Gupta and Hau Vong for your valuable inputs throughout the development.