Blog

WordPress core added the block editor in version 5.0.0, changing fundamentally how WordPress stores and displays elements in post (page) content. The block editor, named gutenberg, is written using react and uses the WordPress REST API to interact with data in WordPress. So you maybe forgiven to believe that it would then be easy to interact with blocks and block data (attributes) via the REST API. However it is not, but to understand why, first you must understand how block data is stored.

This example shows how a the core/button block is stored in post content in the database. Blocks are represented as mixture of html comments (with a json object) and html tags. A block is defined with the starting comment

<!– wp: . Blocks can either have an opening and closing tag (like the button block above) or a single line like the following search block example.

<!-- wp:search /-->

Block data is used to define how the block behaves and renders. Block data can be stored in one of two ways. First, as a json blob in the html comment, see the className in the core button in the above example. Handling this data, once parsed, is pretty simple in javascript or in other systems that can modify json. The second way of storing data is as part of the html markup inside the block. Take for example how the button block is defined.

As you can see from this block definition, attributes like url, title and text are all stored in the html of the block. Data can be stored in the contains of the html tags, meaning everything that appears after the close of the opening tag and before the end of the tag. Or data can also be stored in attributes of the tag, such as the href or title. This can make working with block data extremely difficult.

Processing block data

When rendering blocks in WordPress in the front end (theme) on for example a post page, the PHP will parse the html that contains blocks (post content). Core does this, as blocks can have registered script associated with them, which are defined using register_block_type. The function parse_blocks uses regex, to find the block definition and return an array of blocks. From there, it loops around each block, and enqueues the registered javascript libraries. Each element in the array, as a field called attributes, this contains all the fields and data defined in the html comment (json blob). However, what is missing from these attributes is the data found in the html attribute / tags. At the time of writing (in WP 5.2), core doesn’t parse these fields and return them as usable data. To be able to extract data from html, PHP would have to parse the html of the block and navigate the doc tree generated using css selectors, to get the tag and field where the data is stored. This would be a resource intensive process and result in false positives. Even if this process wasn’t resource intensive, at time of writing it is not possible to process all blocks. Many blocks both core and ones defined in plugins are only defined in the javascript, meaning the PHP is completely unaware of the block and associated block data.

Defining block data

As part of an going project to be able to use gutenberg in the WordPress mobile app, the gutenberg team have been working on a RFC to define block structure. This project hopes to define the structure of a block as a platform independent json file. This file can then be used by both the javascript and PHP, to define each block and make so the both the server and front end are aware of all blocks. Once the blocks are defined, the core team plan to make two new REST APIs. These new apis are documented in this post.

Fetching the available block types through REST APIs.Fetching block objects from posts through REST APIs.

Once these apis exists, headerless applications like the WordPress mobile app or a headerless frontend written in react or a similiar framework, could much easier use block data to render.

It is likely that it take some time to get all existing blocks converted to this new style and will require efforts from developers to do so. It is likely that functions required to register the block in PHP, will not be merged until 5.3 at the earliest.

Accessing block data now

What if you need access to this block data now and you can’t wait? Well, you are in luck, there is a proof of concept plugin, called wp-rest-blocks written to expose this data. Once installed and activated, this plugin adds two field to the post / page REST API endpoints. These fields are ‘has_blocks’ and ‘blocks’. Has block is a boolean, the denote, if the post content contains blocks and the blocks field, is an array of blocks. An example of the output is displayed below.

As detailed above, it is not possible to get all block data currently. But this plugin does get existing block data found in the html comment which in many cases would be enough. But there is also a “rendered” field, that has the fully rendered block. This is specially useful for dynamic blocks. If the block uses lots of html attibributes to store block data, the innerContent or rendered fields, can be used to extract this data. The api, even supports nested inner blocks, returning an array of the inner blocks, in each block recursively.

I have also made an effort to try and extract block data of core blocks. Current I have mapped data of the following core blocks.

Image

Gallery

Heading

Paragraph

Button

Video

Audio

File

There is even some unit test coverage, in case there are some breaking changes in core in the future. This plugin is in the very early stages, but maybe a good starting point for anyone wishing to build a front end using headerless WordPress.

I had the honour of being invited to talk on the hosts of the WordPress weekly podcast this week (episode 356). I have been listening to the podcast for years and been friends with JJJ (the co host) for a while.

I talked about on many topics and spoke a little bit about how I got into the industry. In the podcast I mention the RFCs the gutenberg team have released. There is an “Add the block registration RFC” and “Add Blocks in Widget Areas RFC“. The widget area RFC, is one where I wished more developers had commented and fedback.

I was lucky enough to attend WordCamp Europe 2018 and it was by far the biggest WordCamp that I have ever been too. Lots of the things I learn at WordCamps are not from the sessions or talks, but from the hallaway track and chat with people in the WordPress community. So here are some of the interesting fidbits that I learn while talking to people on the ground.

Automattic are enabling developers on there VIP GO platform, to empower themselves with a growing toolkit. The VIP GO team have adding some new tools. First of which is the ability to allow you to use tools like Circle CI and travis, to build / compile your code. Read more at there official blog post here. This will enable more javascript / scss heavy projects, to compile and minify there code. This will enable more faster workflows using Frameworks like React or Vue.js. Automattic announced CLI tools, written in node. This tool uses github to authenticate and enables developers to sync data from different environments and I am sure functionality will only ever grow.

I saw an interesting talk from the Human made team, discussing there use of react and the rest api, with the redesign of Tech crunch. Hosted on VIP GO, the site has a react based theme. With the site being a WordPress theme, they also adding none javascript fallback to pages and allowed react to replace content that appears on the site. This means that the site fade in gracefully and there isn’t such a jump in. They said, that they noticed no lose in SEO, from moving to javascript. There also mention that is was very hard to integrate tracking libraries, like google analytics and nicely load ads from there custom platform.

There are some interesting talk about javascript apis, documented in the slides. All new functionality for javascript is being put on NPM and made more generic. This means, that other members of the javascript community can jump and use them libraries and start contributing back.

WordPress in 2019, was an interesting talk about, the current state of WordPress. It was a little bleak and mentions that growth will not continue if the experience in the admin terminal and on site doesn’t improve. Noel mentions many AI and machine learning to improve user experience.

Matt Mullenweg keynote was brief. He talked about the state of the gutenberg project and the plans for the rest of the year. The plan is to get everything merged by September. But that automattic has provided a plugin called gutenberg ramp. This plugin, disables gutenberg, and will allow you to selectively turn this functionality on or off. I am sure this will be welcome to many smaller hosts and agencies that are dreading when the new editor drops into core. There are plans to put blocks in every part of the site. The plan is to make everything blocks on WordPress. Get rid of menus, widgets and headers. Also removing custom meta boxes by using custom post types and fixed templates. These fixed templates looks pretty clean and I become a believer as seeing it in action. The Q&A was a bit of a mixed bag for me. Many of the questions were weird or pointless. The only useful question to me was the move away from SVN / Trac for issue management and version control. A question I was going to ask myself until someone beat me to it. He said, that he believed they were better tools out there and he wanted to move to them someday, but it was low down the list of stuff they want to get done.

On the contributor day, I sat with the REST api team for most of the day. This team, are extremely smart and hard working and they don’t get the credit they deserve. The rest api team having being doing a lot of work on the register meta function. This means there is lots more work being done on make the data stored in WordPress much more structured. See this ticket for more information, this has since been merged id which amazing, as it has been in the works for over 2 years. 😀

One of the biggest missing pieces of the rest api, is a way in core to authenticate requests with the REST API. I pressed the team of this and got some interesting information. At the moment, the most popular methods are Oauth1, Oauth2 and JWT tokens. Regarding these solutions, they said.

Regarding JWT tokens, they are novel solution to the problem, but not workable for core.

Oauth1 plugin is just a proof of concept and is not be actively worked on, as the workflow is too hard for many applications.

Oauth2 plugin is currently the recommended plugin to auth the rest api.

The Rest team are also looking into X-Auth, an auth method that Twitter first implemented and have since removed support for.

I know there is more that I am missing, so let me know what you learnt and remember to follow me on twitter for more twitter news.

So this is a post that I have been meaning to write for a long time. I think that the fields API and how core handles meta is without doubt the most important problem that is facing the WordPress project. But looking the work done to date on the fields api, it can get a little confusing, the project has changed owners and scope a number of times. In this post, I hope to clear up a little of the history and explain my ideas for the fields API going forward.

So what is the fields api anyway and not having it is such a problem?

The fields API is a bit of a different name for what it really is. It is in short, a library to build custom meta boxes and UI to interact with meta data. The first reference to field api goes all the way back to 2011, in ticket 18179 – WP_Meta_Box. Basically the original idea what to make the need for plugins like advanced custom fields, fields manager and CMB 2 mute by making it easy for developers to register meta boxes and UI for post meta data in core. This would make the job of plugin / theme developers much easier, as it would make adding new fields the post screen extremely easy and would mean that testing on WordPress core updates, much simpler. Plugin / theme developers have long wanted this, as building out custom meta, is a key part of the developing anything custom for WordPress. I believe it true to say, that it is the first thing that most WordPress developers do when building a project, is to set up either ACF or CMB 2. This setup is normally a manual process, as currently there is no way to require interdependencies with plugins (if this plugin /theme is installed, then install this library / plugin). This gives developers two choices. Either hard code the meta boxes / fields that you want and build out their own libraries or bundle a meta box library in with their plugin. Either solution is not great. First, building your own meta boxes, takes time and testing. It means that whenever WordPress core updates itself, that plugins require retesting. Option 2, means having a whole library in the plugin, that adds overhead. If any other plugin is including the same library, it may mean that having both plugins activate at once will break your site.

So having the a fields api in core seems to make sense, from both a user and developer level. It makes developers lives easier and makes WordPress a more usable piece of software for users. But the fields api should not just be limited to post meta, users, terms and comments all to have meta in core. Also multisite too, will have some meta tables, with some work that I am currently working on, by adding site (blog) and network ( site ) meta. So my previous blog post on the subject on site (bl0g) meta and check on the tickets for site and network meta. It is possible to make the fields api the interface for all meta data in core. However, I have also seen people discussing the idea of using the fields api, as a general api for drawing UI for any data stored in the database. That would mean it could also be used in on the core objects like Posts, Users and Options. Where on the face of it, this may seem like a good idea, I am a strongly against this. For me, this muddies the water of the fields api and makes the scope of the project limitless. It makes the project a silver bullet for the project and nearly impossible to ship. The fields api project has been going since 2013 and has had a numberdifferent versions and teams working on it. For some reason, this project hasn’t taken hold or got the support it should have. What is worse, is with Matt’s three focuses of customiser, editor and API that this and many other projects are being sidelined.

An API driven future

The REST api was merged in WordPress 4.7 and is started to be used widely by the WordPress community, even by the new editor Gutenberg. It is fair to say with the raise of frameworks like Vue and react, that the world is going towards a more javascript central front end. Matt himself said, “learn javascript deeply” in his state of the Word. For the core data types like Posts, this is fine, as these endpoints are already in core. However, you may we shocked to find that there is little support for meta data in the core API. This means react based Calypso can only handle editing core data types and limited meta keys. It means that if you have a plugin / theme registers a meta boxes, this simply doesn’t appear in the Calyso interface. This is of course a horrible user interface, as settings once important part of workflows on desktop (wp admin interface), just disappear in Calyso or on the iOS or Andriod apps.

Protected data
Core developers have no idea of knowing what data is private / sensitive and should not be displayed publicly.

Serialised data
WordPress can store serialised objects in meta data and these can be a security problem. There is no way of knowing data type

There is much more to Ryan’s post, do check it out. But it comes down to one problem, meta is not registered anywhere, so there is no way to know, if it should be in the api or not, it’s data type, what are valid values and permissions to read / edit / delete are not defined.

A new hope

In WordPress 4.6 Jeremy Felt made some changes to a little known function called register meta. These changes were designed to allow developers to register some details about the meta data. It allows for some important information to be registered, such as.

Sanitize callback
Allows for meta data to formatted on save, meaning data meta can be trusted to be the correct format / data type.

Type
Does type hinting, so gives core an idea of what data type of data stored.

show_in_rest
Allows developers to flag weather or not this meta data in the api at all.

As you might understand, this is a big step forward for meta data. For the first time, meta data is much more control on it’s processing, handling and saving.

This register meta function, is basically the fields api. It would require little work to core, we could add some new params to this function to define the kind of input that is displayed to user (select, textarea, date, text etc), a save callback, location of input and default value(s). Once these params are in, this could be returned API. Once in the API, these fields could be read by something like Calypso and metabox / meta data could automatically register this interface. This would make plugins automatically supported by javascript interfaces. It would just require the javascript interface to implement the fields.

It also makes improvement to multisite a lot easier. Multisite has been missing a settings api for a long time. As under the hood, network settings are just (site) meta data, the fields api could replace the need for a network settings api.

This doesn’t even mean that plugins like ACF would die as a plugin. I would still have a place, support fields that core didn’t like repeaters and google maps. But under the hood ACF could be converted to use the fields api, converting many sites already using this plugin, a more structured data set in the database.

Conclusion

With the fields api in core, we can work on having the user experience an enjoyable and consistent user interface across all platforms. With form fields being generated from one place, we can make the process of translation and accessibility much easier, making aWordPress and much more user friendly piece of software. It would allow those using the WordPress via apps or third party connected software, have a much consent experience and hopefully mean less breakable in the future.

But this gets down to the core issue, I have with the WordPress project. The user comes first. All the work that is currently being done on core, is about making end user experience better. I understand this desire, but it forgets one simple thing, that developers are users too. That a part like having a fields api, is a problem for both developers and users, and WordPress would be better if we solved this problem ahead of other things. The fields api, makes things cleaner, faster and more reliable for custom development. It means that developers, can save time and add value to the projects they are working on in other ways.

Greeting all and welcome to the Spacedmonkey blog. This blog will discuss all things WordPress and share tips, tricks and code snippets of some of the more advanced things we are working on here in Spacedmonkey.

We are big fans of open source here and all the code we write is open sourced when possible.
We also take requests, so if there is anything you want us to talk about, just in touch in the coments below.