Review Incentive Trial Program

With the recent changes to the Theme Review infrastructure, we are now much more easily able to track Theme review and approval of new Themes, previously reviewed Themes, and updates to currently approved Themes. While we have a much higher volume of tickets being closed than we have had in the past, we end up mired in a cycle of Themes being not-approved, re-submitted, and then not-approved again. If we are ever to gain long-term control of the review queue, we need to break that cycle – and the only real way to do that is to ensure that, as often as possible, all Themes submitted eventually get through the review process, and approved.

In an effort not only to bring the Theme Review queue under control but also to further our ultimate objective: to ensure Themes are approved and made available in the Theme directory, we are going to try an incentive program of sorts. As a side benefit, this incentive program will partially address another area of frequent discussion: the Featured Themes list.

This Trac Report tracks new and previously reviewed Themes that have been approved and made live in the past 30 days. It will be the basis for our experimental incentive program.

Every month, we will determine the three reviewers with the most tickets in this report. Each of those reviewers will get to select a Theme to be included in the Featured Themes list for the next month.

Details:

“Winners” will be selected on the first of the month

To count, tickets must be for never-before-approved Themes (i.e. new and previously-reviewed, but not theme-update)

To count, tickets must not only be “approved” by the reviewer, but closed and made “live” by an admin

Selected Featured Themes must be currently approved in the Theme directory, and must be current

(Yes, you can choose one of your own Themes)

Final decisions are at the sole discretion of the TRT admins

The incentive program is intended to be positive; if it becomes contentious or causes problems, it will be discontinued

Additional details/clarification will be added, as needed

Again, the goals of this program are to help bring the review queue under long-term control, to facilitate more Themes completing the review process through approval, and to provide some community input into the Featured Themes list, while providing a bit of incentive to Theme reviewers.

Please discuss in the comments. Let us know what you think of the idea, or if you have any questions.

“only approved”? It take same effort to “approve” or “disapprove” theme. What I disapprove a theme and author never submit a new version? I am worried about we are encourage wrong behavior. At the end, we add more responsibility for admins.

There’s simply no fair way to include not-approved Themes. How would we reasonably be able to differentiate between a sincere, full review that results in “not-approved”, and a “find-a-single-problem-and-close-as-not-approved” review? It’s just not practical, unfortunately.

Also, this particular objective isn’t just about reducing the queue, it is about getting Themes approved – and that objective is intentional.

I would suggest not closing tickets right away. Leave comments – including comments asking the developer to respond, leave the ticket open, and move on. Follow up on it a week later. Take advantage of the recent infrastructure changes, since by leaving the ticket open, any subsequent submissions will be appended to the open ticket. Let developers know that, as long as they are working with you and the ticket stays open, submitted updates will be appended to the open ticket, and the Theme will retain its place in the queue.

I suspect that you’ll quickly get a feel for differentiating between the tickets submitted by developers who will follow up and work with you, and tickets submitted by developers who will abandon the effort.

Yes, I meant in case they simply don’t respond with updates after doing all the above you suggested i.e. putting a good faith effort to get the theme approved, but if there are too many of those, i guess you will do right by us anyway….

There are only 1,828 approved themes in the repo. There must be countless others abandoned or who decided just to host themselves. I’d love to see the ratio of submitted : approved themes.

A few years back I had a pretty tough time getting themes approved. Was pretty intimidated by the whole process and frankly, only made it through with reviewers who really did a lot of hand-holding to get me there. If I didn’t have such a desire to get in the repo, I would have been gone.

If I would have known about the Theme-Check plugin at the time, would have saved me a lot of grief. A great way to get themes approved would probably be if you can tell they’re new to the review process because of the amount of commonly overlooked issues their theme presents, referring them to https://wordpress.org/plugins/theme-check/ will go along way.

First theme is always tough. On other hand, this process do help reviewer differentiate “required” and “recommended”. Personally, I never ask author to do things that are “recommended” only. Most authors do not even know the difference.

It takes time to get there, I hadnt reviewed anything for a year when I started again and its only by reading alot of other reviews and also reviewing themes made by reviewers that I hopefully will find a middle way. Its not as easy as required or recommended.

“approved” by the reviewer, but closed and made “live” – Approved and Live are the keywords! Makes the whole idea and system that much fair as a “Live” theme is a true measure that it was a good (Meet requirements) Approval.

Having had 2 of my reviews and “Approved” themes re-opened goes to show that just the “Approval” alone as the measuring factor is not enough.

Big 5 thumbs up for this idea. I hope this will cut down on the number of quick partial reviews, with no offer to re-review. I also hope it will motivate me to pick up my pace with reviews a bit. I’ve been slacking since non-admins stopped reviewing Priority #1 queue, as the ration of “interesting interactions with talented developers” to “irritating interactions with people that have no understanding of the GPL and are just here to game the system with their totally unoriginal theme” has taken a fairly unhealthy swing towards the latter.

By the way, the Priority #1 queue thing was never actually made policy. We discussed it, but the feedback was mixed, so we never made a decision. Feel free to continue to review Priority #1 tickets if you want.

I wish you knew just how much I approve of this incentive. I’ve submitted a theme since the beginning of this year and each time it gets rejected for (and this is very subjective) very small reasons that I actually fix in minutes….but I have to wait 5 more weeks only to be rejected for something even smaller. 7 months and still counting.
I’ll keep reviewing other submitted themes; thing is you grow with each review. The sad thing is sometimes your eye for detail is blurred when reviewing your own theme prior to submission