Roadmap discussion for Vanilla 2017

It's been a year since I wrote a major update about our roadmap, and most of that discussion is outdated at this point.

There are already a few really big things in Vanilla's master branch that didn't make the 2.3 release fork last April. These include:

Categories were rebuilt with clearer "display as" functionality and now scale really well. We added a paginated "Flat" category structure for when you need hundreds of parallel categories.

Dashboard v3 is in, with a complete refresh of our design and functionality. It has a complete style guide and a scaffold of tools and modules for developers to more easily build settings pages for their addons.

Under the hood, we rebuilt our plugin scaffold into a generic "Addon Manager" and have refactored the dispatcher to work more clearly and consistently. This is groundwork for a new API and self-generating API docs.

Looking ahead, we're gearing up to implement API v2, a native, internal API that will streamline addon development and enable us to run more automated testing against core. It will also allow more granular permissions, and be a more complete & pluggable external endpoint solution than anything currently available. I expect this to be complete (for some definitions of "complete") sometime during summer 2017.

In the mean time, we're also hoping to add "profile cards" support (a popup mini-profile when clicking a username), redesign our base theme (maintaining support for legacy themes), and refactor how edit history is stored (to separate it from logging).

The options I like best are to either start a 2.4 release cycle immediately (without the native API) and continue on the 2.x "point" releases, or wait for the API v2 feature and increment directly to a 3.0 release. Why 3.0? I think rebuilding all our innards & our Dashboard from scratch with a full native API counts as a product refresh. It's not a huge backwards-compatibility break nor a restart like 2.0, but I feel like it would give folks a reason to have a second look at Vanilla if it's been a while.

A year after my last update, I also worry about increasing the pace of our releases. We still have some major addons not compatible with 2.3 a month after its release and after a 6-month testing period. That leads me to believe we don't have critical mass for a more aggressive release schedule. Also, it's very stressful and time-consuming to do a major release for our small team. Hitting annually or slightly better seems like a nice compromise. The 3.0 plan gets us closer to that pace.

I'm currently leaning strongly in the direction of a 3.0 release in 9 months or so, but nothing is set in stone. Constructive feedback welcome.

Comments

@Linc said:
It's been a year since I wrote a major update about our roadmap, and most of that discussion is outdated at this point.

>

The options I like best are to either start a 2.4 release cycle immediately (without the native API) and continue on the 2.x "point" releases, or wait for the API v2 feature and increment directly to a 3.0 release. Why 3.0? I think rebuilding all our innards & our Dashboard from scratch with a full native API counts as a product refresh. It's not a huge backwards-compatibility break nor a restart like 2.0, but I feel like it would give folks a reason to have a second look at Vanilla if it's been a while.

A year after my last update, I also worry about increasing the pace of our releases. We still have some major addons not compatible with 2.3 a month after its release and after a 6-month testing period. That leads me to believe we don't have critical mass for a more aggressive release schedule. Also, it's very stressful and time-consuming to do a major release for our small team. Hitting annually or slightly better seems like a nice compromise. The 3.0 plan gets us closer to that pace.

I'm currently leaning strongly in the direction of a 3.0 release in 9 months or so, but nothing is set in stone. Constructive feedback welcome.

I'll do my best to provide constructive ffedback.

I like best this option: a 2.4 release cycle immediately (without the native API)

while api could be great, the other functionality in 2.4 and bug fixes, core plugins would be nice to get out in a timely manner, since as you say now is the optimal time to start a release cycle if you are going to do a 2.4 release.

A year after my last update, I also worry about increasing the pace of our releases. We still have some major addons not compatible with 2.3 a month after its release and after a 6-month testing period. That leads me to believe we don't have critical mass for a more aggressive release schedule.

From user testing standpoint, the testing period at least by a large group of users was probably small, because incremental changes are not downloadable by zip. The burden of a build and github, may be a deterrent to a larger number of users than you think. Whereas, a downloadable (workable w/o build) zip from github at incremental points would give your test user base several orders of magnitude larger.

We still have some major addons not compatible with 2.3 a month after its release

I don't think basing vanilla releases on third party add-ons is a benefit. Plugin authors will always abandon add-ons, this is a fact of life. If the add-ons are significant and help market vanilla why not co-opt the add-ons and fix them. if a third party add-on causes you to think about altering a release don't worry about the add-ons, or produce your own add-ons that mimic the third party add-ons that you feel are important enough to delay a release. Waiting 3 more months in a release cycle probably won't fix a broken third party add-on.

Downloadable working vanilla zip of alpha, beta or rc without a build would go a long way in average user testing and give you the largest test base.

Also, it's very stressful and time-consuming to do a major release for our small team.

understood, you need to do what works best for you.

since you state:

I'm currently leaning strongly in the direction of a 3.0 release in 9 months or so, but nothing is set in stone.

The reason I suggest against this is because guestimations of release dates are usually more optimistic leaning than realistic leaning for all types of software as well as vanilla software. There are always unknowns. and this could stretch out well beyond 9 months. Look back at your own time predictions and see if I might be correct.

Pragmatism is all I have to offer. Avoiding the sidelines and providing centerline pro-tips.

Thanks for the insights! I really appreciate that you keep us informed of what is going on at Vanilla Inc.
By the way: I find the "our reading list" discussion very cool

Not much of a constructive feedback from my side, but more of an opinion concerning release cycles: in my opinion there should be a lot of smaller releases instead of one major release every X month where "X months" feels to me like once a year.

Sometimes bugs and glitches are noticed by several new users and having to answer "this is a known bug" more than one time makes me feel uneasy. So if the community would do pull requests against the current stable OS release, would you consider releasing patched minor versions?

Not much of a constructive feedback from my side, but more of an opinion concerning release cycles: in my opinion there should be a lot of smaller releases instead of one major release every X month where "X months" feels to me like once a year.

Sometimes bugs and glitches are noticed by several new users and having to answer "this is a known bug" more than one time makes me feel uneasy. So if the community would do pull requests against the current stable OS release, would you consider releasing patched minor versions?

concur with opinion.

ABSOLUTELY. The slower response (reality) to the issues for whatever reason seems to make numerous users of vanilla give up on trying to help with pull requests let alone issues.

While the software may be great for vanilla cloud, if you defer to vanilla 3.0 with APIs - this will be a very very very long lag time with vanilla open source. Essentially you leave behind lots of user input by having such a large gap in terms of commits as well as time. Numerous issues in core plugins have also been fixed and these too will lag behind.

Regarding looking forward to API 2.0. I would be keen to see it, But not at the expense of timing of a release Vanilla which could be gotten out the door sooner, it is greatly behind cloud.

Now if the reason to keep open source behind the Cloud by a year or two or three, might be advantageous in the short run, it might prove to disadvantageous in the long run. Same as combining org and com. might provide a short term boost. alienation of community perhaps due to overmarketing hype. But you got to do what you got to do. just don't shoot yourself in the foot by making wrong marketing decisions.

Pragmatism is all I have to offer. Avoiding the sidelines and providing centerline pro-tips.

@R_J said:
So if the community would do pull requests against the current stable OS release, would you consider releasing patched minor versions?

I'm good with doing patch releases. We don't really have strong criteria for when to do a patch release right now, aside from security fixes. As long as the issue also gets fixed on master and the patch is limited in scope, the only real impediment is whether someone takes the time to identify the patches and create the pull requests.

@R_J said:
Sometimes bugs and glitches are noticed by several new users and having to answer "this is a known bug" more than one time makes me feel uneasy.

This is to everyone; I'm just quoting @R_J since he raised the point succinctly.

There's a lot of factors in play when "known bugs" hang around. How severe is the issue? Who is it important to? Has is been communicated well? How complex is the fix? What are the repercussions of the fix? What is the value vs cost of fixing it? And, has it been seen or assessed by someone who can evaluate all those things accurately?

To approach this challenge as "did X, Y, and Z get fixed yes/no?" is to dramatically oversimplify the hundreds of decisions large and small that need to get made every week. There are bugs I've known about for years that I just don't care about because the fix is worse than the problem!

The key is to get the issue raised on GitHub and then having a conversation. If it's important to the open source community, I'd like to see the community trying to take the lead in discussing and implementing a fix. That doesn't mean operating independently — I think the staff were pretty responsive throughout 2016 on GitHub and will continue to be next year.

Ask a knowledgeable developer to comment before you spend a bunch of time! The easiest way to burn yourself out is to work on a thing for hours and then have it rejected because it contradicts other concerns or plans. I know because I've done it more than anyone!

I've got the company doing nearly all of our issue filing and patch discussions in public precisely so open source developers can follow along and participate. Now how many folks not on staff wants to spend the amount of time necessary to have their heads sufficiently in the product to follow along - that's the million dollar question.

When there is critical mass of product knowledge outside the company communicating well with those inside — that is when you're likely to see those pesky issues disappear at a pace more to your liking. And that is a ton of work.

The amazing chose will be to rebuild all with MeterJs, it's an amazing Framework all in Javascript (node JS for the backend). It might be possible to use Cordova for building it native app for ios, Android, etc. I am a little bit crazy but It will be the dream.

Great news React is just a small part of MeteorJs. Meteor is a real complete solution for an app especially for a forum who needs a real time and off line solution. But it's just my opinion and maybe it will be too complicated for a complete migration.

@R_J said:
In fact it would be nice to see garden as a stand alone framework at one point in time.

As we refactor the framework to be testable, it's slowly moving to library/Garden. I don't think we currently have plans to break it off into a separate repo, but it will ultimately be separate for all intents and purposes.

@R_J said:
May I throw in Vue.js? From what is said its easier to learn than React and React requires the usage of JSX while Vue.js could easily be integrated in HTML but also could use JSX if needed.

We can mention it to @Todd but last I heard he had his mind pretty much made up. It's been under consideration for quite some time. We're going to do a trial balloon project with it internally before attempting to use it in core.

@R_J said:
But since we are talking on frameworks: do you plan to use a css framework?

@Clément said:
React is just a small part of MeteorJs. Meteor is a real complete solution for an app especially for a forum who needs a real time and off line solution. But it's just my opinion and maybe it will be too complicated for a complete migration.

We take a very conservative approach to our tooling and inclusion of third-party projects. Part of Vanilla's ethos is to stay lightweight at its core. When you start grabbing third-party frameworks and all-in-one tools, that goes out the window pretty quickly. The risk of being Composer-based is how quickly the dependency rabbit hole can spool out into the darkness. We won't allow anything into core that we feel exposes us to significant bloating.

Three weeks later, it appears the only opinions I've collected on the matter of versioning are in favor of smaller, more frequent releases. Given that, I think it would be appropriate to start looking at forking for 2.4 as soon as it looks reasonable to do so.

We reworked categories dramatically to improve consistency and scaling, and added a new "Flat" type. This will have implications for folks' sites that rely on edge cases in how the old categories "automagically" decided how to display themselves. I imagine folks will need to manually adjust their "Display As" settings in some cases when upgrading to 2.4. Most often, this just means switching the Display As of your root categories to "Heading" or "Categories", which is a 2-minute fix usually. Unfortunately there's no solid criteria I'm aware of for us to detect and automatically set this.

The Smarty 3.0 upgrade can be less fault tolerant. I've seen sites get a fatal error upon upgrade because of poorly constructed Smarty tags that passed unnoticed by 2.x. I don't know a good solution other than warning folks. We're already using Smarty in backwards-compatibility mode.

The new Dashboard may require deleting the /cache/Smarty folder contents before you'll be able to access it.

We did some reworking of the request object & dispatcher to support the coming API v2. It's possible some folks, particularly nginx users, may need to rework their server config slightly depending on how they did it in the first place. My hope is anyone who's writing their own nginx config won't find this challenging in the rare cases there is a problem, but it's something to be careful about. For the record, I didn't need to modify my own nginx config at all.

As ever there are a hundred other things I wish were done, but those are the only serious speed bumps that come to mind.

Happy belated New Year. I've been away on travels and about to embark on another one in a week. Haven't abandoned this forum or my plugins, just a long pause. Happy to see continued progress. With all the disruptive nature of new versions on production forums I have to acknowledge that it is well mitigated by advancements, solutions to problems and proof of activity/viability. I'm therefore happy that the powers in charge decided on multiple cycles. Looking forward to participate again after my return in March.