Magento Installation Process Thoughts

The Magento team has been looking at the best way to build and then update sites for Magento 2. The team has committed to using Composer where each module, theme, and language pack will be a Composer package. There were and are some interesting challenges. I thought others might find these interesting as well.

Now I sort of expect someone to jump up and say “didn’t you look at the installation process for product X?” (WordPress being the most likely.) Yes, the team looked around as well. I thought the following discussion was still interesting enough to be worth a blog post and it’s simpler here to not explain what other projects do here to keep the length down.

Disclaimer: These opinions are my own and not necessarily that of my employer. This blog post is also personal observations, not a commitment of strategy of how the final installation process will be done. However, opinions (with reasoning why) always appreciated!

Approach 1 – Build Site with Composer

You can build a full site using Composer alone today. The Magento team leveraged the work of a previous community hackathon project using Composer to build Magento 1.X sites. (Believe it or not, the core team does try to learn from what the community tells us!)

It is worthwhile remembering however that the hackathon project was to get Composer to work on the assumption they could not change the Magento directory structure. The core Magento 2 team does not have this same restriction (other than the effort required to change the directory structure around). For simplicity in this post, I am ignoring any potential directory structure changes that could simplify the installation problem. The problems I describe here would be reduced, not eliminated with such changes.

Based on the hackathon project using Composer and Magento specific Composer plugins, it is possible to download a series of modules, themes, and language packs and have them installed into the correct directories as required by Magento. Importantly, there is one additional package with Magento 2 which I will refer to as the “skeleton” which creates the overall directory structure. (I am purposely leaving out details here. For example, it might really be a few packages, but that does not matter for the purpose of this blog post.) So Magento is kind of interesting in that it installs a first “skeleton” directory structure package, then adds modules, themes, and language packs under this directory tree. (Composer normally installs different packages alongside each other under the vendor directory.)

Why is this interesting? Well, it creates interesting situations when you need to update the skeleton Composer package. Normally composer when installing a package removes the old directory tree, then extracts the new package. This is robust because you start from a clean slate each time. The skeleton package is more interesting however as it has other packages sitting within its directory tree. So you cannot just delete the whole tree and start again.

But we believe we have this working with Magento 2 and Composer today. It might not support moving the directory where packages are installed (e.g. if we wanted to rename app/code to app/modules in the future), but it basically works.

Challenge – Merging in Local Changes

A design goal for Magento 2 is to allow all customizations by adding additional packages. You should not directly modify core Magento files or those of extension developers. You should layer your customizations on top using new modules or themes you define and add to the project.

However, while things are better in Magento 2, it is not yet clear this can be relied on 100%. There may be changes to .htaccess files required due to the version of Apache being run, or a hot fix you must get going today to get a customer site running. So while absolutely discouraged, it is advisable to have a development practice that allows such “hacks” to be made. (I will use the word “hack” here to distinguish it from an officially supported override using a module or theme, even though the term “hack” is probably a bit too negative. For example, a hot fix “hack” may be important and necessary.)

So what is the problem?

The simplest (and possibly worst) approach is to build your site with Composer and start changing the files directly. The problem with this is the next time you do a Composer update command to get say a patch for a module, it will discard the changes you made and it will be hard for you to notice. Changes to the skeleton files (non-module and theme files) are particularly tricky here.

Approach 2 – Build Vanilla Site with Composer, and then Apply Local Hacks

A common solution is to have a source set of files that are pristine (untouched), and then apply your local hacks to a copy of those files to build the final site. For simple cases there may never be a hack – it may be all customizable by adding new modules, themes, and language packs. (Remember, that is the ultimate intended goal.) But if hacks are needed, then make sure they are applied to the clean set of files to build your final site.

So how to keep the local hacks correct when the Composer update has changed files you have modified? Put simply, there is no way to avoid this being the responsibility of the person making the local hacks. By definition, it is doing things not intended to be supported. All that can be done is to try and make the hacks minimal, and easy to identify and review for correctness the next time you do a Composer update. Developers should also take it as an encouragement to avoid doing hacks and try to make any changes made via a supported mechanism such as defining a module that customizes other modules using supported technologies. It may look a bit more work the first time, but there will be less upgrade work later when you want to fetch that next available patch via Composer.

Approach 2a – Build Site with a Script

One approach to applying local hacks is to keep a directory that is built purely using Composer and downloaded packages. Then a script is used to construct the “real” web server directory from the clean installation, making any desired changes. To be clear, this script might be as simple as copying the original directory tree then copying another set of locally modified files over the top. (Can anyone say “modman” by the way?)

Note there are a few interesting variations possible here, such as whether to copy files into the final directory tree or use symlinks. Should Magento 2 ‘generated’ code be merged into the main PHP directories and possibly speed up class loading. Also the final directory tree can be quite different to the original if desired, based on your preferences. For example, production may have all the test cases removed, or you might include either the Apache configuration files or Nginx configuration files, or you might put all the web served files under a ‘pub’ directory to have a cleaner final install tree.

Note that a potential side benefit of this approach is the need for Composer plugins goes away. Composer is now only needed to download a compatible set of packages. All the packages (including the skeleton package) can just sit under the ‘vendor’ directory. A separate script is responsible for assembling the real web server directory from the locally available packages. (A script would be supplied with Magento to do this for the standard files. Only local hacks would require an additional script.)

Approach 2b – Use Git Branching and Merging

Another approach to avoid writing a shell script is to take advantage of Git branching and merging. Create a first branch in which all you ever do is Composer update commands. No other local changes are made here. Then create a localized branch from this “vanilla” branch and make your hacks there. Whenever the vanilla branch is updated, merge it back into your customized branch taking advantage of Git’s merging capabilities. When you release to site, you can tag your customized branch so you know exactly what you released. If the vanilla and customized branches have only had minor changes, the merges will be easy; if more extensive changes, merging will be harder.

(I am sure the Git experts can tell me 10 better branching strategies to manage the code base – my point really is to say you can use Git merging to apply local changes instead of writing a shell script to build real directories on disk.)

Approach 3 – Make Magento more Flexible

I feel I should mention the ultimate solution to me is to make Magento more flexible so different parts of the system are less dependent on a particular directory structure. That would mean the master GitHub repository and a local site would not have to be the same directory structure. This to me is a longer term goal where incremental progress based on community feedback may be beneficial (possibly up to GA after which we will lock down changes). I can imagine lots of debates about what the ideal directory structure should look like. For example, should there be a ‘pub’ directory, or should index.php be in the root? There are lots of different scenarios with different requirements (e.g. low end hosting may require an index.php in the root directory.)

Changing the directory structure more radically may open up other approaches to be more Composer friendly, reducing the merging problems described above.

Conclusion

So which approach is best, particularly between 2a and 2b? Actually, I wrote this in part to see what other people think!

I can see the benefits of having a script apply local hacks in that there is a lot more flexibility if you have preferences around the final directory structure.

I can see benefits in using Git in that you can use its file merge capabilities when dealing with files that have changed due to composer updates and that were locally hacked.

It is not yet clear if Magento needs to add explicit support for one scheme or the other. To me it is more interesting as approach 2a could mean dropping Composer plugins and might be more robust when updating the skeleton package. But it puts more work on developers – they have to write scripts to apply hacks instead of just using Git.

The ultimate success will be when localized changes are not required. But until then, any thoughts on 2a versus 2b welcome!

Share this:

Like this:

Related

5 comments

Please excuse my ignorance here, I don’t pretend to be a composer expert, but I have used and have a rough idea on how this works.

Currently in Magento 1.x what devs do is put local code changes into app/code/local. Extensions go in app/code/community, and obviously we have the app/code/core. The app/code/local directory is loaded last so it will overwrite any changes in app/code/community or app/code/core.

So in my mind why do we not have similar here? Then its down to the dev to sort as you say, but at least their changes will not be wiped out by doing a composer update. It’s upto them then how they organise this.

With github we find a lot of people don’t use it, and although we can argue everyone should I don’t believe you can limit people. We have tons of clients that have no source control whatsoever in place, so just be careful if you are wiping data or totally changing things around here, as you might end up with a lot of confused devs on the support forums. Unfortunately not everyone is a genius, any architecture/process needs to take account of that.

Whatever you do here we need a UI for managing installs/uninstalls/rollbacks/etc. This is an absolute must to make Magento work better for merchants. I understand this is a low level discussion, but please do not forget this.

Thanks for the feedback. Ignorance excused – at least I got some feedback! 😉

Composer does not have the idea of different directories. You can keep your master source code in different directories and build your packages from that. Composer is based around all packages being installed the same way. The multiple directory overlaying approach (local, community, core etc) will not be the way its done in Magento 2. Instead, the goal is to guide people to do local extensions using their own modules they develop. These modules should extend based on supported extension approaches. That should make upgrades easier.

I *suspect* people will need to do local “hacks” still. Hot fixes sometimes just sounds likely. We want to make it harder to do (discourage the practice over adding new modules), but not stupidly hard. So trying to find the right balance here.

A key point seems to be how much to use Composer plugin points versus keeping Composer more Vanilla and writing our own “site assembly” script. Right now we are proceeding down the Composer route, and it seems to be OK, but it does involve quite a bit of work with plugins etc. That sort of tight coupling to any technology always makes me nervous.

So I believe what is wanted is a slightly painful solution (so people are not encouraged to take short cuts), but have a scheme for when it is still possible to do hacks in emergencies.

Thanks for explanation. I agree with you on trying to make this tight but you will hit issues with this approach as right now probably 80-90% of sites I see have these local ‘hacks’. I think you will see people modifying straight on top of the composer installs if you don’t give them a route, which would be a bit of a disaster IMO. It comes back to the ‘everyone isn’t a genius statement’. You must cater for that as its actually quite a lot (most) of the devs.

I agree 100%. Hence the request for feedback on 2a vs 2b in post (or proposals for 2c, 2d, etc). Given old approach is now gone, how best to do it? But I acknowledge it is hard for people to express solid opinions because people have not developed much with Magento 2 yet.

I really like the idea of the 2A approach as it becomes a nice choke-point for doing some really powerful pre-deploy operations.

Compiling an optimized version of the framework (think composer dump-autoload –optimize) could be very powerful and forcing the final code structure through a code sniffer or test suite might save people from a lot of headaches. It could prove very useful on the asset side of things as well.

If you’re going to go that route though, provide us hooks into that deploy script!