Frustrated by Magento? Then you’ll love Commerce Bug, the must have debugging extension for anyone using Magento. Whether you’re just starting out or you’re a seasoned pro, Commerce Bug will save you and your team hours everyday. Grab a copy and start working with Magento instead of against it.

As Magento 2 approaches its first half-birthday, one thing is clear: Magento 2 is leaning heavily on PHP Composer for its developer workflow, and for the merchant facing Marketplace.

If you’re a developer working with Magento, you may be familiar with repo.magento.com. This is Magento 2’s composer repository, and up until Magento Imagine 2016, its main purpose was to provide modern PHP developers with a way to install Magento 2 using PHP composer.

At Imagine 2016 Magento unveiled their Magento Connect replacement, Magento Marketplace. Behind the scenes, Marketplace is running on PHP Composer, and repo.magento.com is the source repository for purchased extensions.

In this article, we’re going to show you how you can mirror your Magento specific composer packages on a local server. While not necessary, this is a useful precaution to take if you want to avoid any unscheduled maintenance bringing down your deployment and development pipelines. Along the way, we’ll also discuss Marketplace’s new composer based architecture, and end up touching on many lesser known composer features.

The package you want is at [THIS Git/SVN/Mercurial URL] and I found an archive at [THIS URL]

Packagist is a composer repository. One thing that catches a few composer newcomers by surprise is packagist, and other composer repositories, don’t actually host any packages. They just point to the location of a package on another server.

Other Repositories

Packagist support is baked into composer. However, it’s possible to point composer at other Composer repositories with a composer.json configuration that looks something like this

This flag ensures the magento/project-community-edition package is downloaded from repo.magento.com and not packagist.

The meta-package method downloads and installs Magento components (modules, themes, language packs, and libraries) into the vendor/magento/ folder. Thanks to Magento’s registration.php file and PSR-4 support, there’s no longer a strictly defined place where Magento modules, themes, language packs and libraries need to be located. These components can now be separated out into individual composer packages.

The key to understanding Magento’s need for credentials is Magento Marketplace.

Magento Marketplace

Magento Marketplace is Magento 2’s replacement for the old Magento Connect. Magento Connect only hosted free Magento modules — commercial module listings pointed off towards independent software vendor’s websites where you could purchase the extension or service directly from the independent vendor.

Magento Marketplace changes that. Marketplace has free extensions available, but it also provides a one-stop shop for purchasing your Magento extensions. Once purchased, you can download the extension package from the My Account section of Magento’s website.

In addition to this download, the extension will also be available via the Magento 2 Component Manager in Magento’s backend, at System -> Web Setup Wizard. The Component Manager is a GUI for installing composer packages from repo.magento.com. It turns out that repo.magento.com is a session-ed packagist repository.

When you log in to repo.magento.com using the aforementioned HTTP Auth credentials, Magento’s composer repository returns a custom list of packages for you to install. This will include

The base Magento 2 CE packages

Any Magento 2 EE versions your account has access to

Any package you’ve purchased

This is what enables you to fetch purchased Magento Marketplace packages via Component Manager. You can also simply add new packages to your composer.json file and then update your system via the command line (i.e. composer update)

Mirroring repo.magento.com

While Magento’s adoption of composer is welcomed, it does add an additional wrinkle to deploying Magento 2 projects. The repo.magento.com repository is a new, single point of failure that, when down, could block a composer based deployment from being updated, or prevent your development team from getting started on a new project. Additionally, unlike the other (still not great) single points of failure in a composer project (packagist.org, github.com, etc.), repo.magento.com isn’t yet a battle tested system.

Because of this, it makes sense to create a local mirror of repo.magento.com. In addition to protecting you from unplanned maintenance/downtime at repo.magento.com, hosting your Magento packages on your local network (or even local development machine) should speed up Magento deployment tremendously.

Mirroring with Satis

The composer project has a second, sibling project called satis. Satis was created to allow developers to create their own local mirrors of packagist.org content. It turns out a stock composer repository requires zero dynamic processing — a repository is just a collection of static json files, and (optionally) mirrored archive packages.

To use satis, you’ll need to clone the GitHub repository to your local development machine with one of the following commands

Satis will output packages.json files and (optionally) package archives to the output folder you specify. You’ll be able to upload these files to any simple web server, and have a working packagist repository.

The JSON configuration file is where you tell satis which repositories you’d like to mirror, which packages in the repositories, as well as any other satis configuration needed.

The syntax for building a satis mirror is

$ php bin/satis build config.json build-folder

The file we’ve named config.json above is often named satis.json by convention.

Satis Configuration

Next up, we’re going to create a satis configuration file and review its options. While the composer manual has a section on satis, the file format isn’t as well documented as it could be. The following is one possible configuration, please get in touch or comment below if you see something egregiously wrong here — I’m learning with the rest of you here!

The name property should be a simple description of your repository. It’s used in a static HTML file generated for the repository, so don’t use a name that would embarrass your mother or your supervisor.

The homepage property is the URL you’re planning on hosting your satis repository at. This can be a URL on the Internet, or a URL local to your network/dev machine. This URL will be used in a static HTML file generated for the repository, so make sure its accurate.

The repositories property is the first vital configuration field. This should be an array of the composer repositories you’d like to mirror with satis. In our case, this is the composer repository at repo.magento.com. Other types of repositories you might see are VCS or pear.

The require-dependencies and require-dev-dependencies flags make sure composer will require any dependencies a specific package may have. Not strictly necessary in our case (see require-all below) but it never hurts to be explicit.

The require-all property tells satis we want to grab every package on the repository. Since our goal is to create a complete local mirror, we want this here. If we were interested in creating a mirror of select packages, we’d use a require property and list out the packages.

The final top level archive property tells satis that, in addition to creating a packages.json file for us that points to packages, we’d like satis to grab and/or build a project archive for us as well.

In the archive object configuration, the directory property tells satis where the archived files should be copied to (i.e. build-folder/dist), the format property tells satis which archive format we should use, the prefix-url should be, again, the URL for your repository, and setting the skip-dev option to false ensures we get every file in a package.

The prefix-url key is important here, as it’s the URL satis will use in the generated packages.json file to point to our local mirrored archive.

Building the Mirror

Once you’ve created your satis.json, you can create the mirror with the following

$ php bin/satis build satis.json public -vvv

If your repo.magento.com HTTP credentials aren’t stored in ~/.composer/auth.json, satis will prompt you for them. The -vvv flag is optional, but will ensure satis is verbose in its output. This can help point to problems in your configuration, or with your network connection.

When the command finishes running, you’ll want to upload the files in public to the web root of whatever URL you configured in satis.json (composer.pulsestorm.dev in our case).

When satis is done running, you can take a look at the repository files using the unix find command

You should see archives of all the community edition files, any EE files your account license grants you access to, and any extension files you’ve purchased. Upload the entire contents of the public folder to your web server, and you’ll have a local composer repository up and running.

With our mirror created, we’re ready to install Magento 2.

Installing Magento 2 with our Mirror

Normally, when installing Magento via composer, you use the following command

You’ll see that its composer.json file points to repo.magento.com. The --repository-url option only applies to the package that create-project grabs. Otherwise, composer will use whatever it finds in the project’s base composer.json file.

This is easy enough to work around — all we need to do is use the no-install flag — this way create-project will only download and extract the project, it won’t run composer install. This will give us a chance to edit the composer.json file before running install.

Installing Magento 2

OK! We’re ready. Step 1, lets clear our composer cache

$ composer clear-cache

This is not strictly necessary, but useful when you’re first setting up a mirror, and may have an invalid package reference stashed in cache somewhere.

Also, if your mirror’s not located on an https server, you may need to set composer’s global secure-http flag to false.

$ composer global config secure-http false

Recent versions of composer will refuse to run over non-encrypted HTTP.

Again, the -vvvs are optional, but viewing composer’s verbose output can help us ensure that no package was/is downloaded from repo.magento.com. When composer’s done running, change into the project-community-edition folder, and take a look at composer.json

and composer will grab all Magento’s packages from your local mirror — no access to repo.magento.com required.

Wrap Up

All of this, of course, is only a start to keeping and maintaining a local composer mirror. EE users will want to check their enterprise license agreement to make sure doing this falls within acceptable use of the EE source code, and regardless of which version of Magento you’re running you’ll want to make sure your mirror isn’t located anywhere online, as you may become an inadvertent distribution point for commercial extensions you don’t have the right to distribute.

You’ll also want to figure out a way to get satis running on a regular basis — otherwise you may miss important updates to the Magento repository. The very bold may want to expand their mirroring to packagist itself, but that’s a larger problem filled with all sorts of blind alleys and large hard drives.