Composer-like installation procedure, composer.json#522

Labels

Milestone

Assignee

26 participants

It would be nice to have an ability to install phpunit the way composer is installed.

So running tests like php phpunit.phar ./ and downloading phpunit like curl -s http://phpunit.de/installation | php or just wget http://phpunit.de/pnpunit.phar is very easy and does not require to have pear installed

I do not use Composer, so I cannot add a composer.json since I would not be able to test it. Why do I not use Composer? Because I fail to see the need for a replacement of the PEAR Installer. Except when you are on non-UNIX platforms such as MacOS X or Windows, the PEAR Installer "just works". Unless I see a compelling argument for using Composer over the PEAR Installer I will continue to only support the PEAR Installer.

Of course, once the remaining issues with the phpunit.phar are sorted, the PHAR will be officially supported as well.

+1 for composer.
The need for a PEAR replacement should become obvious when you read the article @drak mentioned. Moreover your argument about OS'es doesn't stand: Windows is used by many PHP developers (even though is not the ideal development platform for PHP) and OSX is a UNIX based OS.

The most important issue that composer fixes is the system wide installation that PEAR does by default. On a production env this may not be a big issue, but on a development machine where you usually host many projects this is the reason why we stopped using PEAR

Compare PEAR to its equivalent in any other language (python, perl, ruby ...). And you'll see that it's pretty limited despite the bloat that got in there. I think it's always good to learn about another language's package manager and see how package management can be objectively awesome.

The article you and @drak refer to does not convince me that Composer is THE replacement for PEAR. You are mixing the different meanings of PEAR. PEAR, as in pear.php.net, should die, IMHO. There is no need for a central package repository. PEAR, as in the PEAR Installer, works just fine -- at least for me.

After talking to the composer devs we agreed that it would be nice to have phpunit installable via composer even so it doesn't make all that much sense for dev tools.

I definitely see the "ease of use" part but you don't really want phpunit to be part of your projects composer.json anyways (why should people using your lib install phpunit?).

The solution preferred by everyone I talked to seems to be "Just give me a phar" and we are working on that.

In the meantime I think it is worthwhile to check out if #pirum 1.1 with the announced 'composer support by reading the pear.xml feature would solve the issue of having to deal with pear for people that don't want to :)

I didn't try it yet but if it works it would be a nice solution that wouldn't require someone creating composer.json files for all parts of phpunit and making that work. (I assume that is why nobody sent a pull request yet? :) )

To sum up:

Composer support will happen if someone does it

Maybe Pirum 1.1 will 'just make it work', I will look into the details tonight. - As far as i can tell there are no docs for it on the pirum site yet

For dev tools: phar >> package manager stuff

If phpunit wouldn't have the process-isolation feature the phar would already work but making that feature work requires a good amount of addition effort, hence the "pending" status.

Yes maybe Composer is not THE replacement for PEAR. But it fixes the limitations PEAR has, it has a small footprint, it is very easy to deploy on any platform, it makes it easy to maintain your private repositories, it promotes today's coding standards (see the psr-0 section http://nelm.io/blog/2011/12/composer-part-2-impact/), it has a great community behind it and an extensible code base. It is not by any means THE complete solution but it is in many respects THE BETTER one. Anyway, ultimately you decide where PHPUnit will go in the future, and you did a great job so far :) . Cheers

@edorian: Pirum 1.1 just means that when you load it via a PEAR repository it will load faster than it did before. The package name should then be pear-phpunit/PHPUnit.

As for having phpunit as part of your requirements, we don't have it yet, but eventually there will be a require-dev section just for that purpose. Only for when you work on a lib, and then having phpunit as part of the deps would be quite nice I think. Having composer support in PHPUnit itself would allow leveraging a few more features than with the PEAR fallback.

I haven't tried installing/running via composer yet, but I think one of the issues someone had was the fact that PHPUnit or one of its dependencies still relies on require_once and the include path to be setup, which just won't work with Composer. Autoloading we can do just fine.

@sebastianbergmann 100% in agreement. I install PHPUnit and all related packages (CodeCoverage, CPD, etc) with pear or Portage on Gentoo and all works fine (especially Portage). I have yet to see a need for another package manager. I will also note that I install numerous packages related to PHP (including separate PEAR ones) this way and other libraries for which I have written ebuilds: https://github.com/tatsh/tatsh-overlay/tree/master/dev-php

I know this is a feature of Composer that I have not seen with pear before, but on Gentoo we've had the ability to create new site roots automatically (and even create the settings file for use with web servers) with site-root-based packages (Drupal, phpmyadmin, WordPress etc) for a long time. webapp-config

@tatsh that doesn't address dependency hell, which I'm currently in with PHPUnit and different projects. I don't think people are voting to kill PEAR - but installing system level PEARs doesn't have to be the only way to use PHPUnit.

@glennpratt I imagine soon Gentoo will have something like ecomposer.class (just as we have epear.class which greatly simplifies installation) which will be a front-end to composer or similar. Mainly so that an ebuild can simply say to install Symfony-YAML and it will install as it does today, in /usr/share/php (and in all of Gentoo's php.ini files, /usr/share/php is by default in the include_path directive).

I'm not against a non-system-level PEAR or Composer (especially on systems where you do not get root access). I simply prefer system-level installation of packages, and prefer to code against 1 version of a library not several. I also do not want to maintain different versions of libraries for different sites.

Basic example: add a library to a site root at a specific version, and add the same library to another site. Now modify site a's version of the library and now you essentially have two different versions. It may have been necessary to modify for the site at hand (as a last resort) but instead what should be sought after is a library that is useful to both. And it would be safe to assume that the library would be at the same place for both sites in that case. Not only that, but I think this library should never be in the site root unless it absolutely must be (crappy shared hosts which no one should ever sign up for).

PHPUnit is for me no different than other packages. I want it system-wide so that I code against one version and it's always immediately available without worrying about the include path. I find it far easier to have Portage manage things than to manually do so or worry about your PATH variable and so forth.

@tatsh well, imho that's a bad practice. If a system administrator updates a library and a user om the server has assumed a version of the library, the entire project can fall apart. If you include the library and version of it in your project (composer.json), you have tested your code against it, and you know that it will work. Upgrading a library can break that.

i often don't understand such long discussions. it's not a decision to go left OR right, it's just ADD option2 to option1.

if someone prefer installing phpunit systemwide via pear perfect, if another person prefer installing different versions application wide via composer also perfect. both is a way to install it, both is ok, both have pros and cons.

but it's an AND not an OR. i really don't see a problem addon one file to the repo, which adds another way to install it.

@grEvenX I have not had such an issue as I manage most of the systems I am on, or I know what is on them (as any developer should). I do not think we are going to reach a point any time soon where system administrators (such as those who run shared hosts) will install PHP libraries system-wide for their sites hosted, other than some common PECL extensions ((mis)perception that C is safer than PHP). So it is either control the system(s) you have, know what libraries they have and which versions, or include your libraries in the project (site root). The latter is the last resort and is reserved for (bad) shared hosts.

I am not against any of these methods but I prefer the system-wide version as that is one version to remember instead of having to potentially do things like feature test in code (which always slows things down and makes code more complicated). I also do not see why I should keep more PHP code in the site root. I prefer to have the site root minimalist as possible. My deployed site roots do not even have non-minified/compiled CSS/JS files. The old days of everything in the site root (the case for Drupal, WordPress, and even big frameworks like Magento and similar) are gone for me and I am glad.

PEAR is an attempt to make libraries global and I am for it as it reduces duplication (this has many benefits besides saving disk space). Gentoo puts such libraries in /usr/share/php.

PHP is unique in being one of the few big languages right now that really lacks generic libraries and standardisation (and I am aware that the author of PHP specifically implied how he did not want PHP to be 'designed', which I consider a huge problem). When you look at even nodejs or Ruby which are newer than PHP, you can see that things are standardised: installation, how to include, documentation (Python) and documentation standards, and even examples are often provided. And, installation (sometimes optionally) generally makes the library globally available to all code of the same language (npm -g), which is useful. And no, I do not think PSR-* is even coming close to the level of these languages. Rather it is making things far more complicated than necessary (inconsistency in bracket usage, forcing usage of namespaces, and so on).

Once someone does it - it will happen. (I'm also working on it and got some issues fixed along the way) I'm also working on the .phar which will make composer support even easier because with have the same issues.

I can't stop people from commenting and discussion this (and I wouldn't if i could because .. well if people feel the need to do so there surely are reasons) but the roadmap is, hopefully, as clear as it gets.

Yea you're right, I think that might be a change needed in pirium. For the moment I have a phpunit.php file in the root of my project like so:

<?php$VENDOR_DIR=__DIR__.'/vendor';// This should really be part of composerset_include_path(get_include_path() .PATH_SEPARATOR.$VENDOR_DIR.'/pear-phpunit/File_Iterator');set_include_path(get_include_path() .PATH_SEPARATOR.$VENDOR_DIR.'/pear-phpunit/PHP_CodeCoverage');set_include_path(get_include_path() .PATH_SEPARATOR.$VENDOR_DIR.'/pear-phpunit/PHPUnit');set_include_path(get_include_path() .PATH_SEPARATOR.$VENDOR_DIR.'/pear-phpunit/PHPUnit_MockObject');set_include_path(get_include_path() .PATH_SEPARATOR.$VENDOR_DIR.'/pear-phpunit/PHP_Timer');set_include_path(get_include_path() .PATH_SEPARATOR.$VENDOR_DIR.'/pear-phpunit/PHP_TokenStream');set_include_path(get_include_path() .PATH_SEPARATOR.$VENDOR_DIR.'/pear-phpunit/Text_Template');// end of composerrequire_once($VENDOR_DIR.'/autoload.php');require_once($VENDOR_DIR.'/pear-phpunit/PHPUnit/phpunit.php');

And you can run your tests from the root of a symfony project for instance with:

php phpunit.php -c app

Alternatively you can use your project's composer.json "include-path" setting for each of those includes.

+1 for composer, though I do not wish the pear version to be abandoned as the projects manager likes it. I only wish he would set up a composer.json for the project so we would (composer fans) be able to grab it with our favorite tool. :)

Thank you Claylo, I will defenitely try your solution/effort out later today! edorian too has a composer try at https://github.com/edorian/phpunit-composer-try, I am still testing that one too. The best would be to be able to have a version that is able to "integrate" phpunit under my project. WHY? Portability, that's why. Everything that has a version number and is being depended on should be under vendor and not installed system wide imho. (at least in my case)

Due to the way composer manipulates include paths, you may have an issue if you already have PHPUnit installed globally. For the purposes of this test, I removed PHPUnit from my global include path.

The file that does the include_path manipulation is located in vendor/composer/include_paths.php.

As you probably know, there are lots of ways to manipulate runtime include-paths. Any one of those methods could allow you to keep a PHPUnit installed globally along with a local installation with a composer-based project. I'll leave that as a an exercise for you, Comment Reader.

PHPUnit 3.7.x (the main dev-master branch) is also installable. I still need to get the @pacakge-version@ tags, etc sorted out, but am working on that.

Hope you guys find this useful, and take time to try it out.

Finally, Sebastian -- if you'd like to discuss how (or if) to merge this work into the main phpunit (and associated dependency) repository, let's discuss. (Since I modified the 3.6.12 tag to support composer, I don't think a straight-up pull request will be enough.) I'm particularly interested in understanding what you'd consider valid for a test case to make sure the composer installs are working as they should.

Now, run php composer.phar install, which will install PHPUnit from my forks into a vendor subdirectory. A bin subdirectory will also be created. When the install is complete, you can run the composer-installed phpunit, like so:

A customized phpunit executable that uses the include path and other dependencies relative to the vendor directory.

A customized PHPUnit_Runner_Version class, which puts out the correct version number without relying on @package_version@ PEAR replace tasks (since doing so modifies the files and interferes with github-clone based installs) and tips the user off that they're using the localized version of PHPUnit.

A generator, which uses pear.claylo.com/Conductor to generate composer.json from your package.xml, hopefully minimizing the need to tweak your build process very much. (Note: as I write this comment, I realize I still need to tweak the generator script to actually update the custom PHPUnit_Runner_Version script. Will do that shortly!)

With all that done to phpunit, there's been much less required of the other dependent packages. In most cases, I just had to set up a composer.json file and re-tag to include the appropriate file along with the tag.

I'd very much like to see this become part of the PHPUnit official distribution/repository. However, if you're not diggin' it, it's far enough along now with the generator, etc., that I suppose I could just as easily continue maintaining the composer stuff in forks. I think that would be a shame, though, as I think people would be less likely to use my forked version than they would be to use the main repo.

Let me know what you think. If you approve, I'll start sending in pull requests. Thanks for considering the submission.

Sebastian, when I run against the config file instead of the Tests directory, the composer install works. Here's the complete session with a clean install from composer, followed by running the tests with phpunit.xml.dist.

I have followed your instructions and now have a Composer-installed PHPUnit running on my machine. On behalf of the people waiting for this: Thanks you!

The only thing I am unsure about is how you deal with the code transformations that happen on install when using the PEAR Installer. It is not just @package_version@ (and even if it were just that, I do not like the idea of not automatically replacing it). @php_bin@ is also used, for instance.

It eludes me why Composer does not support these transformations on install.

I can't speak for the composer guys on why they don't support those transformations, though I can speculate a little bit: one of the benefits that a composer-based install has is that if you specify a git origin as I did in the file you tested with, end-users can pull bleeding-edge versions from github without YOU having to repackage for deployment. (They have to configure composer to do this with the "minimum-stability": "dev" setting, so they knowingly step into the fray of unstable code.)

With that, if there are post-download file modifications, now the end-user has a modified git checkout, rather than a straight download, which causes updates to fail -- composer alerts the user that their local version has changed since it was installed, and aborts the update. I believe this is to prevent you from accidentally overwriting changes you may have meant to keep (such as an experimental patch, etc), without requiring composer to become a full-blown git client.

Other than @package_version@ replacements, PHPUnit uses @php_bin@ and @php_dir@. @php_dir@ is irrelevant with the localized installation, which leaves @php_bin@ ... which is only used in the executables.

I don't have a great answer for that, other than continued reliance on /usr/bin/env php -- Composer has no equivalent of the .pearrc that would manipulate a pointer to the preferred php executable. I assume the emphasis on project-localized installations includes the assumption that /usr/bin/env php should be good enough. If you know enough to know you want a localized install, you probably know enough to change your environment variables too if you need to.

I'm just guessing here, I don't know the Composer guys.

Question: would the way I've handled the @package_version@ replacement, and the non-support of @php_bin@ and @php_dir@ values prevent inclusion of Composer support in the main distribution of PHPUnit?

How about this: do not do any code transformations / text replacements. That way a Composer-installed PHPUnit would behave like a Git clone (showing @package_version@, for instance). Once this is in place we can have a look at PHPUnit_Runner_Version and change it to dynamically read version information from Git, for instance.

Ok, I'll remove the workarounds I have for @package_version@. What about my solution to regarding the phpunit executable? I don't think there's a way around what I've done (which is to load the composer-managed autoload.php) and still keep it a relevant/functional project-localized installation.

One of the things that Composer does while installing packages defined in composer.json is create an autoload "suite" of sorts -- it's a bunch of autoload-related code that supports class-to-file mappings, include-path manipulation, and psr-0 namespace-to-directory resolution.

In order to make sure that the actual files for the installed packages are loaded from the project's vendor/ directory, files in the project need to use vendor/autoload.php. That's how PHPUnit can be installed locally for a project managed by Composer, and globally managed by PEAR. I have explored the autoload.php files in all of the PHPUnit related packages, as well as those in Composer's autoload "suite", and crafted the composer.json file for phpunit/phpunit to Do The Right Thing -- namely, to load the local-path versions of PHPUnit's autoload.php files, rather than falling back to global include_path.

The only way to make sure that happens is to bypass the hard-coded paths relative to @php_dir@ in the executable ... unless you're interested in a somewhat deeper integration of composer support, where we'd add some additional checks to phpunit.php to have it also checks for a __DIR__."/../vendor/phpunit" location.

Even with that, Composer unfortunately does not have PEAR's "install as" capability, so phpunit.php gets installed in the bin dir as a symlink named phpunit.php instead of simply phpunit. The executable in the composer/ directory of my fork addresses both of those problems.

PHPUnit does not follow PSR-0 and it never will. All my software packages, including PHPUnit, use an automatically generated autoload map. Not only is this the best possible way to handle autoloading but for the components related to PHPUnit these maps also provide the information which files should be automatically blacklisted for code coverage.

TL;DR: A Composer-installed PHPUnit must use the Autoload.php files provided by the components involved.

I understand that, which is why I spent a considerable amount of time digging through the way Composer generates its include_path manipulations, handles file autoload types (where you specify a specific file that does non-psr-0 autoloading, etc.

However the hard-coded path relative to @php_dir@ that exists in the top-level phpunit.php file simply don't work with the location Composer puts the PHPUnit components.

The solutions:

Allow installation of a composer-only executable that loads Composer's autoload.php, which in turn points to PHPUnit's component-specific Autoload.php files.

Do a deeper integration of composer support, so that phpunit.php will attempt to look in the Composer-installed PHPUnit component locations.

TL;DR: If you wade through the vendor/autoload.php file the setup steps above generated, you'll find that eventually the local PHPUnit install does use the Autoload.php files provided by the PHPUnit components.

Will do, thank you. I will spend some time setting up build tools as you have them, so that I can include the composer-generator script in the build process. So, pull requests will arrive over the weekend, most likely.

@jamescarr I've been working through my forks of phpunit dependencies and updating build.xml to use the newly-created package2composer script -- my goal is to have completed all of those for both the 3.6.12 release tag as well as the 3.7.0 master by the end of this week.

Ultimately, I'd like Sebastian to be able to support Composer by doing little more than running his build script as usual. By hooking composer.json generation off of package.xml into the build process, I think it'll be pretty hands-off after that.