Is the libraries api intended to hold the definition of where certain libraries files are to be download from? This would include an index of a lot of libraries? In D7 it will be easier to deal with because of hook_library().

Or, is the intent for the definition to live in the jquery_ui module and the libraries api just has the ability to download it for the jquery_ui module?

I would suggest it is up to the modules to say where they need things downloading from. More sense from an architecture point of view, and also means the module maintainer can specify the version that matches their code.

I can't find docs on how the API works so I can't make more concrete suggestions...

We would need to add a key for the files location which is fine. At the same time, you shouldn't have 2 versions of the same library on a site in most cases. It will just create namespace conflicts and other issues.

I also recommended to read up most of the details in the Libraries API issue linked on the project page, which should answer 90% of all questions. Or just look into the queue: #618370: Reasoning Behind Libraries API?

Is the libraries api intended to hold the definition of where certain libraries files are to be download from?

This is one of the remaining major issues that needs to be discussed (elsewhere). We face a N:N relationship here, where neither of both parties can reliably provide standardized library information. So the logical conclusion would seem to additionally allow a N:1:N relationship, which would mean that Libraries API would define a couple of libraries, but installation profiles as well as other modules would still be able to provide their own definitions. However, as soon as there is a 1:N relationship in terms of the library definition, we'll most likely have a problem. This particular question requires to understand the underlying, multi-dimensional challenge, which Libraries API is trying to solve.

To actually allow downloading of libraries, we need to think about how Libraries API should access the files. Currently the $library['download url'] is for humans to visit and click on the right link, depending on the most recent version. So we need some automatism to do that.
Idea:
- Specify 'download url' like the following:http://jquery.com/download/jquery.[VERSION].js
or similar.
- Either declare the latest version statically (bad), or introduce a 'latest version callback' that determines the latest version (hard)
- When downloading, do:

As briefly discussed above, this seems to the crux of the issue: How do we find out which version to donwload? What this would mean is that everytime a new version of a library comes out, we would have to change the library information. Maybe as a first step, which would probably already help people a lot, we could pass the current version as an argument to the drush command. Then modules using Libraries API could document on their project page for instance: "Run drush libraries-download tinymce 4.3.1 for automatic installation of the library." Of course we would ideally like to find a way to automate all of that, but that would already be a huge step forward I think. Thoughts?

- The empty $name to download all needs to be an explicit --all option. Wysiwyg and other modules are going to register many libraries ;) That said, not sure whether this makes sense at all; dropping it entirely might make more sense.

- As soon as this works with (tar) archives, I'd be happy to commit alpha-quality code.

- We need a way to declare whether the downloaded library archive extracts itself into a folder already or not. The folder possibly has to be renamed. Quite potentially, it's going to be easier to define whether extraction should skip a folder level (à la patch -p1) and declare the folder name to extract into (which we already know == library name).

- Revision might make sense for version controlled, checked out repositories, but perhaps not for releases. We might need an additional release identifier option.

The empty $name to download all needs to be an explicit --all option. Wysiwyg and other modules are going to register many libraries ;) That said, not sure whether this makes sense at all; dropping it entirely might make more sense.

I like your --all idea, but I think we should just drop the support for that and push it to a separate issue.

As soon as this works with (tar) archives, I'd be happy to commit alpha-quality code.

We need a way to declare whether the downloaded library archive extracts itself into a folder already or not. The folder possibly has to be renamed. Quite potentially, it's going to be easier to define whether extraction should skip a folder level (à la patch -p1) and declare the folder name to extract into (which we already know == library name).

Drush Make handles that and renames the extracted folder to the library name. Probably seems like something we should push over to the Drush Make issue queue?

Revision might make sense for version controlled, checked out repositories, but perhaps not for releases. We might need an additional release identifier option.

It definitely depends on which type of download you're getting. You don't need the revision option when you're downloading a "file" type. Maybe this just means that the module developer needs to be more cautious with the version callback and variants? It should be up to the maintainer as to how that's handled.

It depends on how the library is downloaded. If TinyMCE is handled by just a file download, then drush libraries-download tinymce --tag="3.3" does not make sense. If it's handled via GIT or SVN though, then that's a valid call. Again, completely depends on the type of download.

The download URL often contains the version number. However, we could as well limit this feature to libraries providing a "latest stable" download URL only.

Maybe extend support for the "versions" array to allow the "download" options?.... Then we could have: drush libraries-download example --version="3.0" and it uses the download options from that version if it's defined?

I don't really get why .tar archive support is a Drush Make issue and not a Drush Core issue, but who am I...

Maybe this just means that the module developer needs to be more cautious with the version callback and variants? It should be up to the maintainer as to how that's handled.

Version callbacks and stuff are only invoked after the fact. This code here runs before the fact.

Libraries API's design idea is that integration code within Drupal mainly or ideally cares for major versions of a library only. As such, a download facility would ideally download the latest version of a library, if provided by the library vendor. After downloading, the Drupal/Libraries integration code determines the library's version and primarily differs per major version. In an ideal world, integration code is compatible with libFooBar version 3.00 to 3.99, and it only needs to change if and when a 4.x comes out.

Long story short, we don't know which exact version to download upfront, because anything < 4.x should work. If the vendor provides a static latest stable release download URL, then we're good to go.

However, we definitely don't want to hard-code any library version or revision to download. So let's simply leave that out entirely.

First off, thanks again for starting this, I really think this will be a game-changer for our users!

I finally got around to fiddling with this for a bit, and in trying to grasp it and make it work for a few example libraries, I changed this and that, and in the end I deviated enough from your patch, to roll a new version (after cleaning up after me, of course).
The patch might look pretty finished, or "full-fledged" (docs, etc...), but it's really just my proposal of how I would do it, what I think are the most usual use-cases, etc.
So if it is the case, "Your patch sucks, #39 is better!" is very much a possible response to this. :)

That said...
...this patch makes it so that you can/have to specify the version you want to download when you are downloading it, i.e.:drush libraries-make jquery 1.7.1
I think that is more in-line with our goal to not have to adapt our release-cycle(s) to that of any libraries.
It enables that by allowing the use of a "!version" placeholder in the download options, for example:$library['download']['url'] = 'http://code.jquery.com/jquery-!version.js';
(Yes, there's also a !variant placeholder.)
When downloading from a Git repository, you can specify$library['download']['tag'] = '!version';
And then the "1.7.1" tag will be checked out upon download.

NOTE: I used this as my primary resource for Drush Make (and also documented it in-code). It only lists the 'tag' parameter (and 'branch', etc.) for git, not for bzr or svn. So for now I only documented the whole thing for 'git'.

With this patch I was successfully able to download:

a static file

a git repository

a zip archive

a tar archive

For convenience, I've attached the .libraries.info files I used.

NOTE2: jQuery doeshttp://code.jquery.com/jquery-1.7.1.min.js vs.http://code.jquery.com/jquery-1.7.1.js
so you can't really specify the !variant placeholder as sometimes it's there and sometimes it's not. For that reason I've allowed the same kind of version- and variant-overloading we do elsewhere for this.
Because I was lazy, I then also utilized that for the different download methods for jQuery UI.
See the info files, for what I mean in detail.

NOTE3: I changed how the drush command detects success/failure, because you're version wasn't working with my version of Drush (4.5), but I think you a lot more into Drush than I am, so I would defer that to you. The way it is in this patch is only because that worked for me.

Unfortunately this hook isn't called if installing a distribution from the .make file via libraries[]. But I suppose a .make file would actually provide the straight up version number and not the "!version" ;-).

Unfortunately this hook isn't called if installing a distribution from the .make file via libraries[].

Hmm... make files will simply call drush make and not drush libraries-make so you have to specify the whole information yourself anyway.
In the long run, of course, it would be nice, if we could have some sort of automation, instead of having to duplicate that information, but I can't even start to wrap my hand around that right now :)

There should be no version numbers anywhere, neither in the library base definition nor on the drush command line.

Libraries API is always passive. Users smack an arbitrary library on it and it is supposed to figure out which library that is and which version. API consumers then react to that knowledge.

The only place where versions of a library are contained is in the extended library definition. However, those versions act as "minimum version and above", or in other words, the "last known version[s] that we know how to deal with."

This means that the library definition and most often also integration code can stay the same until there is an API change in the upstream library. Only if that happens, the library definition needs to be updated.

Users should always download the latest version of a library.

For this reason, there should never be any hard-coded version numbers anywhere in Libraries API and library definitions.

--
This is key to understand. Would be beneficial to document this in the handbook, or mayhaps even in the code docs.

Thinking through this, I really think that we should only support downloading of libraries, which provide a stable download URL. Or in precise other words:

If a library does not support this, then downloading is not supported. People need to ask the vendor to provide it. Possible in multiple ways, most simple being a symlink to an existing, version-specific file like the former.

If a library does not support this, then downloading is not supported. People need to ask the vendor to provide it. Possible in multiple ways, most simple being a symlink to an existing, version-specific file like the former.

As much as I would love for this to happen, most projects don't actually support this. Symfony has a pretty complex build system, and it might actually be a good test-bed to try this stuff out. I stuck the download options into the Symfony module, as well as the option to build from the sources via drush symfony-download. But, it doesn't support downloading the latest stable build.

Maybe this is where Packagist and Composer come in to handle that stuff? Either way, I'd say the latest patch is the way to go.

This is the composer.json from one project I looked at. The json does not declare any download locations. It just says name and version number. Composer looks into packagist or any repository, to find the download location. For the start, we should just rely on packagist, and not care about other download locations.

Libraries that are not in packagist, or not supported by composer at all, can still be downloaded the manual way.

Problems:
- What if more than one module adds this library? The usual module_invoke_all() will overwrite the version info.

------------------------

Next idea: composer.json file per module.
Problem: That's too much! We only need the "require" part.

--------------

Next idea: Key in the info file.

name = My module
; Yes, we want this to work also for 7.x, not just 8.x
core = 7.x
dependencies[] = libraries
composer[Monolog/monolog] = 1.0.*
; or
composer[] = guzzle/guzzle >= 2.0.0

The nice thing is, this would seamlessly integrate with Drupal 7.
Libraries can use hook_system_info_alter() to somehow block the module from being installed, while the library is missing. Or do this in some other way.

Composer module (the git extension) can have a commanddrush composer-install-all
to learn about missing libraries from system info, download them with composer, and store them in a vendor directory.
(that we still have to agree about)

Why libraries module and composer module?
Simple:
Composer is for the drush command, and is not installed on-site as a regular Drupal module.
Libraries is a regular Drupal module, and thus able to prevent modules from being installed, if their libraries are not present.

------------

What about class loading?
There are different ways to do this, and it would be great to have choice:
- libraries can simply include the autoload.php file generated by composer. Done. Yes, it should be libraries module which does that, because composer module is not available at request time.
- classloader module could register the namespaces found in the composer-generated autoload_namespaces.php
- xautoload module could do the same.

Imo, libraries should simply make sure that one of those solutions is in place.

-------------------

Location of the composer vendor directory?
I'd say we should not mix that with sites/all/libraries. Why? There can be things in sites/all/libraries which totally contradict composer.
So, I suggest
- sites/all/composer
- sites/all/vendor
- sites/all/composer/vendor
Is there anything in D8 already? It could also be a different location in D8, because sites/all/ is no longer the "place for everything".

What about composer dependencies in libraries downloaded manually?
E.g. your custom library downloaded from github contains a composer.json. However, that library itself is not in composer.
Solution:
- Modules can declare in hook_libraries_info(), that a given library contains a composer.json. This is to prevent ridiculously deep file scans, or false positives caused by in-development files.
- Composer / libraries use this info to refuse module enabling, or to download those additional libraries.

This will explode, until you install libraries-7.x-3.x, which will use hook_system_info_alter() to clean up the dependencies[].
Problem: It can also explode in other modules' implementation of hook_system_info_alter(). So better not..
[/crazy idea]

Rethinking this:
Actually there is no reason why this all has to be parts of libraries module.

There are several reasons to make this a separate module:
1. There is very little in current libraries module, which this composer stuff could build on.
2. Composer libraries should live outside of sites/all/libraries (imo)
3. It is easier to specify a dependency for a new module (e.g. "composer_libraries") than for a specific version of libraries module.
4. The new module can have its own issue queue, versioning etc.
5. Libraries can continue to do the job for non-composer libraries.

The little problem: The most logical name for this module ("composer") is already taken by a drush extension. What we need is a regular module which can be installed on-site. I don't know whether it is feasible to turn composer from a drush extension project to a regular module, and what would be the implications of that.

Composer module (the git extension) can have a command
drush composer-install-all

Was playing around with that idea: #1452984: drush composer-mass install ... Would be nice to just download all your modules, run a composer-install-all and have everything work would be nice. Currently, it's executed on the drush dl hook, but tighter integration would be great.

Composer is for the drush command, and is not installed on-site as a regular Drupal module.

Correct. Bootstrapping the vendor code in your module means loading autoload.php. There is the Composer Autoload module that'll do that for you though.

Vendor code location is handled by autoload.php, so we should not worry about that. composer.json's vendor-dir in config is responsible for that. All we need to do is load autoload.php.

Is there anything in D8 already? It could also be a different location in D8, because sites/all/ is no longer the "place for everything".

It's in /core/vendor in Drupal 8.

What about class loading? libraries, classloader module, xautoload. Imo, libraries should simply make sure that one of those solutions is in place.

Libraries, XAutoload, Classloader, Composer Autoload, they all accomplish different things. Although they're hitting similar goals, they all accomplish them differently. Whether a module wants to use one either the other should be up to the module maintainer. XAutoload and Classloader seem to be the most similar. XAutoloader does provide a bit more flexibility than Classloader though.

Where Libraries differs from them all is that it can be used to load JavaScript/CSS vendor code. Been playing around with the idea of having a Require.js config written for JavaScript/CSS components from Composer (think a autoload.js file): https://github.com/robloach/components .... But that's FAR off, and I'm not even sure that's the correct direction for JavaScript/CSS assets, but it is something to play with. This would seem like a PHP-implementation of Component, JamJS, Bower, or any of the other web browser package managers out there.

Actually, you can declare your own packages that aren't on Packagist in the repositories key.

This is cool.
So, we have the choice between
- composer.json per module, with "require" and "repositories".
- module info file, with "composer.require[] = .." and "composer.repositories[] = .."

That's a typical DX bikeshed :) I have my own preference (more on that below), but it is not really a blocker for anything. Both should work ok technically, if the only purpose is to know which composer libraries are needed.

Composer has almost become the defacto-standard for maintaining and defining PHP packages. It comes with many features that would be silly to ignore

Drupal modules are far away from being "PHP packages". Thy are typically designed to only just work in Drupal, and nowhere else. And they are hosted on drupal.org, (except for custom modules).

This doesn't have to stay this way forever, btw. It is attractive to think about Drupal modules shipped with composer-powered PHP packages. But that's a long way, and requires more thinking than just a composer.json put in the module dir. As long as modules are just modules, a per-module composer.json will confuse more than it helps (imo).

Currently, if you want more of your Drupal module reusable outside of Drupal, the solution is probably to
- have the reusable stuff in a composer repository
- have the drupal-specific stuff hosted on drupal.org, with a composer dependency

Generation of a static class-map so that PSR-0 lookups arn't even needed (composer install --optimize)

I doubt the performance benefit of that in a Drupal context, if compared to xautoload (with or without APC cache).
It might be useful for classes that are used a lot, or for projects that don't have a lot of classes.
And of course, if your PSR-0 implementation does not scale to a high number of namespaces.
Some benchmarks could be nice.

Allows use of "provide" to allow packages to ask for implementations of a certain interfaces. Like a module asking for an implementation of a Plugin without having to worry about what that Plugin is.

"provide" is cool.
But Drupal modules are not composer packages.
- composer packages (and even custom-downloaded libraries) can provide a composer.json with "provide".
- Drupal modules should not "provide" anything, because composer does not treat them as packages.

-------------------

Ok, before I continue:
I think the major question is: Should we treat Drupal modules as composer packages? In D7? What would be necessary to do that, what would be the implications?
Or can composer packages include Drupal modules?
I think this is all very sexy, but goes far beyond the problems we want to solve in this issue: Make composer libraries available to Drupal, and let modules declare which libraries they need.

Vendor code location is handled by autoload.php, so we should not worry about that. composer.json's vendor-dir in config is responsible for that. All we need to do is load autoload.php.

[..]

It's in /core/vendor in Drupal 8.

This does not solve any problem.
What we want is one single place within a Drupal install for composer libraries.
/core/vendor is not an option, because that's for stuff shipped with core.
So it has to be sth like sites/all/composer, or sites/all/composer/vendor for D7, and something else for D8.

"composer.json's vendor-dir in config" would only help if we have one global composer.json for the Drupal install.
This is not going to happen. Or if it happens, then it is going to be generated by composer or libraries or whatever, and then we still need to make that decision deliberately.

Having modules negotiate where they want their composer vendor directories is asking for trouble.
The module dir itself is not an option, because this is wiped on update, or would make a mess in git. Also, libraries would no longer be shared if we have a per-module vendor dir.

- Modules provide their composer dependencies in one way or another. Modules themselves are not treated as composer packages.
- drush composer creates a sites/all/composer/composer.json (D7) or just composer/composer.json (D8), with all dependencies assembled.
- drush composer does the usual composer recursive downloading, starting with that generated composer.json. Vendor directory will be sites/all/composer/vendor/, or just composer/vendor/

Re #57:
This all refers to what modules are in D7.
If #1398772: composer.json for modules and other changes land in D8, then some of my arguments might no longer apply - but should still be considered for the D7 version.

My understanding when talking about adding composer files to modules and such was not that they would be considered packages that can be used out of drupal, since in fact the type can be set to drupal-module which is clearly drupal specific. The benefit is that we do not have to a) maintain a drupal specific package management system (drush make), b) easier for people to jump into drupal since composer is more widespread, c) bonus features and future improvements from composer.

I have given substantial thought to what is necessary to replace drush make with composer which it sounds like this conversation is on the verge of working through.

It seems rather silly to make a distinction between modules and other code which are all reusable units and have the same management need, just that modules are drupal specific (not sure if everyone is aware of https://github.com/composer/installers/blob/master/src/Composer/Installe... and the related code in there). Meaning that modules would remain drupal specific just managed by composer instead of drush make which makes a lot of sense.

What is the reasoning for "forking" the composer.json file into .info files? This seems like a needlessly introduced Drupalism. If anything we should be looking at doing the reverse since composer files contain more information than just dependencies (like title and description) and are only missing a few drupalisms. Keeping them separate also makes sense, but dumping composer.json into .info file seems counter intuitive.

#61
- a module can not be downloaded separately (only the "project" can).
- a package can not be "enabled".

If composer gets to see a package with a number of modules, and some of those modules have dependencies, then how can composer know which of these are enabled? Because we don't need to download dependencies for disabled modules.

The issue about disabled modules is an issue with drush make currently and would have to be code that sits on top..or people splitting out the modules within their projects into more logical units, but both of those problems are not composer specific and exist now. In fact they exist in the composer work...you simply download all dependencies regardless of whether you use the code.

- a module can not be downloaded separately (only the "project" can).

Sure that makes sense, I tend to think of module as a project since more are or you depend on the primary one anyhow, but yes they are separate...again this problem has not been solved by drush make either so we are just switching packaging systems and this needs to be solved separately. It all boils down to the fact that projects should be separated if they are to be downloaded separately.

I get the idea, whatever D8 does with composer, should rather live in core. Especially, if we want to use that for module downloading.
For D7, we should find a conservative solution in contrib, we cannot just reinvent the info file and replace it with something else.

Also I consider our problem scope for D7 very limited: Find a solution to automatically download 3rd party stuff with composer, based on dependency information found in modules.

Or maybe I am on the wrong issue, because this is tagged 8.x-3.x-dev.

Sure that makes sense, I tend to think of module as a project since more are or you depend on the primary one anyhow, but yes they are separate...again this problem has not been solved by drush make either so we are just switching packaging systems and this needs to be solved separately.

Ok let's assume we replace drush make with composer. What does this change?
"Projects" become "packages", and can have a composer.json if you like.
Modules are still modules. We can have more than one per package, they can have dependencies to other modules and to packages, and we need to know which of them are enabled.
Packages can contain some code that is always available once the package is downloaded, and other code that is only available if a specific module within the package is enabled.

Also consider that modules in D8 are also in some way Symfony "bundles" (or they contain them, or whatever), so this adds a 3rd level to the problem: Package vs module vs bundle. And bundles have yml files..