''Disclaimer: this page is not authored by a p2 expert, I merely reproduced some information which I found difficult to collect -- Stephan''

+

''Disclaimer: this page is a community maintained page, not part of the p2 project, per se, and is fairly specific to the Eclipse.org infrastructure. The p2.mirrorsURL concept was implemented by p2 team primarily for the Eclipse.org Infrastructure, to allow p2 updates and installs to use its worldwide system of mirrors, though other companies or networks can use in a similar way for their specific mirroring system.''

+

+

''Thanks to Stephan Herrmann for recognizing the need and getting this page started, updating it occasionally with new information, and trying to keep us all 'honest'. Such contributions make Eclipse better.''

==Enabling your repository to leverage mirrors==

==Enabling your repository to leverage mirrors==

Line 35:

Line 37:

===How to add?===

===How to add?===

+

+

==== via site.xml ====

+

+

Note: site.xml files are pretty "out dated" and normally should not exist on the repo site itself, even if you use it temporarily with some old tools to produce your metadata, its recommended you not leave on the repo site itself. The reason is, is that in some rare circumstances, p2 may misinterpret this and think it has found an old update manager site and try to handle it as such.]

+

If you use a <code>site.xml</code> the word goes that this is where you should specify this property. The [[Equinox/p2/Publisher]] ''should'' automatically

If you use a <code>site.xml</code> the word goes that this is where you should specify this property. The [[Equinox/p2/Publisher]] ''should'' automatically

transfer this information into your metadata.

transfer this information into your metadata.

Line 40:

Line 47:

If you don't use a <code>site.xml</code> you may consider manually editing <code>artifacts.xml</code> (which of course can be automated using, e.g.,

If you don't use a <code>site.xml</code> you may consider manually editing <code>artifacts.xml</code> (which of course can be automated using, e.g.,

an xsl template, like this [http://dev.eclipse.org/viewcvs/viewvc.cgi/org.eclipse.mylyn/org.eclipse.mylyn.releng/bin/p2.xsl?view=markup&root=Mylyn_Project p2.xsl] from the Mylyn project). When manually editing <code>artifacts.xml</code> add the property to the existing list of properties right at the top of the document. Don't forget to update the <code>size</code> attribute of the enclosing <code><properties></code> element.

an xsl template, like this [http://dev.eclipse.org/viewcvs/viewvc.cgi/org.eclipse.mylyn/org.eclipse.mylyn.releng/bin/p2.xsl?view=markup&root=Mylyn_Project p2.xsl] from the Mylyn project). When manually editing <code>artifacts.xml</code> add the property to the existing list of properties right at the top of the document. Don't forget to update the <code>size</code> attribute of the enclosing <code><properties></code> element.

Others have reported that the [http://help.eclipse.org/indigo/index.jsp?topic=%2Forg.eclipse.platform.doc.isv%2Fguide%2Fp2_mirror.html p2.mirror ant task] can be used to insert properties like p2.mirrorsURL while mirroring the build output to the final destination. The following two snippets may serve for inspiration:

Others have reported that the [http://help.eclipse.org/indigo/index.jsp?topic=%2Forg.eclipse.platform.doc.isv%2Fguide%2Fp2_mirror.html p2.mirror ant task] can be used to insert properties like p2.mirrorsURL while mirroring the build output to the final destination. The following two snippets may serve for inspiration:

Line 70:

Line 79:

These snippets were extracted from this [http://git.eclipse.org/c/gyrex/gyrex-releng.git/tree/org.eclipse.gyrex.releng/builder/configs/repository/build.xml build.xml] from the gyrex project.

These snippets were extracted from this [http://git.eclipse.org/c/gyrex/gyrex-releng.git/tree/org.eclipse.gyrex.releng/builder/configs/repository/build.xml build.xml] from the gyrex project.

+

+

==== Custom Eclipse applications ====

+

+

Some projects have written Eclipse applications to set or change or remove the p2.mirrorsURL property, such as see [[WTP/Releng/Tools/addRepoProperties]] as one example. You might be able to re-use this code, as is, or feel free to copy (to your own namespace) and customize for your project.

+

+

==== How to avoid .blobstore for *.jar.pack.gz files ====

+

+

Technically, repositories that use .blobstore to store pack.gz files can be used just like any other repository with the same p2.mirrorsURL attribute as described in this page. (See {{bug|383145}}.) But, many people prefer to avoid .blobstore, since it goes into a "dot directory" (hidden by default for most systems and tools) it can be confusing to some people and tools.

+

+

In some scenarios and tools, apparently, the .blobstore form is never created, and the pack.gz files stored as siblings of the .jar files, without any action being needed.

+

+

But, if you find your scenarios and tools are creating one, and you wish to avoid it, you basically need to specify a property in your artifacts.jar/xml before creating or mirroring the pack.gz files:

+

+

<nowiki>

+

<property name="publishPackFilesAsSiblings" value="true/">

+

</nowiki>

+

+

+

One easy way to do this, if you are starting from scratch, is to have a "near empty" artifact.xml file, and then in your p2.mirror "destination" attribute, specify you'd like to use that near empty artifact.xml file, via the "format" attribute, to have it used as a template from which to get properties from ... publishPackFilesAsSiblings being one of the possible properties.

+

+

For example, the Platform builds use this sort of template when getting started:

And, don't worry, it sounds more complicated that it is ... once you get it set up the first time.

===How to test?===

===How to test?===

−

{{Tip|Avoid Mirror invalidation|In some cases, it is easy to temporarily "invalidate" your mirror list; that is, make it appear, temporarily, that there are no mirrors. This happens if you change the directory timestamps that contains the artifact.jar as you edit. It is the directory that is used in p2.mirrorsURL, so it will appear there are no mirrors if the timestamp changes to current time, even though all the mirrors still have your actual artifacts. But -- this is the tip -- the directory's timestamp only changes if files are added or removed, but not if simply the content of a file changes. So, you can avoid temporarily invalidating mirrors by copying the artifact.jar file to some 'temp' directory, unzip it it there, update it, and zip it back to artifacts.jar and then copy it back to the original location (all carefully of course). If you do somehow invalidate the mirrors list temporarily, it should not take too long to get some mirrors back (a few hours, or overnight for sure), since there's no real work to do ... just update the fact that they are in sync}}

−

To test, first, you may paste the full URL you specified as the value in the p2.mirrorsURL attribute into your browser (now collapse &amp;amp; to &amp;). The result should be an XML file in your broser with lots of mirror URLs. Pasting one of these URLs into the browser should in turn take you to the directory holding your <code>artifacts.jar</code> and its peers. (You may have to add "artifacts.jar" or "content.jar", etc, to the URL, if directory browsing is not allowed on a mirror).

+

If you change your artifacts.jar/xml file, you may have to wait a a while, for mirrors to get "in sync" before being able to test, such as a few hours, or at most overnight. Seriously, it might "appear" to be mirrored, but not, so give it at least half a day before starting to worry.

−

Secondly, to test whether install successfully uses mirrors (and which ones) you may use the following options file (stored, e.g., as <code>/tmp/p2options</code> or simply placed in .options file in 'eclipse' directory (the default location)):

+

{{Tip|Avoid Mirror invalidation|In some cases, it is easy to temporarily "invalidate" your mirror list; that is, make it appear, temporarily, that there are no mirrors. This happens if you change the directory timestamps that contains the artifact.jar as you edit. It is the directory that is used in p2.mirrorsURL, so it will appear there are no mirrors if the timestamp changes to current time, even though all the mirrors still have your actual artifacts. But -- this is the tip -- the directory's timestamp only changes if files are added or removed, but not if simply the content of a file changes. So, you can avoid temporarily invalidating mirrors by copying the artifact.jar file to some 'temp' directory, unzip it it there, update it, and zip it back to artifacts.jar and then copy it back to the original location (all carefully of course). If you do somehow invalidate the mirrors list temporarily, it should not take too long to get some mirrors back (a few hours, or overnight for sure), since there's no real work to do ... just update the fact that they are in sync. }}

+

+

====Confirm your artifacts site is being mirrored====

+

+

The first test you do should be with a browser, to confirm the download.php script does indeed return some mirrors for your site. To do this, paste the full URL you specified as the value in the p2.mirrorsURL attribute into your browser (now collapse &amp;amp; to &amp;). The result should be an XML file in your browser with a list of mirror URLs. For an example, you can paste the following URL into your browser, to see who is mirroring the Juno SR1 artifacts:

Pasting one of these resulting URLs into the browser should in turn take you to the directory holding your <code>artifacts.jar</code> and its peers. (You may have to add "artifacts.jar" to the end of the URL, if directory browsing is not allowed on a particular mirror).

+

+

Note that not all "builds" (or repositories) are mirrored. For example, nightlies, integration builds, staging sites, etc. are explicitly excluded. See [[IT_Infrastructure_Doc#Use_mirror_sites.2Fsee_which_mirrors_are_mirroring_my_files.3F| IT Wiki page]] for details.

+

+

====Confirm your p2.mirrorsURL works with p2 operations====

+

+

{{note|You can not test from eclipse.org|For those with shell accounts, or those thinking it might be good to test as part of a Hudson job ... you can not. The "internal network" at the Eclipse Foundation and the "mirroring system" knows not to try and get something from a mirror, but instead to get it directly ... since it is right there, close by, just 3 Gbs away. Hence, to test if your mirrors are working as expected, you need to test from you own network. There might be similar issues or limitations on your company's network (hypothetically) if they have their own redirection/proxy/mirroring systems in place. If in doubt, you can ask.}}

+

+

To test whether install successfully uses mirrors (and which ones) you may use the following options file (stored, e.g., as <code>/tmp/p2options</code> or simply placed in .options file in 'eclipse' directory (the default location)):

org.eclipse.equinox.p2.core/debug=true

org.eclipse.equinox.p2.core/debug=true

org.eclipse.equinox.p2.core/artifacts/mirrors=true

org.eclipse.equinox.p2.core/artifacts/mirrors=true

Line 84:

Line 142:

Then you may want to unpack a fresh Eclipse package and change into the <code>eclipse</code> directory.

Then you may want to unpack a fresh Eclipse package and change into the <code>eclipse</code> directory.

−

Finally, use the normal Eclipse UI to do an "update" or for more focus, you can use the [[Equinox/p2/Director p2Director application]] to install your feature using a command line similar to the following:

+

Finally, use the normal Eclipse UI to do an "update" or for more focus, you can use the [[Equinox/p2/Director_application| p2 Director application]] to install your feature using a command line similar to the following:

This should install your feature and print to the console where each artifact was downloaded from.

This should install your feature and print to the console where each artifact was downloaded from.

Line 96:

Line 157:

2>&1 | tee fulllog.txt

2>&1 | tee fulllog.txt

−

Once "mirror selection tracing" is in such a log file, you can produce a quick "histogram" of frequencies of what mirrors are being selected with a command similar to

+

Once "mirror selection tracing" is in such a log file, you can produce a quick "histogram" of frequencies of what mirrors are being selected with a command similar to the following (which was inspired by Markus Knauer in {{bug|372504#c9}}):

Only if you are developing a repository for your own "private" or company use and do not want it to go to other mirrors for any reason. This is certainly legitimate for non-Eclipse projects. But, also, occasionally companies mirror an Eclipse repository for their own internal use, and if they do not want their internal users to be redirected to other sites, they may need to "manually" remove the attribute from their mirrored version of the repository. Follow (or contribute to) {{bug|262005}} for requested enhancements to the p2.mirror task to make this easier.

+

+

==Can the p2.mirrorsURL property be harmful?==

+

+

Yes. If it is incorrect. For example, if someone has released a "3.7" version, but their p2.mirrrorsURL value still points to "3.6 repositories" it will result in a lot of "wild goose chases" for artifacts it will never find and end up going back to eclipse.org anyway (and causing the bandwidth problems there, it was designed to help with).

+

+

So, please make sure your values are accurate and work as expected. Nicolas Bros, in {{bug|368826#c15}} contributed this handy "one liner" for listing your p2.mirrorsURL values in your subdirectories of repositories (which could be easily re-written as a non-one-liner, if you desire to have in your own script file):

+

+

<pre><nowiki>

+

find . -type f -name 'artifacts.jar' -or -name 'artifacts.xml' \

+

| while read i; \

+

do echo "$i"; \

+

if [[ "$i" =~ .*/artifacts.jar ]]; \

+

then echo " "$(unzip -p "$i"|grep "p2.mirrorsURL"); \

+

else echo " "$(cat "$i"|grep "p2.mirrorsURL"); \

+

fi; \

+

done

+

</nowiki></pre>

+

+

Plus, there has, in the past, been discussions if the use of mirrors by p2 is a good thing, such as see the [http://dev.eclipse.org/mhonarc/lists/p2-dev/msg04437.html p2-dev list], but until the Eclipse Foundation gets its own world-wide server farm, it is the only workable solution known for the millions of installs and updates that Eclipse users do each year. (Note, it was primarily designed for installs and updates ... and it is "using it for builds" that appear to be what is mostly complained about). Not to mention, many improvements and bug fixes have been made over the years, and all users and committers are encouraged to open new bugs or feature requests (and/or contribute patches) if they find bugs or inadequate behavior.

Many of these have not been used/updated for a while, so they are listed here for information only. Its hoped you can find the tools useful (or help refine them) but use at your own risk ... and on non-production test systems, at least initially.

Revision as of 05:11, 16 October 2012

Disclaimer: this page is a community maintained page, not part of the p2 project, per se, and is fairly specific to the Eclipse.org infrastructure. The p2.mirrorsURL concept was implemented by p2 team primarily for the Eclipse.org Infrastructure, to allow p2 updates and installs to use its worldwide system of mirrors, though other companies or networks can use in a similar way for their specific mirroring system.

Thanks to Stephan Herrmann for recognizing the need and getting this page started, updating it occasionally with new information, and trying to keep us all 'honest'. Such contributions make Eclipse better.

Enabling your repository to leverage mirrors

The p2.mirrorsURL should be added to your metadata so that p2 will see the list
of available mirrors to choose during installation.

Who should add?

Everyone. Every build. At least, everyone at eclipse.org, that is. Other companies or networks may have their own mirroring system and scripts which are similar to this, but these instructions are specific for the Eclipse Foundation infrastructure.

It is true that not each and every build or repository is mirrored (e.g. integration builds usually are not) but having the property there, even if there are no mirrors, should not hurt anything ... p2 knows to simply fall back to the normal "originally requested" location if no mirrors are found (usually 'http://download.eclipse.org/...').

It is also true that those mirroring eclipse downloads and repositories have some choice in what they mirror (there are several options, such as "full", "releases only", "platform only" and similar). So, everyone should add the property, since you can never be sure if/when your repository might be mirrored. If it is not mirrored, by anyone, at some point in time, having the property there should not hurt anything ... p2 knows to simply fall back to the normal "originally requested" location if no mirrors are found (usually 'http://download.eclipse.org/...').

Where to add?

Adding this property to artifacts.xml is sufficient, since p2 uses a list of mirrors while retrieving artifacts, not the metadata itself. (Hence, no need to have one in content.jar/xml or compositeContent.jar/xml files). Don't forget, artifacts.jar
must be updated or replaced after changing artifact.xml. (If both exist, artifacts.jar will always be fetched before artifacts.xml.)

Should composite repos be treated, too, or just their children (leaves)?

Just the leaves. No need to to specify the property in compositeArtifacts.xml/jar.

yes, &amp; is 5 characters, in this context the escaped version is required.

the last part, &amp;protocol=http is important, since it restricts the list of mirrors to those that support http. Monitor bug 349679 for status/progress on p2 working with ftp sites. (In general, such as for zip files linked on download pages, and similar, the 'protocol' parameter is optional for the download.php script and when omitted both http and ftp mirrors are returned).bug 349679 seems to have been a red-herring (or some special case) so ftp and http should both work fine, so there is no need to specify the protocol attribute and the mirrors script (download.php) will return a list that contains both types ... and the more there are to try, the better.

This p2.mirrorsURL property will end up giving p2 a list of 0 to 100 or so URLs to try and fetch artifacts from. If your project is not mirrored (or, there is some other problem with mirrors) p2 will fall back automatically to using the originally requested site location (usually 'http://download.eclipse.org/...').

How to add?

via site.xml

Note: site.xml files are pretty "out dated" and normally should not exist on the repo site itself, even if you use it temporarily with some old tools to produce your metadata, its recommended you not leave on the repo site itself. The reason is, is that in some rare circumstances, p2 may misinterpret this and think it has found an old update manager site and try to handle it as such.]

If you use a site.xml the word goes that this is where you should specify this property. The Equinox/p2/Publishershould automatically
transfer this information into your metadata.

If you don't use a site.xml you may consider manually editing artifacts.xml (which of course can be automated using, e.g.,
an xsl template, like this p2.xsl from the Mylyn project). When manually editing artifacts.xml add the property to the existing list of properties right at the top of the document. Don't forget to update the size attribute of the enclosing <properties> element.

copying properties from existing repositories during mirroring

Others have reported that the p2.mirror ant task can be used to insert properties like p2.mirrorsURL while mirroring the build output to the final destination. The following two snippets may serve for inspiration:

These snippets were extracted from this build.xml from the gyrex project.

Custom Eclipse applications

Some projects have written Eclipse applications to set or change or remove the p2.mirrorsURL property, such as see WTP/Releng/Tools/addRepoProperties as one example. You might be able to re-use this code, as is, or feel free to copy (to your own namespace) and customize for your project.

How to avoid .blobstore for *.jar.pack.gz files

Technically, repositories that use .blobstore to store pack.gz files can be used just like any other repository with the same p2.mirrorsURL attribute as described in this page. (See bug 383145.) But, many people prefer to avoid .blobstore, since it goes into a "dot directory" (hidden by default for most systems and tools) it can be confusing to some people and tools.

In some scenarios and tools, apparently, the .blobstore form is never created, and the pack.gz files stored as siblings of the .jar files, without any action being needed.

But, if you find your scenarios and tools are creating one, and you wish to avoid it, you basically need to specify a property in your artifacts.jar/xml before creating or mirroring the pack.gz files:

<property name="publishPackFilesAsSiblings" value="true/">

One easy way to do this, if you are starting from scratch, is to have a "near empty" artifact.xml file, and then in your p2.mirror "destination" attribute, specify you'd like to use that near empty artifact.xml file, via the "format" attribute, to have it used as a template from which to get properties from ... publishPackFilesAsSiblings being one of the possible properties.

For example, the Platform builds use this sort of template when getting started:

And, don't worry, it sounds more complicated that it is ... once you get it set up the first time.

How to test?

If you change your artifacts.jar/xml file, you may have to wait a a while, for mirrors to get "in sync" before being able to test, such as a few hours, or at most overnight. Seriously, it might "appear" to be mirrored, but not, so give it at least half a day before starting to worry.

Avoid Mirror invalidationIn some cases, it is easy to temporarily "invalidate" your mirror list; that is, make it appear, temporarily, that there are no mirrors. This happens if you change the directory timestamps that contains the artifact.jar as you edit. It is the directory that is used in p2.mirrorsURL, so it will appear there are no mirrors if the timestamp changes to current time, even though all the mirrors still have your actual artifacts. But -- this is the tip -- the directory's timestamp only changes if files are added or removed, but not if simply the content of a file changes. So, you can avoid temporarily invalidating mirrors by copying the artifact.jar file to some 'temp' directory, unzip it it there, update it, and zip it back to artifacts.jar and then copy it back to the original location (all carefully of course). If you do somehow invalidate the mirrors list temporarily, it should not take too long to get some mirrors back (a few hours, or overnight for sure), since there's no real work to do ... just update the fact that they are in sync.

Confirm your artifacts site is being mirrored

The first test you do should be with a browser, to confirm the download.php script does indeed return some mirrors for your site. To do this, paste the full URL you specified as the value in the p2.mirrorsURL attribute into your browser (now collapse &amp; to &). The result should be an XML file in your browser with a list of mirror URLs. For an example, you can paste the following URL into your browser, to see who is mirroring the Juno SR1 artifacts:

Pasting one of these resulting URLs into the browser should in turn take you to the directory holding your artifacts.jar and its peers. (You may have to add "artifacts.jar" to the end of the URL, if directory browsing is not allowed on a particular mirror).

Note that not all "builds" (or repositories) are mirrored. For example, nightlies, integration builds, staging sites, etc. are explicitly excluded. See IT Wiki page for details.

Confirm your p2.mirrorsURL works with p2 operations

You can not test from eclipse.orgFor those with shell accounts, or those thinking it might be good to test as part of a Hudson job ... you can not. The "internal network" at the Eclipse Foundation and the "mirroring system" knows not to try and get something from a mirror, but instead to get it directly ... since it is right there, close by, just 3 Gbs away. Hence, to test if your mirrors are working as expected, you need to test from you own network. There might be similar issues or limitations on your company's network (hypothetically) if they have their own redirection/proxy/mirroring systems in place. If in doubt, you can ask.

To test whether install successfully uses mirrors (and which ones) you may use the following options file (stored, e.g., as /tmp/p2options or simply placed in .options file in 'eclipse' directory (the default location)):

This should install your feature and print to the console where each artifact was downloaded from.
If p2.mirrorsURL is not properly configured no such messages will be shown.

You may want to "capture" the console results in a separate file by adding something like the following to the end of your 'eclipse' command (at least on Linux):

2>&1 | tee fulllog.txt

Once "mirror selection tracing" is in such a log file, you can produce a quick "histogram" of frequencies of what mirrors are being selected with a command similar to the following (which was inspired by Markus Knauer in bug 372504#c9):

If things are working well, you should see few, if any, times that "download.eclipse.org" is selected. As one example, here are the results of one run, updating Indigo SR1 Java EE IDE to SR2 from the built-in repositories:

When not to use the p2.mirrorsURL property?

Only if you are developing a repository for your own "private" or company use and do not want it to go to other mirrors for any reason. This is certainly legitimate for non-Eclipse projects. But, also, occasionally companies mirror an Eclipse repository for their own internal use, and if they do not want their internal users to be redirected to other sites, they may need to "manually" remove the attribute from their mirrored version of the repository. Follow (or contribute to) bug 262005 for requested enhancements to the p2.mirror task to make this easier.

Can the p2.mirrorsURL property be harmful?

Yes. If it is incorrect. For example, if someone has released a "3.7" version, but their p2.mirrrorsURL value still points to "3.6 repositories" it will result in a lot of "wild goose chases" for artifacts it will never find and end up going back to eclipse.org anyway (and causing the bandwidth problems there, it was designed to help with).

So, please make sure your values are accurate and work as expected. Nicolas Bros, in bug 368826#c15 contributed this handy "one liner" for listing your p2.mirrorsURL values in your subdirectories of repositories (which could be easily re-written as a non-one-liner, if you desire to have in your own script file):

Plus, there has, in the past, been discussions if the use of mirrors by p2 is a good thing, such as see the p2-dev list, but until the Eclipse Foundation gets its own world-wide server farm, it is the only workable solution known for the millions of installs and updates that Eclipse users do each year. (Note, it was primarily designed for installs and updates ... and it is "using it for builds" that appear to be what is mostly complained about). Not to mention, many improvements and bug fixes have been made over the years, and all users and committers are encouraged to open new bugs or feature requests (and/or contribute patches) if they find bugs or inadequate behavior.

Avoiding mirrors, even when using p2.mirrorsURL

You can disable the automatic mirror selection mechanism by setting the VM argument eclipse.p2.mirrors to false (remember that vm args have to be last on the command line preceded by the switch -vmargs). Normally this is not desired or recommended since it may be slower or, if many people did it, all the time, cause Eclipse Infrastructure to slow down for everyone and overuse bandwidth. But, it can be used occasionally for tests if you suspect a problem with mirroring (for example, if it appears you are getting old or stale content you might want to try by-passing mirrors temporarily, to test that.

Moving a repo to archive.eclipse.org

Another situation where redirected downloads are desirable is when a repo is moved
from download.eclipse.org to archive.eclipse.org. For files references using the
Find a Mirror script (http://www.eclipse.org/downloads/download.php?file=/path/to/a/file.zip)
moving to archive is transparently handled by that script. This section is exploring how the same effect can be achieved for p2 repositories.

The obvious way might be through the use of p2.mirrorsURL, however, I couldn't get this to work -- Stephan.

The following strategy seems to do the job (although admittedly, not a candidate for beauty contests):

when moving everything to archive.eclipse.org keep a copy of all metadata (artifacts.jar and all those) on download.eclipse.org.

to be on the safe side increment the p2.timestamp value in the top list of properties

After editing put the .xml files back into their respective jar.

An alternative, cleaner strategy for leaf repositories has been discussed in bug 349141#c3:

redirect using a composite repository as a proxy:

move repository as-is to archive (maybe should remove p2.mirrorsURL at this point? No need to remove p2.mirrorsURL. When queried, if no longer mirrored, it returns a list of zero length and p2 then knows to use originally requested location URL.)

create new composite repository pointing to archive.eclipse.org/...

Interesting bugs or pages with tools attached

Many of these have not been used/updated for a while, so they are listed here for information only. Its hoped you can find the tools useful (or help refine them) but use at your own risk ... and on non-production test systems, at least initially.