On Sun, Oct 17, 2010 at 12:14 PM, James Tucker <jftucker at gmail.com> wrote:
> What exactly is it that you think is so wrong? Maybe if you can explain what the actual specific problem is, I can address it.
I'm kinda vague on what is wrong as well.
If we are talking about regressions for certain scenarios (e.g.
feature 'a' breaks when upgrading rubygems from version x to version y
on platform/environment/interpreter z), then again I would say that
continuous integration is the proper way to mitigate those risks.
Whatever that scenario is, have an integration test which sets up that
environment, performs the scenario, and asserts that it does not fail.
For interpreter-specific issues, RVM makes this much easier than it
used to be (and we've always had multi-ruby).
This is complex and not easy to do, but it is possible, especially in
an EC2 environment with a distributed-build CI tool such as
Hudson/Teamcity where you can completely automate the spin-up of boxes
for specific distro/environment (including windows).
Even better, you can make this type of CI environment easily available
to anyone who wishes to hack on Rubygems (e.g. via pre-packaged AMIs),
and make green tests across the board a condition of patch acceptance.
Is it unattainable pie in the sky dreaming to have this level of
automated regression testing? In the short term, yes.
But, we can start thinking along these lines - such as creating a
official checklist of "things which must not regress for a non-major
release", and having concerned volunteers (such as the ones voicing
concern on this thread) manually test them and sign off before a
release.
So, yes, we need to be careful not to break "production" systems, but
the way to achieve that goal is to have more rigorous regression
testing for important scenarios. Criticizing the current developer
leads (de-facto or otherwise) for past breakages is less productive
way to approach the problem which does not address the root cause.
Taking proactive actions, even if it is just documenting a manual
pre-release checklist, is a more productive way to approach the
problem.
-- Chad