From harmony-dev-return-9626-apmail-incubator-harmony-dev-archive=incubator.apache.org@incubator.apache.org Mon Jul 03 15:17:39 2006
Return-Path:
Delivered-To: apmail-incubator-harmony-dev-archive@www.apache.org
Received: (qmail 58152 invoked from network); 3 Jul 2006 15:17:38 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (209.237.227.199)
by minotaur.apache.org with SMTP; 3 Jul 2006 15:17:38 -0000
Received: (qmail 77778 invoked by uid 500); 3 Jul 2006 15:17:36 -0000
Delivered-To: apmail-incubator-harmony-dev-archive@incubator.apache.org
Received: (qmail 77729 invoked by uid 500); 3 Jul 2006 15:17:36 -0000
Mailing-List: contact harmony-dev-help@incubator.apache.org; run by ezmlm
Precedence: bulk
List-Help:
List-Unsubscribe:
List-Post:
List-Id:
Reply-To: harmony-dev@incubator.apache.org
Delivered-To: mailing list harmony-dev@incubator.apache.org
Received: (qmail 77717 invoked by uid 99); 3 Jul 2006 15:17:35 -0000
Received: from asf.osuosl.org (HELO asf.osuosl.org) (140.211.166.49)
by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 03 Jul 2006 08:17:35 -0700
X-ASF-Spam-Status: No, hits=1.4 required=10.0
tests=SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (asf.osuosl.org: 64.74.244.71 is neither permitted nor denied by domain of geir@pobox.com)
Received: from [64.74.244.71] (HELO chi.mobile-health-diary.com) (64.74.244.71)
by apache.org (qpsmtpd/0.29) with SMTP; Mon, 03 Jul 2006 08:17:34 -0700
Received: (qmail 15742 invoked from network); 3 Jul 2006 15:17:10 -0000
Received: from ool-43560edb.dyn.optonline.net (HELO ?192.168.1.106?) (geir@67.86.14.219)
by b014.internal.mobile-health-diary.com with SMTP; 3 Jul 2006 15:17:10 -0000
Message-ID: <44A934F6.7070801@pobox.com>
Date: Mon, 03 Jul 2006 11:17:10 -0400
From: Geir Magnusson Jr
Reply-To: geir@pobox.com
User-Agent: Thunderbird 1.5.0.4 (Windows/20060516)
MIME-Version: 1.0
To: harmony-dev@incubator.apache.org
Subject: Re: [classlib][testing] excluding the failed tests
References: <005401c69be3$a571fbf0$0b01a8c0@LITTLEGUY> <44A8F1AA.5090500@gmail.com>
In-Reply-To: <44A8F1AA.5090500@gmail.com>
X-Enigmail-Version: 0.94.0.0
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit
X-Virus-Checked: Checked by ClamAV on apache.org
X-Spam-Rating: minotaur.apache.org 1.6.2 0/1000/N
Is this the case where we have two 'categories'?
1) tests that never worked
2) tests that recently broke
I think that a #2 should never persist for more than one build
iteration, as either things get fixed or backed out. I suppose then we
are really talking about category #1, and that we don't have the "broken
window" problem as we never had the window there in the first place?
I think it's important to understand this (if it's actually true).
geir
Tim Ellison wrote:
> Nathan Beyer wrote:
>> How are other projects handling this? My opinion is that tests, which are
>> expected and know to pass should always be running and if they fail and the
>> failure can be independently recreated, then it's something to be posted on
>> the list, if trivial (typo in build file?), or logged as a JIRA issue.
>
> Agreed, the tests we have enabled are run on each build (hourly if
> things are being committed), and failures are sent to commit list.
>
>> If it's broken for a significant amount of time (weeks, months), then rather
>> than excluding the test, I would propose moving it to a "broken" or
>> "possibly invalid" source folder that's out of the test path. If it doesn't
>> already have JIRA issue, then one should be created.
>
> Yes, though I'd be inclined to move it sooner -- tests should not stay
> broken for more than a couple of days.
>
> Recently our breakages have been invalid tests rather than broken
> implementation, but they still need to be investigated/resolved.
>
>> I've been living with consistently failing tests for a long time now.
>> Recently it was the unstable Socket tests, but I've been seeing the WinXP
>> long file name [1] test failing for months.
>
> IMHO you should be shouting about it! The alternative is that we
> tolerate a few broken windows and overall quality slips.
>
>> I think we may be unnecessarily complicating some of this by assuming that
>> all of the donated tests that are currently excluded and failing are
>> completely valid. I believe that the currently excluded tests are either
>> failing because they aren't isolated according to the suggested test layout
>> or they are invalid test; I suspect that HARMONY-619 [1] is a case of the
>> later.
>>
>> So I go back to my original suggestion, implement the testing proposal, then
>> fix/move any excluded tests to where they work properly or determine that
>> they are invalid and delete them.
>
> Yes, the tests do need improvements too.
>
> Regards,
> Tim
>
>
>> [1] https://issues.apache.org/jira/browse/HARMONY-619
>>
>
>
---------------------------------------------------------------------
Terms of use : http://incubator.apache.org/harmony/mailing.html
To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
For additional commands, e-mail: harmony-dev-help@incubator.apache.org