From harmony-dev-return-10870-apmail-incubator-harmony-dev-archive=incubator.apache.org@incubator.apache.org Tue Jul 25 12:48:11 2006
Return-Path:
Delivered-To: apmail-incubator-harmony-dev-archive@www.apache.org
Received: (qmail 80769 invoked from network); 25 Jul 2006 12:48:11 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (209.237.227.199)
by minotaur.apache.org with SMTP; 25 Jul 2006 12:48:11 -0000
Received: (qmail 77626 invoked by uid 500); 25 Jul 2006 12:48:06 -0000
Delivered-To: apmail-incubator-harmony-dev-archive@incubator.apache.org
Received: (qmail 76906 invoked by uid 500); 25 Jul 2006 12:48:05 -0000
Mailing-List: contact harmony-dev-help@incubator.apache.org; run by ezmlm
Precedence: bulk
List-Help:
List-Unsubscribe:
List-Post:
List-Id:
Reply-To: harmony-dev@incubator.apache.org
Delivered-To: mailing list harmony-dev@incubator.apache.org
Received: (qmail 76895 invoked by uid 99); 25 Jul 2006 12:48:05 -0000
Received: from asf.osuosl.org (HELO asf.osuosl.org) (140.211.166.49)
by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 25 Jul 2006 05:48:05 -0700
X-ASF-Spam-Status: No, hits=0.5 required=10.0
tests=DNS_FROM_RFC_ABUSE,HTML_MESSAGE,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (asf.osuosl.org: domain of stepan.mishura@gmail.com designates 64.233.184.234 as permitted sender)
Received: from [64.233.184.234] (HELO wr-out-0506.google.com) (64.233.184.234)
by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 25 Jul 2006 05:48:04 -0700
Received: by wr-out-0506.google.com with SMTP id i34so811651wra
for ; Tue, 25 Jul 2006 05:47:43 -0700 (PDT)
DomainKey-Signature: a=rsa-sha1; q=dns; c=nofws;
s=beta; d=gmail.com;
h=received:message-id:date:from:to:subject:in-reply-to:mime-version:content-type:references;
b=fwiSzBKuKA0AHMoFxl3S4MwX1tYuTtZzxYBxRgOWDO2v+JcBPkCCvTpAjOE6NUAbmm4W7oUPqDnXnJLYnVTrddBRFatfILxW5vuJIkxAl9YPGzjsTGMa0keA+hSBpKpTkrHZFoe5uoPB9WXM1OV0kuf52HkqimYg3KEqVy+enWQ=
Received: by 10.65.219.7 with SMTP id w7mr5130174qbq;
Tue, 25 Jul 2006 05:47:43 -0700 (PDT)
Received: by 10.65.186.7 with HTTP; Tue, 25 Jul 2006 05:47:42 -0700 (PDT)
Message-ID: <6e47b64f0607250547t674b7f25oa52768b4b8281400@mail.gmail.com>
Date: Tue, 25 Jul 2006 19:47:42 +0700
From: "Stepan Mishura"
To: harmony-dev@incubator.apache.org
Subject: Re: [classlib] Testing conventions - a proposal
In-Reply-To: <44BF49C0.1020502@googlemail.com>
MIME-Version: 1.0
Content-Type: multipart/alternative;
boundary="----=_Part_91912_16138296.1153831662942"
References: <44ABB451.30806@googlemail.com> <44B75E41.7020901@googlemail.com>
<44BDCE12.4050303@gmail.com> <44BE0541.1050507@googlemail.com>
<44BEF3A0.7000901@gmail.com> <44BF49C0.1020502@googlemail.com>
X-Virus-Checked: Checked by ClamAV on apache.org
X-Spam-Rating: minotaur.apache.org 1.6.2 0/1000/N
------=_Part_91912_16138296.1153831662942
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
Content-Disposition: inline
On 7/20/06, George Harley wrote:
>
>
> Anyway, the point I guess that I am trying to make here is that it is
> possible in TestNG to select the methods to test dynamically using a
> little bit of scripting that (a) gives us a lot more power than the
> include/exclude technique and (b) will work the same across every
> platform we test on. Because BeanShell allows us to instantiate and use
> Java objects of any type on the classpath then the possibility of using
> more than just group membership to decide on tests to run becomes
> available to us. Please refer to the TestNG documentation for more on
> the capabilities of BeanShell and the TestNG API. I had never heard of
> it before never mind used it but still managed to get stuff working in a
> relatively short space of time.
>
> I hope this helps. Maybe I need to write a page on the wiki or something ?
Hi George,
It would be great to have your proposal for using TestNG on web-site like we
have for "Testing conventions" [1].
Thanks,
Stepan.
[1]
http://incubator.apache.org/harmony/subcomponents/classlibrary/testing.html
Best regards,
> George
>
>
>
> >> Best regards,
> >> George
> >>
> >>
> >>
> >>>> Thanks for reading this far.
> >>>>
> >>>> Best regards,
> >>>> George
> >>>>
> >>>>
> >>>>
> >>>> George Harley wrote:
> >>>>> Hi,
> >>>>>
> >>>>> Just seen Tim's note on test support classes and it really caught
> >>>>> my attention as I have been mulling over this issue for a little
> >>>>> while now. I think that it is a good time for us to return to the
> >>>>> topic of class library test layouts.
> >>>>>
> >>>>> The current proposal [1] sets out to segment our different types
> >>>>> of test by placing them in different file locations. After looking
> >>>>> at the recent changes to the LUNI module tests (where the layout
> >>>>> guidelines were applied) I have a real concern that there are
> >>>>> serious problems with this approach. We have started down a track
> >>>>> of just continually growing the number of test source folders as
> >>>>> new categories of test are identified and IMHO that is going to
> >>>>> bring complexity and maintenance issues with these tests.
> >>>>>
> >>>>> Consider the dimensions of tests that we have ...
> >>>>>
> >>>>> API
> >>>>> Harmony-specific
> >>>>> Platform-specific
> >>>>> Run on classpath
> >>>>> Run on bootclasspath
> >>>>> Behaves different between Harmony and RI
> >>>>> Stress
> >>>>> ...and so on...
> >>>>>
> >>>>>
> >>>>> If you weigh up all of the different possible permutations and
> >>>>> then consider that the above list is highly likely to be extended
> >>>>> as things progress it is obvious that we are eventually heading
> >>>>> for large amounts of related test code scattered or possibly
> >>>>> duplicated across numerous "hard wired" source directories. How
> >>>>> maintainable is that going to be ?
> >>>>>
> >>>>> If we want to run different tests in different configurations then
> >>>>> IMHO we need to be thinking a whole lot smarter. We need to be
> >>>>> thinking about keeping tests for specific areas of functionality
> >>>>> together (thus easing maintenance); we need something quick and
> >>>>> simple to re-configure if necessary (pushing whole directories of
> >>>>> files around the place does not seem a particularly lightweight
> >>>>> approach); and something that is not going to potentially mess up
> >>>>> contributed patches when the file they patch is found to have been
> >>>>> recently pushed from source folder A to B.
> >>>>>
> >>>>> To connect into another recent thread, there have been some posts
> >>>>> lately about handling some test methods that fail on Harmony and
> >>>>> have meant that entire test case classes have been excluded from
> >>>>> our test runs. I have also been noticing some API test methods
> >>>>> that pass fine on Harmony but fail when run against the RI. Are
> >>>>> the different behaviours down to errors in the Harmony
> >>>>> implementation ? An error in the RI implementation ? A bug in the
> >>>>> RI Javadoc ? Only after some investigation has been carried out do
> >>>>> we know for sure. That takes time. What do we do with the test
> >>>>> methods in the meantime ? Do we push them round the file system
> >>>>> into yet another new source folder ? IMHO we need a testing
> >>>>> strategy that enables such "problem" methods to be tracked easily
> >>>>> without disruption to the rest of the other tests.
> >>>>>
> >>>>> A couple of weeks ago I mentioned that the TestNG framework [2]
> >>>>> seemed like a reasonably good way of allowing us to both group
> >>>>> together different kinds of tests and permit the exclusion of
> >>>>> individual tests/groups of tests [3]. I would like to strongly
> >>>>> propose that we consider using TestNG as a means of providing the
> >>>>> different test configurations required by Harmony. Using a
> >>>>> combination of annotations and XML to capture the kinds of
> >>>>> sophisticated test configurations that people need, and that
> >>>>> allows us to specify down to the individual method, has got to be
> >>>>> more scalable and flexible than where we are headed now.
> >>>>>
> >>>>> Thanks for reading this far.
> >>>>>
> >>>>> Best regards,
> >>>>> George
> >>>>>
> >>>>>
> >>>>> [1]
> >>>>>
> http://incubator.apache.org/harmony/subcomponents/classlibrary/testing.html
> >>>>>
> >>>>> [2] http://testng.org
> >>>>> [3]
> >>>>>
> http://mail-archives.apache.org/mod_mbox/incubator-harmony-dev/200606.mbox/%3c44A163B3.6080005@googlemail.com%3e
> >>>>>
>
>
--
Thanks,
Stepan Mishura
Intel Middleware Products Division
------------------------------------------------------
Terms of use : http://incubator.apache.org/harmony/mailing.html
To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
For additional commands, e-mail: harmony-dev-help@incubator.apache.org
------=_Part_91912_16138296.1153831662942--