On 03/23/2011 08:52 AM, Pete Muir wrote:
> I think you should define more carefully what you are proposing, then people might agree more...
>> * if you write a regression test, it must have a linked JIRA OR it must reference a forum post/mailing list archive (you might have written a test to prove that there isn't a problem to a user, but silly to "waste" that test...)
> * if you write a test for a new feature, and the feature has a JIRA issue, then you should link it. New feature development doesn't always work well with an issue tracker IMO.
>> You also need to think about linkage. Each test should have a comment
>> /**
> * Testing blah...
> *
> * JIRA: JBAS-1234
> */
>> and all bugs should also give the FQMN of the associated tests - this way you can go from one to the other very easily.
>> We've used this scheme a lot with Weld and the CDI TCK and I find it works very well, especially 9 months later when you come back to an issue and can't remember what I did when.
What Pete has done here is outline what I'm talking about much more
succinctly.
S,
ALR
>>> On 23 Mar 2011, at 06:10, Andrew Lee Rubinger wrote:
>>> I'm still confused by the general hesitation towards a JIRA association
>> for each test. To me this is almost a redundant policy; since when are
>> we committing *anything* without linking back to an issue?
>>>> Comments inline.
>>>> On 03/22/2011 12:11 PM, Shelly McGowan wrote:
>>> I realize the intent of @IntegrationTest annotation for the CDI TCK. It was just an example in response to the comments about:
>>>>>> Jason: IMO a better way to organize it would be by functional area, with generous descriptions. All spec tests should reference spec
>>> sections, for example.
>>>>>> ALR: I'm a fan of organizing by functional areas and spec references.
>>>>>> therefore proposing to use annotations for such organization instead of JIRA. We can devise our own annotations that make sense.
>>>> OK, what tooling do you envision would make use of this metadata? Or is
>> it just there as a documentation marker for us?
>>>> To me, annotations are to be consumed/processed/enforced by software;
>> comments are to be read by developers.
>>>>> Another consideration is because they are integration tests, it is common that more than one
>>> technology is used in the test cases. Having tests organized by directory structure does not always reflect that.
>>>>>> integration/src/test/java/org/jboss/as/testsuite/integration/jms/JMSTestCase.java
>>>>>> These tests could contain test cases that may also cover MDB, XA, JCA. So as in [3] below, we could do something like:
>>>>>> @Test(groups = { "JMS", "XA", "MDB"})
>>> public void testJms() throws Exception {}
>>>>>> or
>>>>>> @Test(groups = { "web_profile" } )
>>> public void testEJBLite throws Exception {}
>>>> What does @Test.groups represent? Is that a proposal for our own
>> extension? JUnit doesn't have this property in @org.junit.Test, so if
>> it's our own thing we should likely rename to something else...
>>>>http://kentbeck.github.com/junit/javadoc/latest/org/junit/Test.html>>>> S,
>> ALR
>>>>>>>> Shelly McGowan
>>> JBoss, by Red Hat
>>>>>> ----- Original Message -----
>>> From: "Aslak Knutsen"<aknutsen at redhat.com>
>>> To: "Andrew Lee Rubinger"<andrew.rubinger at redhat.com>
>>> Cc: jboss-as7-dev at lists.jboss.org>>> Sent: Tuesday, March 22, 2011 10:38:08 AM
>>> Subject: Re: [jboss-as7-dev] Requirements and Design Proposal: AS7 TestSuite
>>>>>>>>>>> Why @IntegrationTest? By simply being in the AS Integration Test
>>>> module, wouldn't that be self-explanatory?
>>>>>>>>>> The use of @IntegrationTest in CDI TCK is basically to separate tests that can run in Weld SE (mocked EE v) vs the ones that need a full EE container.
>>>>>> Not sure the AS Suite needs that kind of separation.
>>>>>> -aslak-
>>>>>>> On 03/20/2011 10:58 AM, Shelly McGowan wrote:
>>>>>>>>>>>>>>>>>>>> Why don't we consider what was done for the CDI and BV TCKs.
>>>>> Annotations[1] are used to define the test type.
>>>>> If it is a spec assertion, the spec reference is clearly identified:
>>>>>>>>>> @SpecAssertion(section = "2.1.1", id = "c")
>>>>>>>>>> or you could annotate the Test with @IntegrationTest which could be
>>>>> used to denote that AS-specific test.
>>>>>>>>>> Test (not code) coverage reports can be generated to identify what
>>>>> the coverage is for each component. The report
>>>>> provides links to fisheye and the test code in svn.
>>>>>>>>>>>>>>> [1]http://community.jboss.org/wiki/BeanValidationTCK#What_do_all_the_annotations_mean>>>>> Other references:
>>>>> [2]http://docs.jboss.org/hibernate/stable/beanvalidation/tck/reference/html_single/#d0e594>>>>> [3]http://anonsvn.jboss.org/repos/weld/cdi-tck/trunk/impl/src/main/java/org/jboss/jsr299/tck/tests/context/conversation/ClientConversationContextTest.java>>>>>>>>>>>>>>> Shelly McGowan
>>>>> JBoss, by Red Hat
>>>>>>>>>> ----- Original Message -----
>>>>> From: "Andrew Lee Rubinger"<andrew.rubinger at redhat.com>
>>>>> To: jboss-as7-dev at lists.jboss.org>>>>> Sent: Saturday, March 19, 2011 5:34:30 PM
>>>>> Subject: Re: [jboss-as7-dev] Requirements and Design Proposal: AS7
>>>>> TestSuite
>>>>>>>>>> It occurs to me I might not have been clear about the JIRA-per-test
>>>>> thing.
>>>>>>>>>> Say you're working on Feature X. You might commit the feature and
>>>>> the
>>>>> test alongside one another, citing the same JIRA, with the commit
>>>>> message:
>>>>>>>>>> "[JBAS-XXX] Implement Feature X and tests".
>>>>>>>>>> That's great. No separate issues needed for the feature and the test
>>>>> each.
>>>>>>>>>> What's no good is going in and adding a bunch of tests for EJB, Web,
>>>>> etc
>>>>> while we pump up our coverage, alongside a commit message:
>>>>>>>>>> "[JBAS-XXX] Add spec coverage to the testsuite"
>>>>>>>>>> ...that tells us nothing about each test being made, just that we
>>>>> dumped
>>>>> 'em in there. Which is obvious, because they're there.
>>>>>>>>>> S,
>>>>> ALR
>>>>>>>>>> On 03/19/2011 05:25 PM, Andrew Lee Rubinger wrote:
>>>>>> Inline.
>>>>>>>>>>>> On 03/19/2011 03:19 PM, Jason Greene wrote:
>>>>>>> I really like the idea of our primary test suite covering API
>>>>>>> interaction and your typical unit test which is tightly coupled to
>>>>>>> internals. I also like your idea to separate spec Apis from JBoss
>>>>>>> aAPIs.
>>>>>>>>>>>>>> However, I disagree on the Jira per test requirement. This is only
>>>>>>> because I worked with a similar model (all jiras require a test),
>>>>>>> and you ended up with lots of duplicate tests, and you always had
>>>>>>> to read Jira to understand what was being tested. Also, the tests
>>>>>>> ended up being too specific, typically an exact replication of the
>>>>>>> reporter's workflow.
>>>>>>>>>>>> Consider this: If you had to read JIRA to understand what was being
>>>>>> tested, that's a clear violation of point 3): Document the test.
>>>>>> And
>>>>>> failing good documentation, with no JIRA to turn to, you're flying
>>>>>> blind.
>>>>>>>>>>>> Also, how exactly does registering a JIRA encourage test
>>>>>> duplication?
>>>>>> Whether the tests are in an issue tracker or not seems to be
>>>>>> unrelated
>>>>>> to folks writing duplicate tests.
>>>>>>>>>>>>>> IMO a better way to organize it would be by functional area, with
>>>>>>> generous descriptions. All spec tests should reference spec
>>>>>>> sections, for example. If we have a Jira that demonstrates lack of
>>>>>>> coverage, we should translate the problem into meaningful and
>>>>>>> reusable tests for the gaps we missed. Those tests should
>>>>>>> primarily refer to the API contract and not the Jira. Although
>>>>>>> commenting a list of jiras for extra info may be useful provided
>>>>>>> the Jira doesn't confuse what the correct behavior should be.
>>>>>>>>>>>> I'm a fan of organizing by functional areas and spec references.
>>>>>> We'd
>>>>>> done that in EJB3 with success: some packages were even named for
>>>>>> the
>>>>>> spec. Regardless, every commit had a JIRA association.
>>>>>>>>>>>> In the end, so long as it's perfectly clear what we're testing and
>>>>>> why,
>>>>>> I'll have no objections. But I do believe that JIRA is the obvious
>>>>>> answer to that question, and requires minimal effort to make a new
>>>>>> issue. Especially since every commit is isolated to one concern and
>>>>>> linked back to JIRA anyway, right? ;)
>>>>>>>>>>>> S,
>>>>>> ALR
>>>>>>>>>>>>>>>>>>>> Sent from my iPad
>>>>>>>>>>>>>> On Mar 18, 2011, at 2:21 AM, Andrew Lee
>>>>>>> Rubinger<andrew.rubinger at redhat.com> wrote:
>>>>>>>>>>>>>>> Looks like a lot of us have different ideas for what the AS7
>>>>>>>> Integration
>>>>>>>> TestSuite should consist of, so I'll kickoff with what I believe
>>>>>>>> is the
>>>>>>>> first design proposal towards getting coverage focused on the
>>>>>>>> end-user
>>>>>>>> (not certifying our own internals).
>>>>>>>>>>>>>>>> I suspect this breaks down into two categories, which may be
>>>>>>>> modelled as
>>>>>>>> separate modules under the existing "testsuite" aggregator
>>>>>>>> parent:
>>>>>>>>>>>>>>>> * Specification
>>>>>>>> * AS-specific APIs
>>>>>>>>>>>>>>>> This isn't difficult work, though I do think it's important we
>>>>>>>> consider
>>>>>>>> some hard rules. IMO we should be developing these suites as if
>>>>>>>> we were
>>>>>>>> application developers, not wearing our server dev hats.
>>>>>>>>>>>>>>>> ----------------
>>>>>>>>>>>>>>>> [End Goal]
>>>>>>>>>>>>>>>> 1) No compile-time dependencies in the module except for what's
>>>>>>>> absolutely necessary.
>>>>>>>>>>>>>>>> For the spec suite, this means: JDK and EE Spec APIs only in the
>>>>>>>> compilation classpath. Testable asset sources and resources (ie.
>>>>>>>> EJBs,
>>>>>>>> Servlets, etc) would live under src/main/* to enforce that. Only
>>>>>>>> the
>>>>>>>> tests themselves would be located under "src/test/*".
>>>>>>>>>>>>>>>> The AS-specific API suite may also add in our own APIs to the
>>>>>>>> compilation classpath, but the line should end there. In "test"
>>>>>>>> scope
>>>>>>>> we can place all runtime dependencies.
>>>>>>>>>>>>>>>> For the specification suite, AS-specific grammars like our own
>>>>>>>> deployment descriptors are fine; these are in many ways
>>>>>>>> equivalent to
>>>>>>>> the TCK porting layer. We're not building a TCK; we're showing
>>>>>>>> that our
>>>>>>>> implementation supports the features advertised.
>>>>>>>>>>>>>>>> 2) Every single new test created is to have an associated JIRA.
>>>>>>>>>>>>>>>> We all remember the nightmare it was when the old AS4-6 suite
>>>>>>>> would fall
>>>>>>>> down. We'd comb through each test, at times trying to determine
>>>>>>>> its
>>>>>>>> purpose. By linking to JIRA we get history of intent, which acts
>>>>>>>> as a
>>>>>>>> nice record even in the case that the test isn't so
>>>>>>>> well-documented.
>>>>>>>> I'd argue that tests are a bigger asset than our code, and we
>>>>>>>> should be
>>>>>>>> thinking about these in terms of long-term maintenance to outlive
>>>>>>>> any
>>>>>>>> specific impl.
>>>>>>>>>>>>>>>> 3) Documentation
>>>>>>>>>>>>>>>> Alongside the JIRA reference, a quick note about we're looking to
>>>>>>>> accomplish is something I find very helpful. I don't personally
>>>>>>>> buy the
>>>>>>>> argument that code is self-documenting if written well. It gets
>>>>>>>> refactored and stale over time.
>>>>>>>>>>>>>>>> 4) Run-mode profiles
>>>>>>>>>>>>>>>> Arquillian provides a wonderful abstraction such that we can get
>>>>>>>> coverage for AS in both remote managed *and* embedded modes
>>>>>>>> without
>>>>>>>> changing the test itself. To certify that everything is working
>>>>>>>> as
>>>>>>>> advertised no matter the runtime, we should be able to run the
>>>>>>>> same
>>>>>>>> suite in standalone, domain, and embedded modes (generally
>>>>>>>> speaking).
>>>>>>>>>>>>>>>> 5) Porting of AS6 Tests
>>>>>>>>>>>>>>>> There's no discounting the value this coverage has given us,
>>>>>>>> though I
>>>>>>>> question the purpose of a lot of these tests. I think a great
>>>>>>>> majority
>>>>>>>> of these need to come into the new codebase, refactored to align
>>>>>>>> if needed.
>>>>>>>>>>>>>>>> ----------------
>>>>>>>>>>>>>>>> [Current State]
>>>>>>>>>>>>>>>> Here[1] is an example of what I believe to be a simple,
>>>>>>>> well-written
>>>>>>>> test, with the exception that the tested Servlet and EJB are in
>>>>>>>> the same
>>>>>>>> test source folder.
>>>>>>>>>>>>>>>> The current "testsuite" aggregator contains modules which mix our
>>>>>>>> end-user certification stuff alongside internals, so I think
>>>>>>>> these
>>>>>>>> should be separated out.
>>>>>>>>>>>>>>>> A lot of this is set up in some fashion already, but I would like
>>>>>>>> to see us:
>>>>>>>>>>>>>>>> 1) Agree upon a strict scope for each type of testsuite along the
>>>>>>>> lines
>>>>>>>> of my points above, once we reach agreement
>>>>>>>> 2) Upgrade to ARQ 1.0.0.Alpha5 (which implies ShrinkWrap
>>>>>>>> 1.0.0-alpha-12), just released tonight. Currently AS is on a
>>>>>>>> forked
>>>>>>>> release of ARQ for OSGi purposes, and these changes, if
>>>>>>>> necessary, need
>>>>>>>> to get upstream so we can do upgrades.
>>>>>>>>>>>>>>>> It's clear that AS7 has made full-steam-ahead progress since last
>>>>>>>> summer, and with a little organization our testsuite can give us
>>>>>>>> a great
>>>>>>>> view of where we stand, from an end-user's perspective, with
>>>>>>>> minimal
>>>>>>>> investment.
>>>>>>>>>>>>>>>> S,
>>>>>>>> ALR
>>>>>>>>>>>>>>>>>>>>>>>> [1]
>>>>>>>>https://github.com/jbossas/jboss-as/blob/master/testsuite/integration/src/test/java/org/jboss/as/testsuite/integration/webejb/ServletInjectionTestCase.java>>>>>>>> _______________________________________________
>>>>>>>> jboss-as7-dev mailing list
>>>>>>>>jboss-as7-dev at lists.jboss.org>>>>>>>>https://lists.jboss.org/mailman/listinfo/jboss-as7-dev>>>>>>>>>>>>>> _______________________________________________
>>>>>>> jboss-as7-dev mailing list
>>>>>>>jboss-as7-dev at lists.jboss.org>>>>>>>https://lists.jboss.org/mailman/listinfo/jboss-as7-dev>>>>>> _______________________________________________
>>>>>> jboss-as7-dev mailing list
>>>>>>jboss-as7-dev at lists.jboss.org>>>>>>https://lists.jboss.org/mailman/listinfo/jboss-as7-dev>>>>> _______________________________________________
>>>>> jboss-as7-dev mailing list
>>>>>jboss-as7-dev at lists.jboss.org>>>>>https://lists.jboss.org/mailman/listinfo/jboss-as7-dev>>>> _______________________________________________
>>>> jboss-as7-dev mailing list
>>>>jboss-as7-dev at lists.jboss.org>>>>https://lists.jboss.org/mailman/listinfo/jboss-as7-dev>>> _______________________________________________
>>> jboss-as7-dev mailing list
>>>jboss-as7-dev at lists.jboss.org>>>https://lists.jboss.org/mailman/listinfo/jboss-as7-dev>> _______________________________________________
>> jboss-as7-dev mailing list
>>jboss-as7-dev at lists.jboss.org>>https://lists.jboss.org/mailman/listinfo/jboss-as7-dev>