create test harness and port TestApp to junit

Details

Description

To both encourage good internal development, and to make it easy for plugin developers to write unit tests of their own code I think we need a harness that makes it easy to unit test updates and queries against Solr (without needing a servlet container)

Once we have this, i think we can/should also retire "TestApp" in favor of some JUnit tests (which would probably make more sense for other developers)

Iv'e already started on this, i thought i'd have something to commit tonight, but i got distracted ... filing this bug as a tracker for now.

Hoss Man
added a comment - 16/Apr/08 00:56 This bug was modified as part of a bulk update using the criteria...
Marked ("Resolved" or "Closed") and "Fixed"
Had no "Fix Version" versions
Was listed in the CHANGES.txt for 1.1
The Fix Version for all 38 issues found was set to 1.1, email notification
was suppressed to prevent excessive email.
For a list of all the issues modified, search jira comments for this
(hopefully) unique string: 20080415hossman3

In the interest of reducing possible confusion (and since i haven't been motivated to refactor SolrTest to use SolrTestHarness in the last 6 months) i think maybe I should just svn remove solr/src/apps and the build.xml references to it before we have an official release.

Hoss Man
added a comment - 29/Nov/06 08:33 In the interest of reducing possible confusion (and since i haven't been motivated to refactor SolrTest to use SolrTestHarness in the last 6 months) i think maybe I should just svn remove solr/src/apps and the build.xml references to it before we have an official release.
we can always restore it in the future if there is a desire.
objections?

Well, the one thing SolrTest could do that we don't have a replacement for that SolrTest was usefull for is load testing SolrCore – it might make sense to leave that part in, but rip out the guts and replace it with using the SolrTestHarness.

Hoss Man
added a comment - 21/Apr/06 14:38 Well, the one thing SolrTest could do that we don't have a replacement for that SolrTest was usefull for is load testing SolrCore – it might make sense to leave that part in, but rip out the guts and replace it with using the SolrTestHarness.

Hoss Man
added a comment - 19/Apr/06 14:16 Everything I set out to do is done. Should this issue be resolved, or should SolrTest be removed from the repository?
Does it serve any useful purpose now that we have the JUnit tests?

leaving issue open for now, pending sub-task to deal with solrconfig.xml loading, and a decission about wether we want to completely remove SolrTest or not. (either way, we should copy/move the config/schema used by the tests somewhere under the test directory, and change the working directory of the JUnit tests)

Hoss Man
added a comment - 11/Apr/06 13:58 filehandle leak was resolved by closing requests, code was commited.
leaving issue open for now, pending sub-task to deal with solrconfig.xml loading, and a decission about wether we want to completely remove SolrTest or not. (either way, we should copy/move the config/schema used by the tests somewhere under the test directory, and change the working directory of the JUnit tests)

zip file contains a bunch of code and a patch file that modifies the build.xml and XML.java

the new code consists of:

the TestHarness from before (with a few additions) which should be usefull for interacting with a SolrCore independent of any specific testing framework

a new AbstractSolrTestCase that is designed to make it easy to write JUnit tests in particular.

a Sample test, and a short test of the basic Solr functions that demonstrate "best practices" of the AbstractSolrTestCase

a ConvertedLegacyTest which is a progromatic translation of src/apps/SolrTest/newtest.txt (the converstion perl script is included in the zip for refrence, but i wasn't planning on commiting it.)

this represents 95% of what i think we need as far as a good testing framework moving forward: the last 5% being modifications to SolrConfig so that individual tests can use different solrconfig.xml files. Even without that, this code can be commited as is – tests will just have to share a solrconfig.xml for the time being, but they can use unique schema files.

The only hitch with all of this, is that i seem to have a filehandle leak somewhere. Or at least i think i do ... running "ant legacyTest" works fine with an openfiles limit of 1024, but "ant junit" runs out of filehandles using the same limit (at 2048 the test runs fine). The index itself only contains about 100 files when this happens so i'm not sure what's going on.

If anyone can help me spot the problem, i'd appreciate it.

I'm going to take a look at it with some fresh eyes (and lsof) in the morning.

Hoss Man
added a comment - 09/Apr/06 15:47 zip file contains a bunch of code and a patch file that modifies the build.xml and XML.java
the new code consists of:
the TestHarness from before (with a few additions) which should be usefull for interacting with a SolrCore independent of any specific testing framework
a new AbstractSolrTestCase that is designed to make it easy to write JUnit tests in particular.
a Sample test, and a short test of the basic Solr functions that demonstrate "best practices" of the AbstractSolrTestCase
a ConvertedLegacyTest which is a progromatic translation of src/apps/SolrTest/newtest.txt (the converstion perl script is included in the zip for refrence, but i wasn't planning on commiting it.)
this represents 95% of what i think we need as far as a good testing framework moving forward: the last 5% being modifications to SolrConfig so that individual tests can use different solrconfig.xml files. Even without that, this code can be commited as is – tests will just have to share a solrconfig.xml for the time being, but they can use unique schema files.
The only hitch with all of this, is that i seem to have a filehandle leak somewhere. Or at least i think i do ... running "ant legacyTest" works fine with an openfiles limit of 1024, but "ant junit" runs out of filehandles using the same limit (at 2048 the test runs fine). The index itself only contains about 100 files when this happens so i'm not sure what's going on.
If anyone can help me spot the problem, i'd appreciate it.
I'm going to take a look at it with some fresh eyes (and lsof) in the morning.

>There are probably some opportunities to use Java5 stuff like varargs...
> doc("id",42,"subject","easy")

...i'm doing that in validateAddDoc (which calls makeSimpleDoc) ... are there other methods you think it also makes sense for?

> Also what we really need is a way to dynamically add/create a schema, so one could add
> a new analysis filter,
> then define a fieldtype & field that uses it and test it out, without modifying some global
> testing schema.

i was operating under the assumption that for stuff like that, the best thing to do would be to have multiple sets of solrconfig/schema files ... for Solr itself there would probably be one big generic file that most tests could use, but any test that wanted to try something exotic would check in a new one (in a directory with the same name as the test class) and speciy it when constructing hte harness.

the only flaw in that plan right now is that SolrCore only lets you specify a schema file path when you construct it, not a solrconfig, but i was going to tackle refactoring that once i had the basics up and running.

Hoss Man
added a comment - 14/Mar/06 02:34 >There are probably some opportunities to use Java5 stuff like varargs...
> doc("id",42,"subject","easy")
...i'm doing that in validateAddDoc (which calls makeSimpleDoc) ... are there other methods you think it also makes sense for?
> Also what we really need is a way to dynamically add/create a schema, so one could add
> a new analysis filter,
> then define a fieldtype & field that uses it and test it out, without modifying some global
> testing schema.
i was operating under the assumption that for stuff like that, the best thing to do would be to have multiple sets of solrconfig/schema files ... for Solr itself there would probably be one big generic file that most tests could use, but any test that wanted to try something exotic would check in a new one (in a directory with the same name as the test class) and speciy it when constructing hte harness.
the only flaw in that plan right now is that SolrCore only lets you specify a schema file path when you construct it, not a solrconfig, but i was going to tackle refactoring that once i had the basics up and running.

There are probably some opportunities to use Java5 stuff like varargs...
doc("id",42,"subject","easy")

Also what we really need is a way to dynamically add/create a schema, so one could add a new analysis filter,
then define a fieldtype & field that uses it and test it out, without modifying some global testing schema.

Yonik Seeley
added a comment - 14/Mar/06 01:22 Looking good Hoss!
There are probably some opportunities to use Java5 stuff like varargs...
doc("id",42,"subject","easy")
Also what we really need is a way to dynamically add/create a schema, so one could add a new analysis filter,
then define a fieldtype & field that uses it and test it out, without modifying some global testing schema.

Here's what i've got so far, still a work in progress – but you can see the basic idea.

I was trying to avoid putting anything junit specific in the TestHarness, but it definitely looks like having an abstract subclass of TestCase that provides some assert methods that can take in adds or requests/xpaths and validate directly would make hte tests easier to read.

Hoss Man
added a comment - 13/Mar/06 15:37 Here's what i've got so far, still a work in progress – but you can see the basic idea.
I was trying to avoid putting anything junit specific in the TestHarness, but it definitely looks like having an abstract subclass of TestCase that provides some assert methods that can take in adds or requests/xpaths and validate directly would make hte tests easier to read.
Any comments would be appreacited.