Suppose you have automation tests (my example uses Selenium) to assert complex business logic for a web site. What is the best way to assert that thousands of different test cases are outputting the correct values in the software? I thought about just copying the business logic that the actual software uses for the values we're expecting, but then wouldn't that just be re-inventing the wheel?

Example:

The goal is to assert that shipping is correct for all order scenarios. eCommerce site has 50,000 products (let's suppose). The shipping costs are determined based on the shipping provider and rates for each client, extra handling fees, base shipping rates whether the user is a wholesaler, retailer, might factor in product quantity thresholds depending on types of items ordered, etc..

Should we move this business logic (from our site server code) to our assert business logic in our Selenium code library when the automation test cases run for all products? Or should we hard code the shipping cost for thousands of products given the parameters of those test cases?

Let me rephrase the question...

All of the test cases I've done in the past have been making sure site features don't break when enhancements are made to our software (ie: all pages in a workflow work and don't crash--like placing an order, create a product given a set of values and making sure the product has the same values when editing the product, redirects work properly, etc..). Those test cases are simple to write. However, with what I'm asking in this question, I'd like to assert/validate that shipping costs (essentially output values based on a set of parameters) are correct based on our business logic. What is the best methodology/approach for this validity testing of business logic? Do I simply move the business logic from our server code to our Selenium class library code to make sure shipping is correct for all products? Or should we have a spreadsheet/matrix of what our parameters are and the expected values and hard code in our Selenium test cases? I expect the business logic to get more complex in the future, but we need to make sure the shipping costs are coming out the same for products we don't expect the business logic to change in the future.

Pricing is more complex than shipping (with discounts, etc..) so I'd like to use the same methodology for that as well.

Sample the behavior for specific products. If you know the business logic but do not want to address the combinatorial problem, you can select a set of products to test that you think are especially important. Your selection criteria might be based on their financial importance, or on their similarity to other products, or some other business-relevant metric. I personally do not like this approach, but I have seen organizations use it.

Regression-test the behavior for specific products. If you do not know the business logic, but you have reasons to believe that the current results are correct for a set of products, you can capture those results and compare them against the results of future releases.

If possible, I recommend performing these tests at the API level rather than the UI level.

Finally, if the business logic is truly complex, you should investigate whether it can be tested at a lower level, i.e. a component at a time. (I will not dwell on whether that would be considered unit testing, or whether such testing is the province of developers or testers. If it makes sense to test at a lower level, your team will need to figure that out for themselves.)

One of the first places to look is what level of coverage of the business logic exists in unit tests. If the unit tests are covering your boundary logic, fee logic and so forth, then you don't need to do more than run tests and ensure that the end data is correct.

For the case where you're looking at correct end data, I'd recommend a harness that will process a test step independently, can string them all together, and pull from a data store to generate multiple transactions. Your data could be something like:

When all the transactions run, you pull the data from the system database, and run a cell-by-cell comparison. Since while you're building your transaction set you're also building your baseline, the only changes that need to happen after this are adding new transactions and their baseline data - I use this method (with a different tool) for a specialized point of sale system and linked ecommerce site where correct calculation of shipping amounts, taxes, customer discounts and the like is essential. It works. Any time a refactor breaks calculations, my team knows about it.

When you say baselines, are you simply saying take past shipping costs from orders placed in a live database (that we know is correct for the product combinations in each order), sync that data to our QA database as baselines and compare those shipping costs against the shipping costs of what our dev site outputs? And in the future, if we add new logic to shipping, make new baselines in our QA database?
–
MacGyverJan 25 '12 at 17:05

2

I'm saying a store of known good costs from orders - whether they're live or not doesn't matter, so long as you know they're correct and they sample the application's behavior. If the logic changes but the costs don't you don't need to change anything - your tests will check that the new logic still gets the same result. If the costs change, you update your baselines so your tests stay correct. So if Order 367 should have shipping of $5.78 then that's what your baseline says.
–
Kate PaulkJan 25 '12 at 19:45

thanks for your input. I gave you some up-votes. ;-)
–
MacGyverJan 25 '12 at 20:02