Overview of A Structured Experiment of Test-Driven Development Jim Kile.

Similar presentations

Presentation on theme: "Overview of A Structured Experiment of Test-Driven Development Jim Kile."— Presentation transcript:

1
Overview of A Structured Experiment of Test-Driven Development Jim Kile

2
June 24, 2005DCS891C - Summer 2005 2 What is this paper about? Problem explored – To empirically analyze and quantify the effectiveness of Test-Driven Development (TDD) Why? – Unless objects are designed judiciously, dependency problems (e.g. tight coupling, fragile super classes, etc.) can creep into code affecting code quality – TDD proponents argue that reduced coupling occurs because the practice guides them to Build only those objects that are actually needed Improves code quality through continuous testing – There is no empirical evidence to support the claims made by proponents

3
June 24, 2005DCS891C - Summer 2005 3 What is this paper about? Hypotheses – The TDD practice will yield code with superior external code quality when compared with code developed in a waterfall-like practice – Programmers who practice TDD will develop code faster than programmers who develop code with a more traditional waterfall-like practice Results – TDD appears to yield code with superior external code quality Code produced in this manner passed 18% more functional black-box test cases – TDD programmers took more time (16%) than control group programmers – On average 80% of professional programmers thought TDD was effective and 78% believed productivity was improved – TDD facilitates simpler design and the lack of upfront design is not a hindrance

4
June 24, 2005DCS891C - Summer 2005 4 Technical background What is test-driven development? A practice of software development that involves the implementation of a system starting from the unit test cases of an object – If you cant write a test for what you are about to code, then you shouldnt even be thinking about coding [2][2] Perceived shortcomings – Lack of upfront design – Applicability of practice for code that is difficult to test using TDD (e.g. GUIs) – Reliance on refactoring to achieve code understanding and manage complexity – High skill level and experience requirements Perceived benefits – Program comprehension (especially useful for maintenance activities) – Efficiency through continuous feedback – Creation and availability of reusable test assets – Reducing defect injection during development

5
June 24, 2005DCS891C - Summer 2005 5 Context of this research Experiments were done with practitioners in their own working environment Sample size was relatively small Experiment was modified after the first trial All programmers worked in pairs – John Deere and RoleModel had used pair programming – Ericcson was introduced to pair programming The code generated was very small (200 LOC) Subjects had varying levels of experience

6
June 24, 2005DCS891C - Summer 2005 6 Research approach Experimentation (quantitative) – A set of three structured experiments conducted with 24 pair programmers One group developed using Test-Driven Development Another control group developed using a waterfall-like approach – Results monitored for black box test cases passed, time taken to complete, and percentage of code coverage Survey (qualitative) – A set of nine close-ended questions aimed at eliciting opinions about How productive the practice is for programmers? How effective is the practice? How difficult is the practice to adopt?

7
June 24, 2005DCS891C - Summer 2005 7 Related research problems (ideas) IDEA 1: Conduct a larger scale study along the same lines – Sample size was small in this study and though statistically there were correlations, one of the three experiments had to be thrown out IDEA 2: Further analyze if TDD can have a positive impact on productivity – This research proved there was a positive correlation between TDD and quality deliverables – Quality can yield future savings and productivity – Hypotheses: There is a learning curve when introducing TDD methods to a team. For small duration efforts, this learning curve will impact productivity. Longer efforts should see productivity equal to or superior to a waterfall baseline. – Note: This can be done in pair fashion as was done here or in non-pair fashion, though interestingly non-pair studies to date seem to have better results IDEA 3: Analyze enhancement projects where the original project followed TDD and there is a base of test cases for regression – Hypotheses: TDD provides a positive impact on development productivity when the initial project uses TDD and those regression cases are available for a subsequent enhancement project IDEA 4: Re-run the study using individual programmers rather than pairs – The study required people to be familiar with a programming technique which is not in wide usage in the industry – Significance: Is there any benefit to doing TDD in existing work environments without introducing pair programming?

8
June 24, 2005DCS891C - Summer 2005 8 Further analyze TDD productivity impact (idea 2) Other studies using students did find the practice to be neutral to productivity [P3]:P3 – This same study indicated that the vast majority of test subjects, on average, had only 4 months experience with Java or TDD Only one individual had set development practices – There was no productivity improvement or reduction using TDD in this environment (productivity-neutral) The conclusion expressed confusion at the results -- they were also expecting productivity improvement This could that the practice is productivity neutral once the learning curve is overcome, however Still other studies using professional programmers found the practice to be neutral or a slight impact on productivity [P2]P2 – This study did not use a control group during the actual experiment, but used ad-hoc testing of a similar prior project as a basis None of these studies used pair programming, but used traditional programming

9
June 24, 2005DCS891C - Summer 2005 9 Productivity for enhancement projects using TDD (idea 3) A couple of studies have indicated the potential for productivity improvements for future enhancement projects [P2, P4]P2P4 – Work to create a base of automated unit test cases would be complete – There would need to be time spent to update automated cases with the enhancements This doesnt seem to have a lot of existing studies – most do not go a second step to record enhancement productivity