Freek Lijten has a new post today continuing his look at the world of PHP extensions and focusing in on testing this time. He hopes to answer the question of what makes a good, effective set of tests to help increase the stability and quality of the extensions you write.

In my previous blog I took you through the process of getting PHP and extensions compiled, generating code coverage and running tests. What I did not talk about was what makes a good test. I hope to correct on this by adding this post and going into more detail on the actual writing of tests itself.

Using the same extension as before (enchant) he goes through the addition of a test for the enchant_dict_add_to_session function. He start by showing how much the function is currently tested (hint: none) and code coverage. He points out that 100% coverage is just one metric in a set that should be considered and not the final goal. He shares a simple test for the function that checks to see if a certain word exists in a dictionary. The coverage report shows all lines being executed, but there's a lot not tested, at least conceptually. He shows how to test "the spirit" of the function with additional tests for non-existent words, spell checking and if a word is not in the dictionary at all. PHP example code shows these tests kinds of tests to illustrate the steps he's talking about.

I spent the vast majority of 2014 not contributing to open source, so I kicked off 2015 by making Humbug available on Github. Humbug is a Mutation Testing framework for PHP. Essentially, it injects deliberate defects into your source code, designed to emulate programmer errors, and then checks whether your unit tests notice. If they notice, good. If they don’t notice, bad. All quite straightforward. [...] This article however is mostly reserved to explain why I wrote Humbug, and why Mutation Testing is so badly needed in PHP. There’s a few closing words on Mutation Testing performance which has traditionally been a concern impeding its adoption.

He starts off by talking about the idea of "code coverage" when it comes to writing unit tests...and how 100% coverage usually ends up being a lie. He points out that the current methods of line-based coverage metrics can lead to false results and that it's more about test quality rather than volume of tests. He then moves into talking about mutation testing and where it fits in the test quality puzzle. Finally, he mentions one thing to watch out for when trying out mutation testing and the performance jump (longer, not shorter) it can introduce into your testing cycle.

Derick Rethans has posted the first in a series focusing on the Xdebug tool and the code coverage functionality it can provide via PHPUnit's testing. In this first post he catches the reader up on the current state of things and what all the Xdebug tool can do.

Since ages Xdebug has provided code coverage support for PHPUnit, a way to show which lines are covered by your test cases. But I never really wrote about how it works. A recently filed bug prompted me to write this post, as well as a follow up post on Code Coverage's future.

He starts off with the early days of Xdebug, how it hooked into the Zend Engine (that powers a lot of PHP behind the scenes) and when it was triggered. This came with its own set of problems so Xdebug was updated to overload some opcodes. He talks about how it can calculate the unused lines and determines which lines can be covered in the code coverage results. He provides some example code showing the execution of the coverage report on a simple function and try/catch handler, complete with the HTML output of the results.

There's a good post over on the NetTuts.com site today talking about unit testing, the myths around code coverage and why it may not be as important as you think.

Most cubicle workplaces disappeared and programmers started loving their craft. With the advent of Agile techniques and the Software Craftsmanship movement, many new tools emerged to help the programmer and the process. TDD is slowly becoming the de facto way of writing code and the secrets of SCRUM or Kanban were revealed even to the programmers in the darkest corners of the cubicle world. Automated testing and test driven development (TDD) are some of the essential techniques Agile provided to us programmers. And a tool that comes with those methodologies is used to produce test code coverage, which is the topic of this article.

The article starts with a brief definition of code coverage and gets right into an example class, PHPUnit test and the results of a code coverage generation. They show both output options, the text-only output and the full HTML output with clickable links and visualization of the covered lines of code. There's also an example of generating the coverage inside an IDE (PHPStorm). The post finishes with a look at the myths of code coverage including: "100% covered is bug free" and that "gaming the system" is pretty easy.

In her latest postAnna Filina proposes that full test coverage is an impractical way to measure the quality of your software. It can provide a false sense of security, even if the tests are poorly written.

Many developers claim that to achieve high quality software, developers must create automated tests that ensure that all possible execution routes have been covered. This is also known as full path coverage. I will argue that different types of software require different testing approaches and that full path coverage is impractical in almost every case. Too many tests simply create clutter.

She looks at how it's impractical to expect that all tests will be written efficiently or even correctly. Even simple tests are enough to show up on code coverage reports but may only be painting part of the picture. She also notes that not all software can be tested the same way - things like APIs require different testing skills/methods than something like consumer software.

In the end, there are no exact rules on how much to test. The most important thing to keep in mind is that writing tests for the sake writing tests is futile and costly. [...] Focus on building great software. Tests are a tool to make it better. Just don’t overdo it.

The Codeception testing tool has released a new major update with some interesting new features - the expected feature for generating code coverage reports (similar to other tools) but there's also the idea of "remote code coverage" introduced.

There is no magic in local codecoverage. XDebug and PHP_CodeCoverage libraries do their job. The tricky thing is remote codecoverage. We attach small script into application's front controller. When a special header is sent this script starts to collect coverage information. And in the end of tests, this data is merged, serialized and sent back to Codeception. So you can test and collect coverage report even on staging servers in real environment.

They also mention a few other updates in the release - new Redis and MongoDb modules, UX improvements and the normal bugfixes. You can find out more about the code coverage feature in their manual or just about the project in general from the main site.

Derick Rethans has a new post to his blog today talking about some work that's been done to speed up XDebug's code coverage generation. Changes in the coming 2.2 release have some improvements that make things perform better and put less stress on PHP in the process.

Code coverage tells you how much of your code base is actually being tested by your unit tests. It's a very useful feature, but sadly, it slows down PHP's execution quite a lot. One part of this slowdown is the overhead to record the information internally, but another part is because I have to overload lots of opcodes. (Opcodes are PHP's internal execution units, similar to assembler instructions) They are always overloaded even if code coverage is not used, because it's only safe to overload them for the whole request.

These changes were from a combination of contributions from Taavi Burns and a new ini setting that will allow you to enable or disable the code coverage in XDebug. Benchmarking shows a good amount of time reduction in coverage runs - dropping anywhere from a few seconds to over a minute. He also mentions the idea of "modes", shortcuts to predefined settings for different types of reporting (like "profiling" or "tracing").

Sebastian Bergmann has a new post to his blog talking about some of the future plans for better code coverage metrics for PHP applications (not just the statistics that we have now as generated from PHPUnit runs combined with Code_Coverage PEAR package and Xdebug).

Xdebug currently only supports what is usually referred to as Line Coverage. This software metric measures whether each executable line was executed. Based on the line coverage information provided by Xdebug, PHP_CodeCoverage also calculates the Function / Method Coverage software metric that measures whether each function or method has been invoked.

The various kinds of coverage they're planning the in future include statement coverage, branch coverage (boolean evaluation), call coverage, path coverage with an alternative of linear code sequence and jump coverage (LCSAJ).

About half the "Build failed" mails I’ve gotten from Jenkins in the last two weeks where not due to me breaking the tests but just PHPUnit segfaulting. "Wait! I know PHPUnit can’t segfault!", only PHP itself can. And it does, quite often. For some reason that probably has to do with using PHP 5.2.OLD it doesn’t survive generate the clover.xml file or the HTML report about 20% of the times it’s being run.

He finds a solution to keep this from happening - not correcting the segfault as much as running the testing again to get it to pass. He wraps the call to PHPUnit inside of a simple shell script that tries to execute the PHPUnit testing suite as long as a segfault isn't thrown. If something was seriously broken, this could cause all sorts of problems, but in theory it's a simple hack that gets the job done.

Volker Dusch has a suggestion for all of those developers out there really concerned about the quality of your code - consider running your unit tests every time you save a file with the help of a tool called PyWatch.

At the moment he placed the Idea in my head that i want my testsuite to run every time a file changes anyway, let someone else figure out how that is triggered. Well, someone did figure that out, thank you. It's called "PyWatch" and is a simple python script that monitors files for changes and executes a script if one has changed.

Obviously you wouldn't want to keep this going for larger code bases (the potential for overlap is horrible) but it could be quite useful when doing test-driven development to save yourself some hassle at the outset. He includes the commands you'll need to get things running and a simple PHP script that points the PyWatch instance to the correct files.