Comments on Mock-Based Testing

很好的一篇文章，深入说了很多关于测试的东西。

A article from Technoetic
- Posted in Software Dev., Agile by Steve Bate

I recently read a blog entry with criticisms of mock-based testing. The author raised several “issues” with using mocks to support unit testing. I’m commenting here since the author has closed comments on the original blog entry.

Issue 1: Poor integration tests, as everything is being tested in isolation

I’ve had good experience with mock-based testing. However, it’s obvious that mocks will only test classes in isolation. I use both unit tests and integration tests (sometimes called system or acceptance tests) together. The need for integration tests is not an issue for mock-based techniques and is not a good reason to use less mocks. It’s just a different aspect of testing. A more common issue in my experience is that people in the agile community who are new to testing often don’t understand these different aspects of testing and seem to believe that mock-based unit testing and integration testing are mutually exclusive options. The lack of common terminology in the community only worsens the problem. For some people, a “unit” is a class or small group of tightly coupled classes. For others, it’s a large portion of the software product. Most agile developers seem to call every test they write a unit test. It’s become so confusing for some teams that I’ve seen terminology like “integration unit tests” being used to describe testing strategies.

But, back to the topic. Poor integration testing is simply the result of lack of integration tests. Mocks do not cause a lack of integration tests. A team makes that choice, probably based on a weak understanding of the tradeoffs between isolation (unit) and integration (system) testing.

Issue 2: Mocks add complexity to the software design.

“I’ve seen numerous occasions where the introduction of mocks has added a large amount of complexity to an otherwise simple design. This complexity leads to higher implementation costs, a higher cognitive load on the developers working on the system, and higher maintenance costs (as there’s more code to maintain). “

The author appears to focused on the increased use of interfaces when using mock-based testing and expresses the opinion that interfaces should only be used where we’d want to be able to replace one implementation with another. First, there are other reasons to use interfaces. In general, interfaces are useful for managing dependencies between software components or subsystems. This can be beneficial even if the implementations do not change (see The Dependency Inversion Principle).

A modular software design will generally make it easier to use mock-based testing without altering the design specifically for the mocks. However, there are times when the software must be modified to support testing. Fortunately, the changes needed to support testing often, if done well, support the modularity goal.

In my experience, extra interfaces don’t add a significant maintenance overhead. Most effort is spent implementing the interface. The time writing the interface itself (or extracting it using an IDE’s refactoring tools) is negligible.

My Conclusions

In almost every case, I see the “simplicity” gained by not using mocks overshadowed by complex test setups to initialize large groups of dependent objects. The dark side of integration testing is that it’s often very slow for large numbers of tests. Some teams are using continuous integration tools like Cruise Control to run their “unit tests” (usually they are actually integration tests). This delays the feedback about broken builds but is often necessary because the tests run so slow. I realize there other reasons for using CC, but this is a common one from what I’ve seen and heard.

I’ve worked on teams where we had thousands of tests that ran in less than 15-20 seconds total on a developer workstation. This was a direct result of heavy use of mock-based testing. We also had a slower suite of integration tests that required 4-5 minutes to run. We didn’t need a continuous integration server because we were able to integrate and run our unit tests before every commit to the source control system. The team integrated 10-20 times/day and broken builds were practically nonexistent over the several years I worked with them. In the very rare cases when the build did break, it was typically fixed in a matter of minutes.

The other benefit of the isolation testing enabled by mocks is the ability to pinpoint problems much more quickly. It’s a form of the divide and conquer problem solving strategy, only the divide part is already done. The conquering is relatively easy compared to tracking down the cause of test failures when many classes are being exercised in a test.

My experience was that our mock based unit tests caught about 98% of the code problems before the code was ever committed to source control. The integration tests caught about another 1% beyond that (almost always because of a flaw in the mock-based testing) and manual testing caught the other 1%.

- Posted in Software Dev., Agile by Steve Bate