Software Testing - Matt Whipple

Site Nav


I'm a fairly staunch supporter of having fairly comprehensive tests for software. While this is a fairly widely accepted best practice, it is also something which can be challenging to introduce to teams depending on culture and mindsets. I tend to alternate between test-driven development where the tests are written first and writing the tests after a first pass at code (which is then normally refactored and tightened up as a result of testing). Ultimate I rely on tests to specify and communicate what a piece of software does and without such tests I'm bereft of confidence that any intended value will be reliably delivered.

Test Frameworks

There is a large and ever-increasing list of test frameworks to choose from across a variety of use cases and environments. Selecting a test framework depends on a variety of factors, some of which are relatively independent and some of which are dictated by the system under test.

Over time I've used, created, and reviewed an array of tools used for testing. These have addressed a range of problems and have strongly reinforced the notion that different classes of verification warrant different approaches and that attempting to pursue a unified testing solution is an invitation to a tangled mess of complexity and dead code.

I'm currently building out a test solution which replaces some minimal, project-local bash scripts with an end-to-end standalone project which uses node and jest [jest-home]. I have plans on capturing some of the bash scripts here and decided to jot down some ideas to contextualize that upcoming content.

Or Lack Thereof

While some test frameworks are fairly minimal, others often represent a very heavy-handed solution to what may be a relatively simple problem. Automated tests typically amount to performing basic assertions which any general purpose programming language can do. Superficially test frameworks add a catalog of mechanisms to perform assertions and some convenience features on top of that core functionality such as allowing for more flexible definition of which tests to run and how the results may be reported.

More significantly if writing whitebox tests that are close to the code (such as unit tests) then test frameworks can help provide some assistance in properly isolating the code under test. This may include features such as generation of mock objects. In my experience such mocking functionality can introduce significant hidden complexity while only offering minor convenience if the code is designed appropriately using a decently expressive language. If the interaction is with a reference to a neatly segregated interface then creating an a more targeted test double which delivers equivalent value should be light work while avoiding whatever from of code generation or reflection is likely behind the provided mocking. In larger projects the light work may add up, and decent segeration may be unfeasible in cases where the interfaces are not under control and the platform relies on nominal typing at the time of definition or is similarly rigid, but for many cases far simpler alternatives exist to facilitate interaction testing or stubbing.

A far more valuable but less obvious feature of test framework runners is process isolation. When running running tests in an execution environment which is shared by the code under test it is desirable (and in many cases imperative) that the execution of tests is not adversely impacted by the execution of the code under test. If writing tests against code that crashes, hangs, exhausts resources, or otherwise causes instability the test framework should be able to identify and communicate that behavior along with any other behavior rather than being rendered unreliable by that behavior. Even in cases where none of the additional features of test frameworks are expected to be used a test runner or equivalent mechanism to provide this level of isolation should be used. Of note, however, is that such isolation may be inherently provided if the tests are more blackbox-y such that the system under test is running on a different process or host. While this may not provide the isolation necessary for more chaotic resource contention/starvation scenarios it does obviate what is provided by standard test frameworks.

In short when writing unit tests or integration tests which are expected to be exercising code it is very likely that some form of test framework or well established test patterns should be used. This also often carries additional benefits such as support for libraries that may be used. When testing a blackbox system which is compiled or is similarly executed then that decision may be far more discretionary. This may between using a test framework written in the same langauge as the system under test, a framework written in another language, or trading a system entirely for code or scripts written to perform the necessary assertions.

It's important to recognize the specific values that are promised by a given testing solution and which of those values are addressing concrete needs and what tradeoffs may be introducted. Using familiar tools can certainly provide significant efficiencies whereas keeping a clear boundary between blackbox tests and the systems that they're testing keep those boxes appropriately abstracted and replaceable. It may also be enormously useful to keep external tests closer to what real interactions with the system will look like rather than binding them to calls which cannot be readily extracted from their containing environment. For example linking and invoking internal methods of a CLI app are less likely to reflect user interactions than a shell invocation and calling a Web application through a specific client library is less likely to translate to actionable user feedback than something more standard such as a curl command.

All other things aside, any framework or lack thereof is ultimately only as valuable as the tests it contains, so whatever solution most fosters creation of valuable tests should be embraced.


  • [jest-home] @miscjest-home, title = Jest : Delightful JavaScript Testing, url =, status = done,

Author: mwhipple

Created: 2020-10-27 Tue 19:36