It’s like Python’s doctest but for CLIs. Given a text file containing snippets of shell sessions (prompts with commands and their expected outputs), clitest executes the snippets and verifies the actual output matches the described output. It is written in shell language. The project lives on GitHub and is available under MIT license. It does not see much development anymore, but that is fairly understandable given its simplicity. Nevertheless, the author continues to maintain deliverables and documentation. The project has comprehensive documentation in its README.

Installation and usage

Installation and usage are trivial: the whole clitest is just a single shell script with no external dependencies. Hence, it is easy to include copies of clitest in repositories. I was curious how is the little tool implemented in shell, and I discovered the code is nice and readable. I could not resist trying to run ShellCheck on it, and it only spat out few style issues: good job!

I tried to use clitest as a testing driver for pyff project examples so that I only need to provide expected output for each example. It worked nice, although the way how clitest executes the commands to test makes it a little sensitive about $CWD, $PATH and the like. Fortunately, these are quite straightforward to sort out. The test files for pyff are standard Python source code files, so I needed to embed the clitest instructions into Python comments (lines starting with the # character). I used the --prefix option for this; it tunes the way how clitest searches for testable snippets in files.

I have encountered a minor pitfall where pyff does not have entirely deterministic output. The output is correct, but the ordering of some elements may be different between runs. clitest fully matches the output against the expected one, so it is harder to use clitest in these cases (I have decided to make pyff deterministic, even if the order does not matter). There is a clitest feature which can alter the matching when using the inline output syntax, but my use case did not match the inline output feature well.

When would I use it?

I like clitest because it is a single file drop-in without dependencies. This makes it easy to drop it into a repository where it can serve as a lightweight integration test driver: all you need are some input files, define how to execute the program under test and specify expected output.

A second nifty use case is mentioned in the clitest documentation: it can be nicely used to test the code snippets in your documentation for correctness. Often, you will write the documentation in markdown or something similar, and it will contain snippets of how to execute the program and what is the expected output. This way, you can feed your documentation straight into clitest which can, without further work, validate your examples are still correct. You may even do this within a CI automatically, so your documentation is always up-to-date.