All right, it took me two weeks, but Scuttle v1.0.0 is now live. I think it’s pretty easy to use, and it’s very lightweight. The full README is over at github, of course, but it basically works as I outlined in the project introduction. No one really needed a new unit test framework, but I wrote one anyway, and now I’m going to use it in subsequent projects as often as I can.
Scuttle is now working as intended for the standard use case. As described in the introductory post, for a C project with a standard layout, adding unit testing with Scuttle requires the following steps:
Install scuttle.h either locally to the project or in the system include path
Install scuttle.sh either locally to the project or somewhere on your $PATH
Add test suite source files under test/, each corresponding to a module of your project, named test_<module>.c, using Scuttle’s simple macros
Add a test target to your main Makefile as follows:
bash scuttle.sh test
$(MAKE) -C test
Scuttle will generate test/Makefile, test/test_<module>.h and test/test_<module>_gen.h for each test/test_<module>.c suite, and the test/test_<project>.c harness; and the generated Makefile’s test target will build and run the test harness and pipe the output to test/log/test_<project>.log.
There’s a lot of room to make this more flexible and add some convenience features, but in terms of functionality it’s at MVP level now.
Tomorrow I’ll do a quick cleanup pass, make a few small tweaks I’ve already thought of, write some basic documentation, and actually push the code up to a public repo so if anyone’s interested in trying it out, it’ll be available.
A basic principle of engineering, software or otherwise, is “don’t reinvent the wheel.” I’m gonna anyhow.
Last week I wrote about wanting a very lightweight C unit testing solution, and, in my bullheaded way, wanting to write it myself rather than learn an already-existing system. It’s a commonplace in software engineering that the impulse to start over from scratch is one of the worst habits of programmers; one way or another, reinventing the wheel means doing lots of work you didn’t have to, usually to get a less satisfactory result. Either you end up hand-rolling something you should have just used a library for; or you look at a mass of legacy code full of cryptic comments and weird edge-case handling, think “whew, what a mess, how hard to maintain! better to just scrap it all and start fresh, so it will be cleaner and more elegant,” only to discover along the way that all those weird edge cases really exist, and end up with just as patchwork a code base as the one you meant to improve.
All that said, I’m doing it anyway. It’s not like I’m busy right now, and this is what’s engaging my brain.
So, suppose you have a project projname, and you want to add unit testing. Your project layout looks like this:
~/projname$ ls include
~/projname$ ls src
You want to add unit tests for the foo module, which is sensible because it’s pretty complex and you want to be sure all of it works exactly right. The foo module looks like this:
#endif /* _FOO_H */
To add unit testing with Scuttle (let’s say it stands for “simple C unit testing tool, limited edition”), you just need to put scuttle.h in your include path and scuttle.sh in your executable path, create a test/ directory, and add test_projname_foo.c:
printf("foo suite init\n");
printf("foo test setup\n");
printf("foo test teardown\n");
int i = foo();
int i = foo();
SREFUTE(i == 69)
You’ll have noticed that no test_projname_foo.h header exists: scuttle.sh will generate that, as well as a test/test_projname_foo_gen.c source file with some data structures, and test/Makefile, for you.
~/projname$ ls test/
This is Scuttle, v1.0.0.
* found suite test/test_projname_foo.c
* generated suite header test/test_projname_foo.h
* generated suite data test/test_projname_foo_gen.c
* generated harness test/test_projname.c
* generated makefile test/Makefile
Type 'make -C test/ test' to build and run your test harness.
~/projname$ ls test/
~/projname$ make -C test/ test[$(CC) output]
test/test_projname > test/log/test_projname.log
~/projname$ cat test/log/test_projname.log
This is Scuttle, v1.0.0.
Running test harness for: projname
Test suite projname_foo:
*** Suite passed: 2 / 2 tests passed.
*** 1 / 1 suites passed
Aside from the simple SASSERT(x) and SREFUTE(x), Scuttle provides convenience macros SASSERT_NULL(x), SASSERT_EQ(x,y), and SASSERT_STREQ(x,y), as well as SREFUTE versions of the same.
As of this writing, scuttle.h is at or near v1.0 completeness, targets for scuttle.sh‘s output are determined, and I’m beginning work on scuttle.sh itself. More on those soon.
At this point you are no doubt thinking, “why on earth aren’t you just using one of the many existing frameworks, which would solve these problems for you?” Excellent question! Now, moving on.
I have a bad habit of getting an idea for a software project, starting to write code without much planning ahead, very quickly discovering something I hadn’t considered or some tool I’m lacking, deciding to pivot to writing a separate utility or library to address that problem so that I can come back to the original thing later, and starting to write code for the new project without much planning ahead.
You can see where this is going, and it’s not Finishedprojectsville.
Over the years, though, I have managed to accumulate some tools to help counteract these tendencies. When I’m working on hobby projects, I usually like to work in plain C, with as little scaffolding as I can get away with — Vim will do nicely, thank you — but even so it turns out there’s some boilerplate that it’s tedious to retype every time I start a new project. So I wrote a Bash script that defines a function cproj():
/home/smadin $ cproj foo
/home/smadin $ ls
/home/smadin $ cd foo
/home/smadin/foo $ ls -F
include/ Makefile src/ test/
/home/smadin/foo $ ls include
/home/smadin/foo $ ls src
/home/smadin/foo $ ls test
foo_test.c Makefile test.h
/home/smadin/foo $ make
mkdir -p obj bin
/usr/bin/gcc -Wall -Werror -Iinclude -c -o obj/main.o src/main.c
/usr/bin/gcc -Wall -Werror -Iinclude -c -o obj/foo.o src/foo.c
make -C test
make: Entering directory '/home/smadi_000/dev/foo/test'
/usr/bin/gcc -g -Wall -Werror -I. -I../include -o foo_test.exe foo_test.c ../obj/foo.o
./foo_test.exe > foo_test.log
make: Leaving directory '/home/smadi_000/dev/foo/test'
/usr/bin/gcc -o bin/foo.exe obj/main.o obj/foo.o
/home/smadin/foo $ cat test/foo_test.log
initializing test data...
test 0 of 1...
foo_dummy() returned 42, expected 42
foo_test: 1 tests passed out of 1
As you can see, cproj() creates a very basic skeleton for a C project, including a very primitive, hand-rolled unit-testing facility. test.h just defines an array of function pointers and manually populates it with the declared test functions:
And foo_test.h:main() simply iterates over the array, breaking as soon as a test returns false. This has the advantage of being very simple and completely self-contained — the cproj() function contains the complete here-documents into which the project name is interpolated, so the only dependencies are the script itself and GCC — but also the serious disadvantage of being…very simple. There are no assertions, so every test case has to do checks and any logging manually, and return a true or false value accordingly. A single failure aborts the entire test harness, preventing accurate logging of pass/fail rate. There’s no facility for defining separate suites of test cases for related functionality, all managed by the same test harness, for more helpful reporting. And worst of all, it’s completely static: the user has to (that is, I have to) manually enter each test case function name 1) in the source file where I define the test case, 2) in the header where I declare the test case function, and 3) in the header where I define the function pointer array.
At this point you are no doubt thinking, “why on earth aren’t you just using one of the many existing unit-testing frameworks, which would solve these problems for you?” Excellent question! Now, moving on.
What I want, and I suspect what most programmers want, out of a unit test framework is for it to cause as little friction as possible. I want to have to think about the framework as little as possible in order to write tests and have them run and provide me useful output, so that in writing tests I’m spending almost all my mental energy on what I’m testing, how it’s supposed to work, and how it might fail. To remedy the defects of the current cproj approach, what I want is:
assertions (true/false, null/not null, equal/not equal, equal/not equal for strings) with some kind of inherent expected/actual logging, so I don’t have to write those sprintf()s in each test function
a boilerplate test harness I don’t have to write each time, which can run all the test cases without aborting on the first failure, and report a total pass rate
test cases split into suites (with appropriate reporting in the harness) so I don’t have to put everything in one big source file
dynamic discovery of tests so I don’t have to copy and paste each function name to three different places
minimal extra headers or other files so that integrating testing into a project is as simple as possible
And I think I can get there, or most of the way there, with one header and one shell script: the header to define assertion macros and so forth, and the script to scan source files (test suites) in the test subdirectory, parse out the suite and test case names, and generate the per-suite headers and the harness source file. I’ve been experimenting with this idea, and I’m partway there already.