Test framework comparison

Here's a report of various testing frameworks which concentrates on that authors needs. A report which looks in detail at how each testing framework sandboxes each test would be nice.

How does CLUnit compare with XPTEST?

I tried using CLUnit for a little project recently, and played only a bit with XPTEST too. Here are my impressions:

In CLUnit, a test has a description and a category. Not a name. Or, sometimes the description is called its name instead. It turns out that you either have to give tests a very short description, or forget about (run-named-test), unless you want to type stuff like (run-named-test "Test whether foo-frobnicate works on Sunday afternoons when your cat stomped over your keyboard").
Both are not as polished as other xUnit frameworks. For example, both do not show anything green if you've been a nice guy (nor anything similar that would work without a GUI toolkit). Nor do they have all those convenient macros/functions to say whether a test failed or succeeded: CLUnit by default wants the output of a test function to be t (or eq[[ua]l[p]]) a given value), in XPTEST you have to call a function called FAILURE to indicate failure (or signal a TEST-FAILURE condition, or pass something false to TEST-ASSERT). Something like assert-equal, assert-not-equal, assert-condition etc. is missing.
XPTEST seems to be a little more like other xUnits. It is based on fixture classes, setup and teardown methods etc.

I am a little disappointed with CLUnit actually, and will look at XPTEST. It looks better, but then, I haven't used it in practice.

[Dunno who wrote the above --- rpg]

Comparison with RT

I have been using the venerable RT framework for testing a large program. It is simple, and seems to be very reliable.

On the other hand, I don't know how scalable it is. For a small system, you can define a bunch of tests, and then you call (DO-TESTS). That's fine. But I'm not sure how one should progress given a larger system, with multiple modules. In this case, I think one would like to have a framework that would permit one to group tests into "families" or modules, and only run the tests for a specific family.

I believe that some of these test frameworks provide this facility, but there are a bewildering number of them. I'm going to try to find a framework that will give me this test-grouping capability without causing too much upset when migrating from RT.

Quick follow-up: Seems like there's a modified version of RT that permits one to attach "notes" to entries (tests). Notes can contain disabling info. So it's possible to do a simple version of what I want using the latest version of RT, I think. On the other hand, the notes framework seems like a general-purpose, low-level tool to do the test grouping, and doesn't seem terribly well integrated into RT (not mentioned in docs, nor is it exercised in RT's self-test suite). Maybe it would be better to get a tool that does just this thing, and does it in a way that's integrated into the framework tightly.

Robert Goldman [rpg --- not that rpg]

This page is linked from: CLUnit test framework XPTEST

CLiki pages can be edited by anyone at any time. Imagine a fearsomely comprehensive disclaimer of liability. Now fear, comprehensively