Unit testing with mock objects in C [LWN.net]

July 17, 2013

This article was contributed by Andreas Schneider and Jakub Hrozek

In software development, unit testing has become a standard part of many projects. Projects often have a set of tests to check some of the functionality of the source code. However, if there are parts which are difficult to test, then most unit testing frameworks in C don't offer an adequate solution. One example might be a program that communicates over a network. The unit tests should exercise not only the network facing components, but should also be able to be executed in environments that intentionally have no networking (such as build systems like Koji or the openSUSE Build Service).

Using a unit-test library with the support of mock objects helps testing situations like that described above. The CMocka unit-testing framework for C is an example of such a framework. We will show examples of how it can be used to add mock objects for testing your C programs. Hopefully that will lead to more use of mock objects by various projects.

Example

Consider a set of unit tests for the following system, which was taken from a Stack Overflow answer (with permission from the author):

You're implementing a model of a restaurant and have several functions in your restaurant representing smaller units, like chef, waiter, and customer. The customer orders a dish from the waiter, which the chef will cook and send (via the waiter) back to the customer.

It is generally easy to envision testing a low-level component like the "chef". In that case, you create a test driver that exercises the chef. One test in the test suite could make orders for different dishes and verifying that the chef behaves correctly and return the dish ordered. The test driver would also try to order dishes which are not on the menu to check that the chef will complain about the order.

Testing a component which is not a leaf but is in the middle of the hierarchy (like the waiter in our example) is much harder. The waiter is influenced by other components and to verify its correct behaviour we need to test it in isolation and make sure the results are not tainted by bugs in other parts of the program.

One way might be to test the waiter the same way the chef was tested. The test driver would again order dishes and make sure the waiter returns the correct dishes. But the test of the waiter component may be dependent on the correct behavior of the chef component. This dependency can be problematic if the chef component has a lot of test-unfriendly characteristics. It is possible that the chef isn't able to cook a dish because of missing ingredients (resources), he can't cook because his tools are not working (dependencies), or he has surprise orders (unexpected behavior).

But, as this is the waiter test, we want to test the waiter and not the chef. We want to make sure that the waiter delivers an order correctly to the chef and returns the ordered dish to the customer correctly. The test might also include a negative test — that the waiter is able to handle a wrong dish handed from the kitchen. In the real world, simulating failures can often be difficult.

Unit testing provides better results when testing different components independently, so the correct approach is to isolate the component or unit you want to test (the waiter in this case). The test driver should be able to create a "test double" (like a stunt double of an actor in a movie) of the chef and control it. It tells the chef what it expects it to return to the waiter after ordering a dish. This is the functionality that is provided by "mock" objects.

A large part of unit testing focuses on behavior, such as how the waiter component interacts with the chef component. A mock-based approach focuses on fully specifying what the correct interaction is and detecting when the object stops interacting the way it should. The mock object knows in advance what is supposed to happen during the test (which functions to call) and it knows how to react (which value it should return). These can be simply described as the behavior and state.

A custom mock object could be developed for the expected behavior of each test case, but a mocking framework strives to allow such a behavior specification to be clearly and easily indicated directly in the test case. The conversation surrounding a mock-based test might look like this:

test driver -> mock chef: expect a hot dog order and give him this dummy hot dog in response
test driver (posing as customer) -> waiter: I would like a hot dog please
waiter -> mock chef: 1 hamburger please
mock chef stops the test: I was told to expect a hot dog order!
test driver notes the problem: TEST FAILED! — the waiter changed the order

CMocka — an overview

One of the principles of CMocka is that a test application should only require the standard C library and CMocka itself, to minimize the conflicts with standard C library headers especially on a variety of different platforms. CMocka is the successor of cmockery, which was developed by Google but has been unmaintained for some time. So, CMocka was forked and will be maintained in the future.

CMocka is released under the Apache License Version 2.0. Currently, it is used by various Free Software projects such as the System Security Services Daemon (SSSD) from the FreeIPA project, csync, a user-level bidirectional file synchronizer, libssh, and elasto, a cloud storage client, which can talk to Azure and Amazon S3.

This article focuses on features that are unique to CMocka when compared to other unit testing frameworks. This includes mock objects and their usage, but it should be noted that CMocka also supports most of the features one would expect from any useful unit-testing framework, such as text fixtures or passing test states. Test fixtures are setup and teardown functions that can be shared across multiple test cases to provide common functions to prepare the test environment and destroy it afterward. With our kitchen example, the fixtures might make sure the kitchen is ready before taking orders from the waiter and cleaned up after the cooking has finished. Test states are used to provide private data which is passed around as a "state" of the unit test. For instance, if the kitchen initialization function returned a pointer to a "kitchen context", the state might contain a pointer to this kitchen context.

Users may want to refer to the CMocka documentation, where the common concepts are well explained and are accompanied by code examples.

How mock objects work in CMocka

As described in the example above, there are usually two parts in testing how an interface under test behaves with respect to other objects or interfaces we are mocking. The first is checking the input to see if the interface under test communicates with the other interfaces correctly. The second is returning pre-programmed output values and return codes in order to test how the interface under test handles both success and failure cases.

Using the waiter/chef interaction described earlier, we can consider a simple waiter function that takes an order from a customer, passes the order to the kitchen ,and then checks if the dish received from the kitchen matches the order:

    /* Waiter return codes:
     * 0  - success
     * -1 - preparing the dish failed in the kitchen
     * -2 - the kitchen succeeded, but cooked a different dish
     */
    int waiter_process_order(char *order, char **dish)
    {
        int rv;

        rv = chef_cook(order, dish);
        if (rv != 0) {
            fprintf(stderr, "Chef couldn't cook %s: %s\n",
                            order, chef_strerror(rv));
            return -1;
        }

        /* Check if we received the dish we wanted from the kitchen */
        if (strcmp(order, *dish) != 0) {
            /* Do not give wrong food to the customer */
            *dish = NULL;
            return -2;
        }

        return 0;
    }

Because it's the waiter interface that we are testing, we want to simulate the chef with a mock object for both positive and negative tests. In other words, we would like to keep only a single instance of a chef_cook() function, but pre-program it depending on the kind of test. This is where the mocking capability of the CMocka library comes to play. Our test driver will be named __wrap_chef_cook() and replace the original chef_cook() function. The name __wrap_chef_cook() was not chosen arbitrarily; as seen below, a linker flag makes it easy to "wrap" calls when named that way.

In order to fake the different results CMocka provides two macros:

will_return(function, value) — This macro adds (i.e. enqueues) a value to the queue of mock values. It is intended to be used by the unit test itself, while programming the behavior of the mocked object. In our example, we will use the will_return() macro to instruct the chef to succeed, fail, or even cook a different dish than he was ordered to.
mock() — The macro dequeues a value from the queue of test values. The user of the mock() macro is the mocked object that uses it to learn how it should behave.

Because will_return() and mock() are intended to be used in pairs, the CMocka library will consider the test to have failed if there are more values enqueued using will_return() than are consumed with mock() and vice-versa.

The following unit-test stub illustrates how a unit test would instruct the mocked object __wrap_chef_cook() to return a particular dish by adding the dish to be returned, as well as the return value, onto the queue. The function names used in the example correspond to those in the full example from the CMocka source.:

    void test_order_hotdog()
    {
        ...
        will_return(__wrap_chef_cook, "hotdog");
        will_return(__wrap_chef_cook, 0);
        ...
    }

Now the __wrap_chef_cook() function would be able to use these values when called (instead of chef_cook()) from the waiter_process_order() interface that is under test. The mocked __wrap_chef_cook() would pop the values from the stack using mock() and return them to the waiter:

    int __wrap_chef_cook(const char *order, char **dish_out)
    {
        ...
        dish_out = (char *) mock();  /* dequeue first value from test driver */
        ...
        return (int) mock();         /* dequeue second value */
    }

The same facility is available for parameter checking. There is a set of macros to enqueue variables, such as expect_string(). This macro adds a string to the queue that will then be consumed by check_expected(), which is called in the mocked function. There are several expect_*() macros that can be used to perform different kinds of checks such as checking whether a value falls into some expected range, is part of an expected set, or matches a value directly.

The following test stub illustrates how to do this in a new test. First is the the function we call in the test driver:

    void test_order_hotdog()
    {
        ...
        /* We expect the chef to receive an order for a hotdog */
        expect_string(__wrap_chef_cook, order, "hotdog");
        ...
    }

Now the chef_cook function can check if the parameter it received is the parameter which is expected by the test driver. This can be done in the following way:

    int __wrap_chef_cook(const char *order, char **dish_out)
    {
        ...
        check_expected(order);
        ...
    }

A CMocka example — chef returning a bad dish

This chef/waiter example is actually a part of the CMocka source code. Let's illustrate CMocka's capabilities with one part of the example source that tests that a waiter can handle when the chef returns a different dish than ordered. The test begins by enqueueing two boolean values and a string using the will_return() macro. The booleans tell the mock chef how to behave. The chef will retrieve the values using the mock() call. The first tells it whether the ordered item is a valid item from the menu, while the second tells it that it has the ingredients necessary to cook the order. Having these booleans allows the mock chef to be used to test the waiter's error handling. The final queued item is the order that the chef should return.

    int test_driver()
    {
        ...
        will_return(__wrap_chef_cook, true);     /* Knows how to cook the dish */
        will_return(__wrap_chef_cook, true);     /* Has the ingredients */
        will_return(__wrap_chef_cook, "burger"); /* Will cook a burger */
        ...
    }

Next, it's time to call the interface under test, the waiter, which will then call the mocked chef. In this test case, the waiter places an order for a "hotdog". As the interface specification described, the waiter must be able to detect when a bad dish was received and return an error code in that case. Also, no dish must be returned to the customer.

    int test_bad_dish()
    {
	int rv;
	char *dish;
	rv = waiter_process("hotdog", &dish);
	assert_int_equal(rv, -2);
	assert_null(dish);
    }

So the test driver programs the mock chef to "successfully" return a burger when it receives an order from the waiter—no matter what the order actually is for. CMocka invokes the waiter which calls the chef asking for a "hotdog". The chef dutifully returns a "burger" and the waiter should then return -2 and no dish. If it does, the test passes, otherwise it fails.

The full example, along with other test cases that use the chef/waiter analogy can be found in the CMocka repository.

Case study — testing the NSS responder in the SSSD

SSSD is a daemon that is able to provide identities and authenticate with accounts stored in a remote server, by using protocols like LDAP, IPA, or Active Directory. Since SSSD communicates with a server over a network, it's not trivial to test the complete functionality, especially considering that the tests must run in limited environments such as build systems. Often these are just minimal virtual machines or chroots. This section will describe how the SSSD uses CMocka for unit tests that simulate fetching accounts from remote servers.

SSSD consists of multiple processes which can be described as "front ends" and "back ends" respectively. The front ends interface with the Linux system libraries (mostly glibc and PAM), while the back ends download the data from the remote server for the front ends to process and return back to the system.

Essentially, the SSSD front end processes requests from the system for account information. If the data is available and valid in its cache, it returns that to the requester. Otherwise it requests the information via the back end; that information is then placed in the cache and the front end is notified. If the information could not be found in the cache, nor retrieved, an empty response is returned.

With traditional unit testing libraries, it's quite easy to test the sequence where valid data is present in the cache. Using stub functions simulating communication with the back end, it's also possible to test the sequence where the back end is asked for an account that does not exist. However, some scenarios are quite difficult to test, such as when the cache contains valid-but-expired data. In that case, the back end is supposed to refresh the cache with current data and return the data that was just fetched from the remote server.

SSSD uses the CMocka library to simulate behavior such as the one described above. In particular, there is a unit test that exercises the functionality of the NSS responder. It creates several mock objects that simulate updating the cache with results obtained from the network by creating a mock object in place of the back end. The mock object injects data into the cache to simulate the lookup. The test driver, which is simulating the system library that wants the account information, then receives the data that was injected.

After this unit test has finished, the test driver asserts that no data was present in the cache before the test started, and that the test returned seemingly valid data as if they were retrieved from some kind of a remote server. A very similar test has been developed to simulate the case where the cache contains some data when the test starts, but the data is not valid anymore. The test driver asserts that different (updated) data is returned to the test driver after the test finishes.

The complete unit test can be found in the SSSD project repository.

Using CMocka with `ld` wrapper support

CMocka has most of the features a standard unit-testing framework offers, but, in addition, has support for mock objects. As CMocka is a framework for C, mock objects normally replace functions: you have the actual implementation of a function and you want to replace it with your mock function. Consider the situation where a library contains an initialization function, in our example let's call it chef_init(), and some worker function, such as chef_cook() in the example above. You can't just mock one and use the other original function, as the same symbol name can't be used twice. There needs to be a way to trick the toolchain into using our mock worker function, but to keep using the original initialization function.

The GNU Linker has the ability to define a wrapper function and call this wrapper function instead of the original function (the gold linker supports this feature, too). This allows us to replace our actual implementation of a function with a mock object in our test code.

Keeping our chef example in mind, let's try to override the chef_cook() function. First, we need to define the wrapper. The name of the wrapper is always __wrap_symbol(), so our mock function will now be named __wrap_chef_cook(). That's a simple search-and-replace in the code, but please keep in mind that the will_return() macros that define what the mock() routines return will also need to change their argument to use the wrapper.

The second step is actually telling the linker to call __wrap_chef_cook() whenever the program would call chef_cook(). This is done by using the --wrap linker option which takes the name of the wrapped function as an argument. If the test was compiled using gcc, the invocation might look like:

    $ gcc -g -Wl,--wrap=chef_cook waiter_test.c chef.c

Another nice feature of the wrap trick is that you can even call the original function from the wrapper — just call a symbol named __real_symbol(), in our case, the test could call the original function by making a call to __real_chef_cook(). This trick is useful for keeping track of when a particular function was called, or for performing some kind of bookkeeping during the test.

You can refer to GNU binutils documentation for more information on the --wrap feature. A fully working implementation of the chef example using CMocka can be found in the CMocka repository.

Conclusion

Using mock objects improves testing efficiency tremendously, which will increase code quality. The authors hope that the article encourages readers to start using mock objects in their unit tests.

[Andreas Schneider and Jakub Hrozek are both Senior Software Engineers working at Red Hat. Jakub works on FreeIPA and SSSD and Andreas on Samba.]

(Log in to post comments)

Unit testing with mock objects in C