Testing your full software stack with cwrap

Testing network applications correctly is hard. The biggest challenge is often to set up the environment to test a client/server application. One option is to set up several virtual machines or containers and run a full client/server interaction between them. But building this environment might not always be possible; for example some build systems typically have no network at all and run as a non-privileged user. Also for newcomers, who want to contribute code to your project, it is often a difficult and time-consuming task to set up that kind of development environment.

Reading and running the test cases is normally a good entry point toward understanding a project, because you learn how it is set up and how you need to use the API to achieve your goal. For these reasons, it would be preferable if there was a way to run the tests locally using a non-root user, while still being able to run in an environment as close to real world as possible. Avoiding the testing of code that requires elevated privileges or networking is usually not an option, because many projects have a test-driven development model. This means to submit new code or to fix issues, a test case is required so regressions are avoided.

The cwrap project

The cwrap project aims to help client/server software development teams that are trying to gain full functional test coverage to complete that task. It makes it possible to run several instances of the full software stack on the same machine and perform local functional testing of complex network configurations. Daemons often require privilege separation and local user and group accounts, separate from the hosting system. The cwrap project does not require virtualization or root credentials and can be used on different operating systems.

It is basically like The Matrix, where reality is simulated and everything is a lie.

cwrap is a new project, but the ideas and the origin of the project are from the Samba codebase. cwrap presents the internals of one of the most advanced FOSS testing systems that has helped Samba developers for many years to test their protocol implementations. Samba is complex, it provides several server components that need to interact with each other. It provides a client executable, a client library, and a testing suite called smbtorture. These need to be run against different server setups to test the correctness of the protocols and server components.

In trying to test your server, you may run into some problems. Your server might need to open privileged ports, which requires superuser access. If you need to run several instances of daemons for different tasks, then the setup becomes more complex. An example would be that you want to test a SSH client with Kerberos. So you need a KDC (key distribution center) and an SSH server. If you provide login or authentication functionality, user and group accounts on the system are required. This means each machine you run the tests on needs to have the same users. To be able to switch to a user after authentication, you have to be root in the first place. All these things make testing harder and the setup more complex.

What you actually want is to be able to run all required components on a single machine: the one a developer is working on. All tests should work as a normal non-privileged user. So what you really want is to just run make test and wait till all tests are finished.

The cwrap project enables you to set up such an environment easily by providing three libraries you can preload to any binary.

What is preloading?

Preloading is a feature of the dynamic linker (ld). It is a available on most Unix systems and allows loading a user-specified, shared library before all other shared libraries that are linked to an executable.

Library preloading is most commonly used when you need a custom version of a library function to be called. You might want to implement your own malloc(3) and free(3) functions that would perform rudimentary leak checking or memory access control for example, or you might want to extend the I/O calls to dump data when reverse engineering a binary blob. In those cases, the library to be preloaded would implement the functions you want to override. Only functions in dynamically loaded libraries can be overridden. You're not able to override a function the application implements by itself or links statically with. More details can be found in the man page of ld.so.

The wrappers use preloading to supply their own variants of several system or library calls suitable for unit testing of networked software or privilege separation. For example, the socket_wrapper includes its version of most of the standard API calls used to communicate over sockets. Its version routes the communication over local sockets.

The wrappers

cwrap consists of three different wrappers. Each of them implements a set of functions to fulfill a testing task. There is socket_wrapper, nss_wrapper and uid_wrapper.

socket_wrapper

This library redirects all network communication to happen over Unix sockets, emulating both IPv4 and IPv6 sockets and addresses. This allows you to start several daemons of a server component on the same machine without any conflicts. You are also able to simulate binding to privileged ports below port 1024, which normally requires root privileges. If you need to understand the packet flow to see what is happening on the wire, you can also capture the network traffic in pcap format and view it later with tools such as Wireshark.

The idea and the first incarnation of socket_wrapper was written by Jelmer Vernooij in 2005. It made it possible to run the Samba torture suite against smbd in make test. From that point in time, we started to write more and more automated tests. We needed more wrappers as the test setup became increasingly complex. With Samba 4.0 we needed to test the user and group management of an Active Directory server and make it simple for developers to do that. The technology has been in use and tested a while now. But because the code was embedded in the Samba source tree, it wasn't possible to use it outside of the Samba code base. The cwrap project now makes this possible.

There are some features in development, like support for IP_PKTINFO in auxiliary messages of sendmsg() and recvmsg(). We also would like to support for fd-passing with auxiliary messages soon to implement and test some new features for the Samba DCERPC infrastructure.

Lets take a look how socket_wrapper works on a single machine. Here is a demo you can run yourself after you have installed it:

    # Open a console and create a directory for the unix sockets.
    $ mktemp -d
    /tmp/tmp.bQRELqDrhM

    # Then start nc to listen for network traffic using the temporary directory.
    $ LD_PRELOAD=libsocket_wrapper.so \
      SOCKET_WRAPPER_DIR=/tmp/tmp.bQRELqDrhM \
      SOCKET_WRAPPER_DEFAULT_IFACE=10 nc -v -l 127.0.0.10 7

    # nc, listens on 127.0.0.10 because it is specified on the command-line
    # and it corresponds to the SOCKET_WRAPPER_DEFAULT_IFACE value specified

    # Now open another console and start 'nc' as a client to connect to the server:
    $ LD_PRELOAD=libsocket_wrapper.so \
      SOCKET_WRAPPER_DIR=/tmp/tmp.bQRELqDrhM \
      SOCKET_WRAPPER_DEFAULT_IFACE=100 \
      SOCKET_WRAPPER_PCAP_FILE=/tmp/sw.pcap nc -v 127.0.0.10 7

    # (The client will use the address 127.0.0.100 when connecting to the server)
    # Now you can type 'Hello!' which will be sent to the server and should appear
    # in the console output of the server.
    # When you have finished, you can examine the network packet dump with
    # "wireshark /tmp/sw.pcap"

nss_wrapper

There are projects that provide daemons needing to be able to create, modify, and delete Unix users. Others just switch user IDs to interact with the system on behalf of another user (e.g. a user space file server). To be able to test these, you need the privilege to modify the passwd and group files. With nss_wrapper it is possible to define your own passwd and group files which will be used by the software while it is under test.

If you have a client and server under test, they normally use functions to resolve network names to addresses (DNS) or vice versa. The nss_wrapper allows you to create a hosts file to set up name resolution for the addresses you use with socket_wrapper.

The user, group, and hosts functionality are all defined as wrappers around the Name Service Switch (NSS) API. The Name Service Switch is a modular system, used by most Unix systems, that allows you to fetch information from several databases (users, groups, hosts, and more) using loadable modules. The list and order of modules is configured in the file /etc/nsswitch.conf. Usually, the nsswitch.conf file contains the "files" module shipped with glibc that looks up users in /etc/passwd, groups in /etc/group, and hosts in /etc/hosts. But it's also possible to define additional sources of information by configuring third party modules — a good example might be looking up users from LDAP using nss_ldap.

Here is an example of using nss_wrapper to handle users and groups:

    $ echo "bob:x:1000:1000:Bob Gecos:/home/test/bob:/bin/false" > passwd
    $ echo "root:x:65534:65532:Root user:/home/test/root:/bin/false" >> passwd
    $ echo "users:x:1000:" > group
    $ echo "root:x:65532:" >> group
    $ LD_PRELOAD=libnss_wrapper.so NSS_WRAPPER_PASSWD=passwd \
        NSS_WRAPPER_GROUP=group getent passwd bob
    bob:x:1000:1000:Bob Gecos:/home/test/bob:/bin/false

The following shows nss_wrapper faking the host name:

    $ LD_PRELOAD=libnss_wrapper.so NSS_WRAPPER_HOSTNAME=test.example.org hostname
    test.example.org

Here, nss_wrapper simulates host name resolution:

    $ echo "fd00::5357:5faa test.cwrap.org" > hosts
    $ echo "127.0.0.170 test.cwrap.org" >> hosts
    # Now query ahostsv6 which returns only IPv6 addresses and
    # calls getaddrinfo() for each the entry.
    $ LD_PRELOAD="libnss_wrapper.so" NSS_WRAPPER_HOSTS=hosts \
      getent ahostsv6 test.cwrap.org
    fd00::5357:5faa DGRAM  test.cwrap.org
    fd00::5357:5faa STREAM test.cwrap.org

uid_wrapper

Some projects, such as a file server, need privilege separation to be able to switch to the user who owns the files and do file operations on their behalf. uid_wrapper convincingly lies to the application, letting it believe it is operating as root and even switching between UIDs and GIDs as needed. You can start any application making it believe it is running as root. We will demonstrate this later. You should keep in mind that you will not gain more permissions or privileges with uid_wrapper than you currently have; remember it is The Matrix.

Maybe you know that glibc has support for switching the user/group only for the local thread. For example calling setuid(1000) synchronizes all threads to change to the given UID. The setuid(), setguid(), etc. functions send a signal to each thread, telling it that it should change the relevant ID. The signal handler of each thread for the signal then uses syscall() using the corresponding SYS_setXid constant for the local thread. So, under glibc, if you want to change the UID only for the local thread, you have to make the system call directly:

    rc = syscall(SYS_setruid, 1000, 0);

uid_wrapper has support for glibc's special privilege separation with threads. It intercepts calls to syscall() to handle the remapping of UIDs and GIDs. Here is an example of uid_wrapper in action:

    $ LD_PRELOAD=libuid_wrapper.so UID_WRAPPER=1 UID_WRAPPER_ROOT=1 id
    uid=0(root) gid=0(root) groups=100(users),0(root)

How are the wrappers tested?

You may sense a bit of a conflict of interest with wrappers. On one hand, this article stated that unit tests with wrappers strive to simulate the real world environment as closely as possible. On the other hand, the wrappers substitute such fundamental calls as socket() and getpwnam(). It's paramount that the wrappers be extremely well tested so that you, as a user of the wrappers, are confident that any failure in testing implemented using the wrappers is a bug in the program under test and not an unwanted side effect of the wrappers. To this end, the wrappers include a large unit test suite that make sure the wrappers function as intended. At the time of this writing, the code coverage for wrappers is pretty high: nss_wrapper 79%, socket_wrapper 77%, and uid_wrapper 85% code coverage.

As an example of a unit test, the socket_wrapper implements a very simple echo server. The unit tests that exercise the read() or write() calls then connect to the echo server instance that is seemingly running on a privileged port. In fact, the echo server is run using socket_wrapper, so all communication is redirected over a local socket. You can inspect the unit test in the Samba repository. The CMakeLists.txt file also gives a good overview of how the tests are set up.

The wrappers leverage the cmocka unit testing framework that was covered in an earlier LWN article. In short, the cmocka library provides unit test developers with the ability to use mock objects. Moreover, the cmocka library has very low dependency footprint; in fact, it requires only the standard C library.

All the wrapper libraries are built using the cmake build system. In order to provide cwrap developers with an easy-to-use dashboard that displays the results of unit tests, an instance of the cdash dashboard is running and scheduling tests on several operating systems including several Linux distributions, FreeBSD, and OpenIndiana (descended from OpenSolaris). Currently the i686 and x86_64 architectures are tested. The dashboard is a one-stop view that lets you see if any of the unit tests has trouble or if compiling the wrappers or their unit tests yields any compiler errors or warnings.

Final thoughts

Regular LWN readers may have read about namespaces in Linux. These provide similar functionality as the lightweight virtualization layer mechanism known as containers. But to set up namespaces, you often will need root privileges. When distributions enable user namespaces, that requirement will go away, but there is another problem: namespaces are not available on BSD or Solaris.

Currently Samba is the only user of the cwrap libraries since cwrap was not available for external consumption until recently. Andreas is currently working on cwrap integration to test libssh against an OpenSSH sshd server. We are also planning to improve the test environment of SSSD, but we didn't have time to work on it yet. At Red Hat, Quality Engineering has started to write tests for nss_ldap using nss_wrapper, but they are not upstream yet. If you plan to use cwrap, join us on the #cwrap IRC channel on Freenode.

Index entries for this article
GuestArticles	Schneider, Andreas

---

to post comments