User: Password:
Subscribe / Log in / New account


Testing your full software stack with cwrap

April 23, 2014

This article was contributed by Andreas Schneider and Jakub Hrozek

Testing network applications correctly is hard. The biggest challenge is often to set up the environment to test a client/server application. One option is to set up several virtual machines or containers and run a full client/server interaction between them. But building this environment might not always be possible; for example some build systems typically have no network at all and run as a non-privileged user. Also for newcomers, who want to contribute code to your project, it is often a difficult and time-consuming task to set up that kind of development environment.

Reading and running the test cases is normally a good entry point toward understanding a project, because you learn how it is set up and how you need to use the API to achieve your goal. For these reasons, it would be preferable if there was a way to run the tests locally using a non-root user, while still being able to run in an environment as close to real world as possible. Avoiding the testing of code that requires elevated privileges or networking is usually not an option, because many projects have a test-driven development model. This means to submit new code or to fix issues, a test case is required so regressions are avoided.

The cwrap project

The cwrap project aims to help client/server software development teams that are trying to gain full functional test coverage to complete that task. It makes it possible to run several instances of the full software stack on the same machine and perform local functional testing of complex network configurations. Daemons often require privilege separation and local user and group accounts, separate from the hosting system. The cwrap project does not require virtualization or root credentials and can be used on different operating systems.

It is basically like The Matrix, where reality is simulated and everything is a lie.

cwrap is a new project, but the ideas and the origin of the project are from the Samba codebase. cwrap presents the internals of one of the most advanced FOSS testing systems that has helped Samba developers for many years to test their protocol implementations. Samba is complex, it provides several server components that need to interact with each other. It provides a client executable, a client library, and a testing suite called smbtorture. These need to be run against different server setups to test the correctness of the protocols and server components.

In trying to test your server, you may run into some problems. Your server might need to open privileged ports, which requires superuser access. If you need to run several instances of daemons for different tasks, then the setup becomes more complex. An example would be that you want to test a SSH client with Kerberos. So you need a KDC (key distribution center) and an SSH server. If you provide login or authentication functionality, user and group accounts on the system are required. This means each machine you run the tests on needs to have the same users. To be able to switch to a user after authentication, you have to be root in the first place. All these things make testing harder and the setup more complex.

What you actually want is to be able to run all required components on a single machine: the one a developer is working on. All tests should work as a normal non-privileged user. So what you really want is to just run make test and wait till all tests are finished.

The cwrap project enables you to set up such an environment easily by providing three libraries you can preload to any binary.

What is preloading?

Preloading is a feature of the dynamic linker (ld). It is a available on most Unix systems and allows loading a user-specified, shared library before all other shared libraries that are linked to an executable.

Library preloading is most commonly used when you need a custom version of a library function to be called. You might want to implement your own malloc(3) and free(3) functions that would perform rudimentary leak checking or memory access control for example, or you might want to extend the I/O calls to dump data when reverse engineering a binary blob. In those cases, the library to be preloaded would implement the functions you want to override. Only functions in dynamically loaded libraries can be overridden. You're not able to override a function the application implements by itself or links statically with. More details can be found in the man page of

The wrappers use preloading to supply their own variants of several system or library calls suitable for unit testing of networked software or privilege separation. For example, the socket_wrapper includes its version of most of the standard API calls used to communicate over sockets. Its version routes the communication over local sockets.

The wrappers

cwrap consists of three different wrappers. Each of them implements a set of functions to fulfill a testing task. There is socket_wrapper, nss_wrapper and uid_wrapper.


This library redirects all network communication to happen over Unix sockets, emulating both IPv4 and IPv6 sockets and addresses. This allows you to start several daemons of a server component on the same machine without any conflicts. You are also able to simulate binding to privileged ports below port 1024, which normally requires root privileges. If you need to understand the packet flow to see what is happening on the wire, you can also capture the network traffic in pcap format and view it later with tools such as Wireshark.

The idea and the first incarnation of socket_wrapper was written by Jelmer Vernooij in 2005. It made it possible to run the Samba torture suite against smbd in make test. From that point in time, we started to write more and more automated tests. We needed more wrappers as the test setup became increasingly complex. With Samba 4.0 we needed to test the user and group management of an Active Directory server and make it simple for developers to do that. The technology has been in use and tested a while now. But because the code was embedded in the Samba source tree, it wasn't possible to use it outside of the Samba code base. The cwrap project now makes this possible.

There are some features in development, like support for IP_PKTINFO in auxiliary messages of sendmsg() and recvmsg(). We also would like to support for fd-passing with auxiliary messages soon to implement and test some new features for the Samba DCERPC infrastructure.

Lets take a look how socket_wrapper works on a single machine. Here is a demo you can run yourself after you have installed it:

    # Open a console and create a directory for the unix sockets.
    $ mktemp -d

    # Then start nc to listen for network traffic using the temporary directory.
    $ \
      SOCKET_WRAPPER_DIR=/tmp/tmp.bQRELqDrhM \

    # nc, listens on because it is specified on the command-line
    # and it corresponds to the SOCKET_WRAPPER_DEFAULT_IFACE value specified

    # Now open another console and start 'nc' as a client to connect to the server:
    $ \
      SOCKET_WRAPPER_DIR=/tmp/tmp.bQRELqDrhM \
      SOCKET_WRAPPER_PCAP_FILE=/tmp/sw.pcap nc -v 7

    # (The client will use the address when connecting to the server)
    # Now you can type 'Hello!' which will be sent to the server and should appear
    # in the console output of the server.
    # When you have finished, you can examine the network packet dump with
    # "wireshark /tmp/sw.pcap"

There are projects that provide daemons needing to be able to create, modify, and delete Unix users. Others just switch user IDs to interact with the system on behalf of another user (e.g. a user space file server). To be able to test these, you need the privilege to modify the passwd and group files. With nss_wrapper it is possible to define your own passwd and group files which will be used by the software while it is under test.

If you have a client and server under test, they normally use functions to resolve network names to addresses (DNS) or vice versa. The nss_wrapper allows you to create a hosts file to set up name resolution for the addresses you use with socket_wrapper.

The user, group, and hosts functionality are all defined as wrappers around the Name Service Switch (NSS) API. The Name Service Switch is a modular system, used by most Unix systems, that allows you to fetch information from several databases (users, groups, hosts, and more) using loadable modules. The list and order of modules is configured in the file /etc/nsswitch.conf. Usually, the nsswitch.conf file contains the "files" module shipped with glibc that looks up users in /etc/passwd, groups in /etc/group, and hosts in /etc/hosts. But it's also possible to define additional sources of information by configuring third party modules — a good example might be looking up users from LDAP using nss_ldap.

Here is an example of using nss_wrapper to handle users and groups:

    $ echo "bob:x:1000:1000:Bob Gecos:/home/test/bob:/bin/false" > passwd
    $ echo "root:x:65534:65532:Root user:/home/test/root:/bin/false" >> passwd
    $ echo "users:x:1000:" > group
    $ echo "root:x:65532:" >> group
    $ NSS_WRAPPER_PASSWD=passwd \
        NSS_WRAPPER_GROUP=group getent passwd bob
    bob:x:1000:1000:Bob Gecos:/home/test/bob:/bin/false
The following shows nss_wrapper faking the host name:
    $ hostname
Here, nss_wrapper simulates host name resolution:
    $ echo "fd00::5357:5faa" > hosts
    $ echo "" >> hosts
    # Now query ahostsv6 which returns only IPv6 addresses and
    # calls getaddrinfo() for each the entry.
      getent ahostsv6
    fd00::5357:5faa DGRAM
    fd00::5357:5faa STREAM

Some projects, such as a file server, need privilege separation to be able to switch to the user who owns the files and do file operations on their behalf. uid_wrapper convincingly lies to the application, letting it believe it is operating as root and even switching between UIDs and GIDs as needed. You can start any application making it believe it is running as root. We will demonstrate this later. You should keep in mind that you will not gain more permissions or privileges with uid_wrapper than you currently have; remember it is The Matrix.

Maybe you know that glibc has support for switching the user/group only for the local thread. For example calling setuid(1000) synchronizes all threads to change to the given UID. The setuid(), setguid(), etc. functions send a signal to each thread, telling it that it should change the relevant ID. The signal handler of each thread for the signal then uses syscall() using the corresponding SYS_setXid constant for the local thread. So, under glibc, if you want to change the UID only for the local thread, you have to make the system call directly:

    rc = syscall(SYS_setruid, 1000, 0);
uid_wrapper has support for glibc's special privilege separation with threads. It intercepts calls to syscall() to handle the remapping of UIDs and GIDs. Here is an example of uid_wrapper in action:
    uid=0(root) gid=0(root) groups=100(users),0(root)

How are the wrappers tested?

You may sense a bit of a conflict of interest with wrappers. On one hand, this article stated that unit tests with wrappers strive to simulate the real world environment as closely as possible. On the other hand, the wrappers substitute such fundamental calls as socket() and getpwnam(). It's paramount that the wrappers be extremely well tested so that you, as a user of the wrappers, are confident that any failure in testing implemented using the wrappers is a bug in the program under test and not an unwanted side effect of the wrappers. To this end, the wrappers include a large unit test suite that make sure the wrappers function as intended. At the time of this writing, the code coverage for wrappers is pretty high: nss_wrapper 79%, socket_wrapper 77%, and uid_wrapper 85% code coverage.

As an example of a unit test, the socket_wrapper implements a very simple echo server. The unit tests that exercise the read() or write() calls then connect to the echo server instance that is seemingly running on a privileged port. In fact, the echo server is run using socket_wrapper, so all communication is redirected over a local socket. You can inspect the unit test in the Samba repository. The CMakeLists.txt file also gives a good overview of how the tests are set up.

The wrappers leverage the cmocka unit testing framework that was covered in an earlier LWN article. In short, the cmocka library provides unit test developers with the ability to use mock objects. Moreover, the cmocka library has very low dependency footprint; in fact, it requires only the standard C library.

All the wrapper libraries are built using the cmake build system. In order to provide cwrap developers with an easy-to-use dashboard that displays the results of unit tests, an instance of the cdash dashboard is running and scheduling tests on several operating systems including several Linux distributions, FreeBSD, and OpenIndiana (descended from OpenSolaris). Currently the i686 and x86_64 architectures are tested. The dashboard is a one-stop view that lets you see if any of the unit tests has trouble or if compiling the wrappers or their unit tests yields any compiler errors or warnings.

Final thoughts

Regular LWN readers may have read about namespaces in Linux. These provide similar functionality as the lightweight virtualization layer mechanism known as containers. But to set up namespaces, you often will need root privileges. When distributions enable user namespaces, that requirement will go away, but there is another problem: namespaces are not available on BSD or Solaris.

Currently Samba is the only user of the cwrap libraries since cwrap was not available for external consumption until recently. Andreas is currently working on cwrap integration to test libssh against an OpenSSH sshd server. We are also planning to improve the test environment of SSSD, but we didn't have time to work on it yet. At Red Hat, Quality Engineering has started to write tests for nss_ldap using nss_wrapper, but they are not upstream yet. If you plan to use cwrap, join us on the #cwrap IRC channel on Freenode.

Comments (1 posted)

Brief items

Quotes of the week

That is, of course, the irony of the patent system. Without patent protection, a competitor can simply replicate an invention and undercut the inventor's price -- which necessarily includes all the time and expense of research and development -- so the incentive to experiment and create is severely inhibited. But if innovators such as Glenn Curtiss cannot build on the progress of others without paying exorbitantly for the privilege, the incentive to continue to experiment and create is similarly inhibited.
Joe Nocera (hat tip to Michael Kerrisk)

Currently it seems that the RAM requirements for Linux desktop use are about 3 years behind the RAM requirements for Windows. This is convenient as a PC is fully depreciated according to the tax office after 3 years. This makes it easy to get 3 year old PCs cheaply (or sometimes for free as rubbish) which work really well for Linux. But it would be nice if we could be 4 or 5 years behind Windows in terms of hardware requirements to reduce the hardware requirements for Linux users even further.
Russell Coker

Comments (9 posted)

QEMU 2.0.0 released

The QEMU team has announced the release of version 2.0.0 of the QEMU "open source machine emulator and virtualizer". New features in the release include support for KVM on AArch64 (64-bit ARM) systems, support for all 64-bit ARMV8 instructions (other than the optional CRC and crypto extensions), support for the Allwinner A10-based cubieboard, CPU hotplug for Q35 x86 systems, better Windows guest performance when doing many floating-point or SIMD operations, live snapshot merging, new management interfaces for CPU and virtio-rng hotplug, direct access to NFSv3 shares using libnfs, and lots more. Detailed information about all of the changes can be found in the changelog.

Comments (3 posted)

ISC releases BIND 10 1.2, renames it, and turns it over to community

Internet Systems Consortium, the non-profit behind the BIND DNS server, has released version 1.2 of BIND 10, which is the last release it will make of the "applications framework for Internet infrastructure, such as DNS". That completes ISC's development effort on BIND 10, so it has renamed the project to Bundy and turned it over to the community for updates and maintenance. "'BIND 10 is an excellent software system,' said Scott Mann, ISC's Vice President of Engineering, 'and a huge step forward in open-source infrastructure software. Unfortunately, we do not have the resources to continue development on both projects, and BIND 9 is much more widely used.' 'The BIND 10 software is open-source,' Scott added, 'so we are making it available for anyone who wants to continue its development. The source will be available from GitHub under the name Bundy, to mitigate the confusion between it and ISC's BIND 9 (a completely separate system). The name 'BIND' is associated with ISC; we have changed its name as a reminder that ISC is no longer involved with the project.'"

Full Story (comments: 28)

GCC 4.9.0 released

Version 4.9.0 of the GNU Compiler Collection is out. "GCC 4.9.0 is a major release containing substantial new functionality not available in GCC 4.8.x or previous GCC releases." The list of new features is indeed long; see the 4.9.0 release page for lots more information.

Full Story (comments: 11)

Linux Test Project released for April 2014

The stable test suite from the Linux Test Project has been updated for April 2014. Notable changes include 20 new syscall test cases, fixes for out-of-tree building and cross-compilation, and the rewrite of several scripts to run in shells other than bash.

Full Story (comments: none)

Newsletters and articles

Development newsletters from the past week

Comments (none posted)

Ars Technica: Tor network’s ranks of relay servers cut because of Heartbleed bug

Ars Technica reports on the impact that the "Heartbleed" bug in OpenSSL has had for the Tor anonymizing network. "The Tor Project team has been moving to provide patches for all of the components, and most of the core network was quickly secured. However, a significant percentage of the relay servers, many of which serve countries with heavy Internet censorship, have remained unpatched. These systems are operated by volunteers and may run unattended."

Comments (6 posted)

Faure: Freedesktop Summit 2014 Report

David Faure has a report on the Freedesktop Summit, which was held recently in Nuremberg. "The meeting also produced an agreement on the future of startup notification in the Wayland world. A protocol based on broadcast of D-Bus signals will be used instead of the current approach with X client messages. This approach is expected to integrate nicely with future frameworks for sandboxed applications. Improvements were also made to the protocol to allow for tab-based applications that make dynamic choices about creating a new tab or a new window depending on the workspace in which a document was opened."

[Editor's note: apologies to Ryan Lortie who wrote this article.]

Comments (3 posted)

Page editor: Nathan Willis
Next page: Announcements>>

Copyright © 2014, Eklektix, Inc.
Comments and public postings are copyrighted by their creators.
Linux is a registered trademark of Linus Torvalds