Writing Linux Test Project test cases

The Linux Test Project (LTP) is an automated kernel test suite designed "to validate the reliability, robustness, and stability of Linux". An earlier article introduced LTP and explained how to run the test cases. This article will focus on how to write an LTP test case.

The main advantage of writing an LTP test case is the availability of the test library. That library contains some useful code patterns that recur in test development. For example, it simplifies test setup significantly by providing "safe" macros that are wrappers for most of the POSIX interfaces. These wrappers abort the test safely if the call to the particular function fails. It's also place where test-suite-wide parameters are defined, environment variables are parsed, and so on.

The library is, on the other hand, designed to be optional. It is entirely possible to create an LTP-compatible test case just by returning right exit status at the end of the test case. Although, for new test cases, use of the test library will ease the development and is generally preferred.

An example test case

Let's dive into a real world test example first. The test we will look at is a simple test for mount() errors.

The test starts with a short description explaining what the test does. Each test case is stored in an array of structures; each entry holds parameters to pass to the mount() call and also the expected result. For this test case, these are all failures with the specified errno value. Although this is not always the case, this pattern is common for simple system call test cases.

The overall test setup is done in the setup() function and overall cleanup in the cleanup() function. The setup() function is called only once at the start of the main(), but it's an unwritten convention for it to be a separate function. On the other hand, the cleanup() function may be called from various parts of the test, even from setup(), and therefore should be able to handle unfinished test initialization as well. The final remaining piece of this big picture overview is the verify_mount() function that takes a pointer to a test case and runs the actual test.

If you are wondering why the test does not test conditions where mount() succeeds, the answer is simple. Keeping both negative and positive test cases in one source file would complicate the design, because the steps to check the expected result are completely different. The positive test cases are implemented in mount01.c and mount03.c.

Closer look at the code

The main header for the test API is include/test.h. Since the LTP build system passes the correct path to the include directory, the file is simply included as test.h. There is also usctest.h that contains a few more macros, but only the TEST() macro (described below) is widely used. That header file is last remnant from earlier times that has not been touched in the large-scale cleanups; it will likely be removed in the future.

Each test must define two global variables. The TCID variable is a string containing the test identifier, which is usually the same as the test's filename without the file extension. The TST_TOTAL variable stores the total number of test cases in the test. The test should report the corresponding number of passes and fails unless the execution was interrupted prematurely. These variables are used by the test library for printing messages. In addition, the TCID is used for the filename template when the test temporary file is created.

The first thing the test does after entering main() is the call to the library function parse_opts() to parse the standard test options. These options can be used, for example, to run the test N times or to run it for T seconds. If a test needs test-specific options, they can be passed to the function as well by passing a structure defining the parameters and a pointer to function to print help information. The TEST_LOOPING() macro is used in the test main loop. It uses the information gathered by parse_opts() and evaluates to true while the test case should continue to execute.

Test results are reported by printf()-like functions that take a few extra parameters. The example test uses tst_resm() (to report results without exiting the test) and tst_brkm() (to exit the test), which are the most commonly used. The main advantage of these functions is that the overall test result is stored internally by the test library, so there is no need to propagate it manually in the code.

The result types are bit flags and the result is a bit field. The possible values are TPASS, TFAIL, TBROK, TWARN, and TCONF. TPASS and TFAIL are self-explanatory. TBROK means that something unexpected happened in the test setup and test was aborted. TWARN means that something unexpected happened but the test carried on. TCONF means that test is not suitable for the current configuration. The overall test exit value is composed of these bit flags. Note that TPASS is actually defined as zero so that the return value for successful test is also zero.

A subset of the printf()-like functions will also exit the test execution immediately. In the example code, the tst_brkm() call will exit. Such functions also include a cleanup callback parameter that, if it is not NULL, is called before the test exits. The cleanup callback is usually a pointer to the overall test cleanup function that is also called at the end of the test. It frees the resources claimed by the test case, which are usually a temporary directory, open file descriptors, loopback devices, and so on.

There are also several additional features these functions bring to the table. For example, all non-success messages include the filename and line number automatically in order to easily track back to the proper location in the test source code.

The tst_sig() library function is used in setup() to install "poisoned" signal handlers that end test case execution when an unexpected signal has arrived. The tst_require_root() library function will exit the test unless the process runs as root (has EUID == 0).

LTP contains two library functions to help with temporary directory creation and deletion. The first is tst_tmpdir() that creates a unique test temporary directory under $TMPDIR and also changes the current working directory to it. The companion function, tst_rmdir(), is called from test cleanup to delete the directory recursively, so there is no need to remove individual temporary files and directories.

Once the temporary directory is created, the test can proceed with creating test files and directories. The so-called safe macros (e.g. SAFE_OPEN(), SAFE_MKDIR()) implement wrappers for most of the POSIX interfaces and for a few more common tasks. These macros will return to the caller only if they were successful. Similar to tst_brkm(), the macros take a cleanup callback parameter that will be called before the test exits in case of failure.

If the call fails, the safe macros produce an error message with as much relevant information as possible. That message will include the source filename and line, the parameters passed to the call in a human readable form, errno if applicable, and so forth.

All safe macros are defined in include/safe_macros.h header located under the LTP source tree. There are also some safe file operations defined in include/safe_file_ops.h. These operations include SAFE_FILE_SCANF() and SAFE_FILE_PRINTF() that are especially useful for reading/writing values from various filesystems such as procfs and sysfs.

Our example test needs to work with a block device, so it makes use of three library functions designed for that purpose. The first two, tst_dev_fs_type() and tst_acquire_device(), return the filesystem type to be used for the testing and a path to the device to be used for the testing. If no device was passed to the top-level test script, the test library will prepare a suitable loop device. The call to tst_mkfs() will call mkfs to format the device with a filesystem. It also handles extra parameters that are needed for certain filesystem types.

Now that we have described the setup process, let's get back to main() and have a look at the test main loop. Apart from previously explained TEST_LOOPING() macro, what the test does is loop over the structure that describes all of the test cases and call verify_mount() to actually do the test for each of them. The TEST() macro actually makes the mount() call. It is just shorthand for:

    errno = 0;
    TEST_RETURN = call();
    TEST_ERRNO = errno;

Output from the example test case:

    mount02     0  TINFO  :  Found free device '/dev/loop0'
    mount02     0  TINFO  :  Formatting /dev/loop0 with ext2 extra opts=''
    mke2fs 1.42.10 (18-May-2014)
    mount02     1  TPASS  :  mount() failed expectedly: TEST_ERRNO=ENODEV(19): No such device
    mount02     2  TPASS  :  mount() failed expectedly: TEST_ERRNO=ENOTBLK(15): Block device required
    mount02     3  TPASS  :  mount() failed expectedly: TEST_ERRNO=EBUSY(16): Device or resource busy
    mount02     4  TPASS  :  mount() failed expectedly: TEST_ERRNO=EBUSY(16): Device or resource busy
    mount02     5  TPASS  :  mount() failed expectedly: TEST_ERRNO=EINVAL(22): Invalid argument
    mount02     6  TPASS  :  mount() failed expectedly: TEST_ERRNO=EINVAL(22): Invalid argument
    mount02     7  TPASS  :  mount() failed expectedly: TEST_ERRNO=EINVAL(22): Invalid argument
    mount02     8  TPASS  :  mount() failed expectedly: TEST_ERRNO=EFAULT(14): Bad address
    mount02     9  TPASS  :  mount() failed expectedly: TEST_ERRNO=EFAULT(14): Bad address
    mount02    10  TPASS  :  mount() failed expectedly: TEST_ERRNO=ENAMETOOLONG(36): File name too long
    mount02    11  TPASS  :  mount() failed expectedly: TEST_ERRNO=ENOENT(2): No such file or directory
    mount02    12  TPASS  :  mount() failed expectedly: TEST_ERRNO=ENOTDIR(20): Not a directory

Example output on a kernel without support for loop devices:

    mount02     0  TINFO  :  Couldn't find free loop device
    mount02     1  TCONF  :  mount02.c:195: Failed to obtain block device
    mount02     2  TCONF  :  mount02.c:195: Remaining cases not appropriate for configuration

Parent-child synchronization, loop devices, and further reading

The LTP test library contains lots more functionality to cover less common, but still repeatedly occurring, patterns. For example, if the test is run from a child process, LTP has tst_record_childstatus() function that waits for the child to exit and applies its exit status to the parent's test results. In addition, to ease parent-child synchronization, LTP implements FIFO-based synchronization primitives. Test cases that involve several threads may fail horribly if the cleanup callback is entered from several threads at once. To avoid this, LTP has TST_DECLARE_ONCE_FN() macro to create a thread-safe cleanup callback.

The library also includes runtime detection for kernel version, filesystem type, filesystem free space, and more. LTP build system and library can also be used to build and load kernel modules. Some test cases can be written as shell programs by using the reporting functions and parts of the library API that are implemented in the shell library.

For comprehensive API documentation, consult the Test Writing Guidelines.

How to port test cases to LTP

Porting existing test cases to LTP is a pretty straightforward process. First of all, test cases must be split into a separate executables (one executable per assertion or group of similar assertions) that can be executed without any manual intervention. Given the size of the project, it is important that all test source files have a unique name. Ideally, all files that are part of a certain test suite should start with common prefix.

The test results must be propagated to the exit value in the expected format. The LTP test exit value is a bit field defined in include/tst_res_flags.h. Using LTP test-reporting functions is preferred, but not strictly required.

The last step is to create a runtest entry that tells the LTP test execution framework which binaries should be executed in a test run. Runtest files are stored under runtest/ directory and their names are (hopefully) self-explanatory. If none of the existing runtest files seems right, new files can be created (set of runtest files used for a default run is stored in scenarios_groups/default file). The runtest file format is simple, all characters up to the first white space are the unique test name that appears in the test logs; the rest of the line is a command line to be executed.

    mount02 mount02

Help wanted

The easiest way to start contributing is to run LTP and look at the results. If there are failed test cases, report them on the mailing list—or, even better, send a patch. Review of test cases, especially complex ones in the area of your expertise, is always welcomed as well.

The size of the interface between the kernel and user space grows faster than the number of test cases, so more tests are always needed. To see what is missing, just compare the list in man 2 syscalls with the content of the testcases/kernel/syscalls directory. Off the top of my head, a few that are missing are open_by_handle_at(), memfd_create(), getrandom(), as well as additional system call flags such as O_TMPFILE and O_BENEATH.

Writing a functional test case is as easy as calling the system call and checking that the result matches the documentation. Then there are numerous kernel interfaces that are not covered at all, for example the kernel input subsystem, where quite a lot could be tested using the uinput interface.

Conclusion

I hope that readers now see how the LTP test library can simplify the job of writing automated test cases. Questions, suggestions, patches, code that can be turned into automated test cases, etc. can be directed to the project's mailing list.