Compiling, Loading, Packaging, and Testing Drivers

This chapter describes the procedure for driver development, including code layout, compilation, packaging, and testing.

21.1. Driver Development Summary

This chapter and the following two chapters, Debugging, Testing, and Tuning Device Drivers and Recommended Coding Practices, provide detailed information on developing a device driver.

  1. Write, compile, and link the new code.

    See Driver Code Layout for the conventions on naming files. Use a C compiler to compile the driver. Link the driver using ld(1). See Compiling and Linking the Driver and Module Dependencies.

  2. Create the necessary hardware configuration files.

    Create a hardware configuration file unique to the device called xx.conf where xx is the prefix for the device. This file is used to update the driver.conf(4) file. See Writing a Hardware Configuration File. For a pseudo device driver, create a pseudo(4) file.

  3. Copy the driver to the appropriate module directory.

    See Copying the Driver to a Module Directory.

  4. Install the device driver using add_drv(1M).

    Installing the driver with add_drv is usually done as part of a postinstall script. See Installing Drivers with add_drv. Use the update_drv(1M) command to make any changes to the driver. See Updating Driver Information.

  5. Load the driver.

    The driver can be loaded automatically by accessing the device. See Loading and Unloading Drivers and Package Postinstall. Drivers can also be loaded by using the modload(1M) command. The modload command does not call any routines in the module and therefore is useful for testing. See Loading and Unloading Test Modules.

  6. Test the driver.

    Drivers should be rigorously tested in the following areas:

    For additional driver-specific testing, see Testing Specific Types of Drivers.

  7. Remove the driver if necessary.

    Use the rem_drv(1M) command to remove a device driver. See Removing the Driver and Package Preremove.

21.2. Driver Code Layout

21.2.1. Header Files

Some of the header file definitions, such as the state structure, might be needed only by the device driver. This information should go in private header files that are only included by the device driver itself.

Any information that an application might require, such as the I/O control commands, should be in public header files. These files are included by the driver and by any applications that need information about the device.

While there is no standard for naming private and public files, one convention is to name the private header file xximpl.h and the public header file xxio.h.

21.2.2. Source Files

21.2.3. Configuration Files

In general, the configuration file for a driver defines all of the properties that the driver needs. Entries in the driver configuration file specify possible device instances that the driver can probe for existence. Driver global properties can be set in the driver's configuration file. See the driver.conf(4) man page for more information.

Driver configuration files are required for devices that are not self-identifying.

Driver configuration files are optional for self-identifying devices (SID). For self-identifying devices, the configuration file can be used to add properties into SID nodes.

21.3. Preparing for Driver Installation

  1. Compile the driver.

  2. Create a configuration file if necessary.

  3. Identify the driver module to the system through either of the following alternatives:

    • Match the driver's name to the name of the device node.

    • Use either add_drv(1M) or update_drv(1M) to inform the system of the module names.

The system maintains a one-to-one association between the name of the driver module and the name of the dev_info node. For example, consider a dev_info node for a device that is named mydevice. The device mydevice is handled by a driver module that is also named mydevice. The mydevice module resides in a subdirectory that is called drv, which is in the module path. The module is in drv/mydevice if you are using a 32-bit kernel. The module is in drv/sparcv9/mydevice if you are using a 64-bit SPARC kernel. The module is in drv/amd64/mydevice if you are using a 64-bit x86 kernel.

If the driver must manage dev_info nodes with different names, the add_drv(1M) utility can create aliases. The -i flag specifies the names of other dev_info nodes that the driver handles. The update_drv command can also modify aliases for an installed device driver.

21.3.1. Compiling and Linking the Driver

You need to compile each driver source file and link the resulting object files into a driver module. illumos is compatible with both the Sun Studio C compiler and the GNU C compiler from the Free Software Foundation, Inc. The examples in this section use the Sun Studio C compiler unless otherwise noted. For information on the Sun Studio C compiler, see the Sun Studio 12: C User’s Guide and the Sun Studio Documentation on the Sun Developer Network web site. For more information on compile and link options, see the Sun Studio Man Pages. The GNU C compiler is supplied in the /usr/sfw directory. For information on the GNU C compiler, see http://gcc.gnu.org/ or check the man pages in /usr/sfw/man.

The example below shows a driver that is called xx with two C source files. A driver module that is called xx is generated. The driver that is created in this example is for a 32-bit kernel. You must use ld -r even if your driver has only one object module. 

% cc -D_KERNEL -c xx1.c
% cc -D_KERNEL -c xx2.c
% ld -r -o xx xx1.o xx2.o

The _KERNEL symbol must be defined to indicate that this code defines a kernel module. No other symbols should be defined, except for driver private symbols. The DEBUG symbol can be defined to enable any calls to ASSERT(9F).

If you are compiling for a 64-bit SPARC architecture using Sun Studio 9, Sun Studio 10, or Sun Studio 11, use the -xarch=v9 option: 

% cc -D_KERNEL -xarch=v9 -c xx.c

If you are compiling for a 64-bit SPARC architecture using Sun Studio 12, use the -m64 option: 

% cc -D_KERNEL -m64 -c xx.c

If you are compiling for a 64-bit x86 architecture using Sun Studio 10 or Sun Studio 11, use both the -xarch=amd64 option and the -xmodel=kernel option: 

% cc -D_KERNEL -xarch=amd64 -xmodel=kernel -c xx.c

If you are compiling for a 64-bit x86 architecture using Sun Studio 12, use the -m64 option, the -xarch=sse2a option, and the -xmodel=kernel option: 

% cc -D_KERNEL -m64 -xarch=sse2a -xmodel=kernel -c xx.c

Sun Studio 9 does not support 64-bit x86 architectures. Use Sun Studio 10, Sun Studio 11, or Sun Studio 12 to compile and debug drivers for 64-bit x86 architectures.

After the driver is stable, you might want to add optimization flags to build a production quality driver. See the cc(1) man page in Sun Studio Man Pages for specific information on optimizations in the Sun Studio C compiler.

Global variables should be treated as volatile in device drivers. The volatile tag is discussed in greater detail in Declaring a Variable Volatile. Use of the flag depends on the platform. See the man pages.

21.3.2. Module Dependencies

If the driver module depends on symbols exported by another kernel module, the dependency can be specified by the -dy and -N options of the loader, ld(1). If the driver depends on a symbol exported by misc/mySymbol, the example below should be used to create the driver binary. 

% ld -dy -r -o xx xx1.o xx2.o -N misc/mySymbol

21.3.3. Writing a Hardware Configuration File

If a device is non-self-identifying, the kernel might require a hardware configuration file for that device. If the driver is called xx, the hardware configuration file for the driver should be called xx.conf.

On the x86 platform, device information is now supplied by the booting system. Hardware configuration files should no longer be needed, even for non-self-identifying devices.

See the driver.conf(4), pseudo(4), sbus(4), scsi_free_consistent_buf(9F), and update_drv(1M) man pages for more information on hardware configuration files.

Arbitrary properties can be defined in hardware configuration files. Entries in the configuration file are in the form property=value, where property is the property name and value is its initial value. The configuration file approach enables devices to be configured by changing the property values.

21.4. Installing, Updating, and Removing Drivers

Before a driver can be used, the system must be informed that the driver exists. The add_drv(1M) utility must be used to correctly install the device driver. After a driver is installed, that driver can be loaded and unloaded from memory without using the add_drv command.

21.4.1. Copying the Driver to a Module Directory

Device drivers reside in the following locations:

/platform/`uname -i`/kernel/drv

Contains 32-bit drivers that run only on a specific platform.

/platform/`uname -i`/kernel/drv/sparcv9

Contains 64-bit drivers that run only on a specific SPARC-based platform.

/platform/`uname -i`/kernel/drv/amd64

Contains 64-bit drivers that run only on a specific x86-based platform.

/platform/`uname -m`/kernel/drv

Contains 32-bit drivers that run only on a specific family of platforms.

/platform/`uname -m`/kernel/drv/sparcv9

Contains 64-bit drivers that run only on a specific family of SPARC-based platforms.

/platform/`uname -m`/kernel/drv/amd64

Contains 64-bit drivers that run only on a specific family of x86-based platforms.

/usr/kernel/drv

Contains 32-bit drivers that are independent of platforms.

/usr/kernel/drv/sparcv9

Contains 64-bit drivers on SPARC-based systems that are independent of platforms.

/usr/kernel/drv/amd64

Contains 64-bit drivers on x86-based systems that are independent of platforms.

To install a 32-bit driver, the driver and its configuration file must be copied to a drv directory in the module path. For example, to copy a driver to /usr/kernel/drv, type: 

$ su
# cp xx /usr/kernel/drv
# cp xx.conf /usr/kernel/drv

To install a SPARC driver, copy the driver to a drv/sparcv9 directory in the module path. Copy the driver configuration file to the drv directory in the module path. For example, to copy a driver to /usr/kernel/drv, you would type: 

$ su
# cp xx /usr/kernel/drv/sparcv9
# cp xx.conf /usr/kernel/drv

To install a 64-bit x86 driver, copy the driver to a drv/amd64 directory in the module path. Copy the driver configuration file to the drv directory in the module path. For example, to copy a driver to /usr/kernel/drv, you would type: 

$ su
# cp xx /usr/kernel/drv/amd64
# cp xx.conf /usr/kernel/drv

All driver configuration files (.conf files) must go in the drv directory in the module path. The .conf files cannot go into any subdirectory of the drv directory.

21.4.2. Installing Drivers with add_drv

Use the add_drv(1M) command to install the driver in the system. If the driver installs successfully, add_drv runs devfsadm(1M) to create the logical names in the /dev directory. 

# add_drv xx

In this case, the device identifies itself as xx. The device special files have default ownership and permissions (0600 root sys). The add_drv command also allows additional names for the device (aliases) to be specified. See the add_drv(1M) man page for information on adding aliases and setting file permissions explicitly.

Do not use the add_drv command to install a STREAMS module. See the STREAMS Programming Guide for details.

If the driver creates minor nodes that do not represent terminal devices such as disks, tapes, or ports, you can modify /etc/devlink.tab to cause devfsadm to create logical device names in /dev. Alternatively, logical names can be created by a program that is run at driver installation time.

21.4.3. Updating Driver Information

Use the update_drv(1M) command to notify the system of any changes to an installed device driver. By default, the system re-reads the driver configuration file and reloads the driver binary module.

21.4.4. Removing the Driver

To remove a driver from the system, use the rem_drv(1M) command, and then delete the driver module and configuration file from the module path. A driver cannot be used again until that driver is reinstalled with add_drv(1M). The removal of a SCSI HBA driver requires a reboot to take effect.

21.5. Loading and Unloading Drivers

Opening a special file (accessing the device) that is associated with a device driver causes that driver to be loaded. You can use the modload(1M) command to load the driver into memory, but modload does not call any routines in the module. The preferred method is to open the device.

Normally, the system automatically unloads device drivers when the device is no longer in use. During development, you might want to use modunload(1M) to unload the driver explicitly. In order for modunload to be successful, the device driver must be inactive. No outstanding references to the device should exist, such as through open(2) or mmap(2).

The modunload command takes a runtime-dependent module_id as an argument. To find the module_id, use grep to search the output of modinfo(1M) for the driver name in question. Check in the first column. 

# modunload -i module-id

To unload all currently unloadable modules, specify module ID zero:

# modunload -i 0

In addition to being inactive, the driver must have working detach(9E) and _fini(9E) routines for modunload(1M) to succeed.

21.6. Driver Packaging

The normal delivery vehicle for software is to create a package that contains all of the software components. A package provides a controlled mechanism for installation and removal of all the components of a software product. In addition to the files for using the product, the package includes control files for installing and uninstalling the application. The postinstall and preremove installation scripts are two such control files.

21.6.1. Package Postinstall

After a package with a driver binary is installed onto a system, the add_drv(1M) command must be run. The add_drv command completes the installation of the driver. Typically, add_drv is run in a postinstall script, as in the following example. 

#!/bin/sh
#
#       @(#)postinstall 1.1

PATH="/usr/bin:/usr/sbin:${PATH}"
export PATH

#
# Driver info
#
DRV=<driver-name>
DRVALIAS="<company-name>,<driver-name>"
DRVPERM='* 0666 root sys'

ADD_DRV=/usr/sbin/add_drv

#
# Select the correct add_drv options to execute.
# add_drv touches /reconfigure to cause the
# next boot to be a reconfigure boot.
#
if [ "${BASEDIR}" = "/" ]; then
    #
    # On a running system, modify the
    # system files and attach the driver
    #
    ADD_DRV_FLAGS=""
else     
    #
    # On a client, modify the system files
    # relative to BASEDIR
    #
    ADD_DRV_FLAGS="-b ${BASEDIR}"
fi       
 
#
# Make sure add_drv has not been previously executed
# before attempting to add the driver.
#
grep "^${DRV} " $BASEDIR/etc/name_to_major > /dev/null 2>&1
if [ $? -ne 0 ]; then
    ${ADD_DRV} ${ADD_DRV_FLAGS} -m "${DRVPERM}" -i "${DRVALIAS}" ${DRV}
    if [ $? -ne 0 ]; then
        echo "postinstall: add_drv $DRV failed\n" >&2
        exit 1
    fi
fi
exit 0

21.6.2. Package Preremove

When removing a package that includes a driver, the rem_drv(1M) command must be run prior to removing the driver binary and other components. The following example demonstrates a preremove script that uses the rem_drv command for driver removal. 

#!/bin/sh
#
#       @(#)preremove  1.1
 
PATH="/usr/bin:/usr/sbin:${PATH}"
export PATH
 
#
# Driver info
#
DRV=<driver-name>
REM_DRV=/usr/sbin/rem_drv
 
#
# Select the correct rem_drv options to execute.
# rem_drv touches /reconfigure to cause the
# next boot to be a reconfigure boot.
#
if [ "${BASEDIR}" = "/" ]; then
    #
    # On a running system, modify the
    # system files and remove the driver
    #
    REM_DRV_FLAGS=""
else     
    #
    # On a client, modify the system files
    # relative to BASEDIR
    #
    REM_DRV_FLAGS="-b ${BASEDIR}"
fi
 
${REM_DRV} ${REM_DRV_FLAGS} ${DRV}
 
exit 0

21.7. Criteria for Testing Drivers

Once a device driver is functional, that driver should be thoroughly tested prior to distribution. Besides testing the features in traditional UNIX device drivers, illumos drivers require testing power management features, such as dynamic loading and unloading of drivers.

21.7.1. Configuration Testing

A driver's ability to handle multiple device configurations is an important part of the test process. Once the driver is working on a simple, or default, configuration, additional configurations should be tested. Depending on the device, configuration testing can be accomplished by changing jumpers or DIP switches. If the number of possible configurations is small, all configurations should be tried. If the number is large, various classes of possible configurations should be defined, and a sampling of configurations from each class should be tested. Defining these classes depends on the potential interactions among the different configuration parameters. These interactions are a function of the type of the device and the way in which the driver was written.

For each device configuration, the basic functions must be tested, which include loading, opening, reading, writing, closing, and unloading the driver. Any function that depends upon the configuration deserves special attention. For example, changing the base memory address of device registers is not likely to affect the behavior of most driver functions. If a driver works well with one address, that driver is likely to work as well with a different address. On the other hand, a special I/O control call might have different effects depending on the particular device configuration.

Loading the driver with varying configurations ensures that the probe(9E) and attach(9E) entry points can find the device at different addresses. For basic functional testing, using regular UNIX commands such as cat(1) or dd(1M) is usually sufficient for character devices. Mounting or booting might be required for block devices.

21.7.2. Functionality Testing

After a driver has been completely tested for configuration, all of the driver's functionality should be thoroughly tested. These tests require exercising the operation of all of the driver's entry points.

Many drivers require custom applications to test functionality. However, basic drivers for devices such as disks, tapes, or asynchronous boards can be tested using standard system utilities. All entry points should be tested in this process, including devmap(9E), chpoll(9E), and ioctl(9E), if applicable. The ioctl tests might be quite different for each driver. For nonstandard devices, a custom testing application is generally required.

21.7.3. Error Handling

A driver might perform correctly in an ideal environment but fail in cases of errors, such as erroneous operations or bad data. Therefore, an important part of driver testing is the testing of the driver's error handling.

All possible error conditions of a driver should be exercised, including error conditions for actual hardware malfunctions. Some hardware error conditions might be difficult to induce, but an effort should be made to force or to simulate such errors if possible. All of these conditions could be encountered in the field. Cables should be removed or be loosened, boards should be removed, and erroneous user application code should be written to test those error paths. See also Hardening illumos Drivers.

Be sure to take proper electrical precautions when testing.

21.7.4. Testing Loading and Unloading

Because a driver that does not load or unload can force unscheduled downtime, loading and unloading must be thoroughly tested.

A script like the following example should suffice:

#!/bin/sh
cd <location_of_driver>
while [ 1 ]
do
    modunload -i 'modinfo | grep " <driver_name> " | cut -cl-3' &
    modload <driver_name> &
done

21.7.5. Stress, Performance, and Interoperability Testing

To help ensure that a driver performs well, that driver should be subjected to vigorous stress testing. For example, running single threads through a driver does not test locking logic or conditional variables that have to wait. Device operations should be performed by multiple processes at once to cause several threads to execute the same code simultaneously.

Techniques for performing simultaneous tests depend upon the driver. Some drivers require special testing applications, while starting several UNIX commands in the background is suitable for others. Appropriate testing depends upon where the particular driver uses locks and condition variables. Testing a driver on a multiprocessor machine is more likely to expose problems than testing on a single-processor machine.

Interoperability between drivers must also be tested, particularly because different devices can share interrupt levels. If possible, configure another device at the same interrupt level as the one being tested. A stress test can determine whether the driver correctly claims its own interrupts and operates according to expectations. Stress tests should be run on both devices at once. Even if the devices do not share an interrupt level, this test can still be valuable. For example, consider a case in which serial communication devices experience errors when a network driver is tested. The same problem might be causing the rest of the system to encounter interrupt latency problems as well.

Driver performance under these stress tests should be measured using UNIX performance-measuring tools. This type of testing can be as simple as using the time(1) command along with commands to be used in the stress tests.

21.7.6. DDI/DKI Compliance Testing

To ensure compatibility with later releases and reliable support for the current release, every driver should be DDI/DKI compliant. Check that only kernel routines in manual pages section 9F: DDI and DKI Kernel Functions and manual pages section 9E: DDI and DKI Driver Entry Points and data structures in manual pages section 9S: DDI and DKI Properties and Data Structures are used.

21.7.7. Installation and Packaging Testing

Drivers are delivered to customers in packages. A package can be added or be removed from the system using a standard mechanism (see the Application Packaging Developer’s Guide).

The ability of a user to add or remove the package from a system should be tested. In testing, the package should be both installed and removed from every type of media to be used for the release. This testing should include several system configurations. Packages must not make unwarranted assumptions about the directory environment of the target system. Certain valid assumptions, however, can be made about where standard kernel files are kept. Also test adding and removing of packages on newly installed machines that have not been modified for a development environment. A common packaging error is for a package to rely on a tool or file that is used in development only. For example, no tools from the Source Compatibility package, SUNWscpu, should be used in driver installation programs.

The driver installation must be tested on a minimal illumos system without any optional packages.

21.7.8. Testing Specific Types of Drivers

This section provides some suggestions about how to test certain types of standard devices.

Tape Drivers

Tape drivers should be tested by performing several archive and restore operations. The cpio(1) and tar(1) commands can be used for this purpose. Use the dd(1M) command to write an entire disk partition to tape. Next, read back the data, and write the data to another partition of the same size. Then compare the two copies. The mt(1) command can exercise most of the I/O controls that are specific to tape drivers. See the mtio(7I) man page. Try to use all the options. These three techniques can test the error-handling capabilities of tape drivers:

Tape drivers typically implement exclusive-access open(9E) calls. These open calls can be tested by opening a device and then having a second process try to open the same device.

Disk Drivers

Disk drivers should be tested in both the raw and block device modes. For block device tests, create a new file system on the device. Then try to mount the new file system. Then try to perform multiple file operations.

The file system uses a page cache, so reading the same file over and over again does not really exercise the driver. The page cache can be forced to retrieve data from the device by memory-mapping the file with mmap(2). Then use msync(3C) to invalidate the in-memory copies.

Copy another (unmounted) partition of the same size to the raw device. Then use a command such as fsck(1M) to verify the correctness of the copy. The new partition can also be mounted and then later compared to the old partition on a file-by-file basis.

Asynchronous Communication Drivers

Asynchronous drivers can be tested at the basic level by setting up a login line to the serial ports. A good test is to see whether a user can log in on this line. To sufficiently test an asynchronous driver, however, all the I/O control functions must be tested, with many interrupts at high speed. A test involving a loopback serial cable and high data transfer rates can help determine the reliability of the driver. You can run uucp(1C) over the line to provide some exercise. However, because uucp performs its own error handling, verify that the driver is not reporting excessive numbers of errors to the uucp process.

These types of devices are usually STREAMS-based. See the STREAMS Programming Guide for more information.

Network Drivers

Network drivers can be tested using standard network utilities. The ftp(1) and rcp(1) commands are useful because the files can be compared on each end of the network. The driver should be tested under heavy network loading, so that various commands can be run by multiple processes.

Network cables should be unplugged while the tests are executing to ensure that the driver recovers gracefully from the resulting error conditions. Another important test is for the driver to receive multiple packets in rapid succession, that is, back-to-back packets. In this case, a relatively fast host on a lightly loaded network should send multiple packets in quick succession to the test machine. Verify that the receiving driver does not drop the second and subsequent packets.

These types of devices are usually STREAMS-based. See the STREAMS Programming Guide for more information.