Source Tree Layout and Makefile System

Introduction

This section will first take you on a tour of the source tree. After that, it will proceed to discuss several of the aspects surrounding the organization of our makefiles and where object files show up.

Source Tree Tour

All sources are found under usr/src. This includes both the sources used to build the ON consolidation and sources for tools and other peripheral utilities needed to build but not shipped as part of illumos. Because it includes only ON sources, it does not contain Java, the windowing system, or packaging and installation tools. Because of contractual obligations, it may not include all code from third-party hardware vendors. The usr/src directory has several subdirectories which are described here.

  • cmd

This directory contains sources for the executable programs and scripts that are part of ON. It includes all the basic commands, daemons, startup scripts, and related data. Most subdirectories are named for the command or commands they provide; however, there are some exceptions listed here.

  • cmd/Adm

Miscellaneous key system files, such as crontabs and data installed in /etc.

  • cmd/cmd-crypto

Basic cryptographic utilities, such as elfsign and digest.

  • cmd/cmd-inet

Network commands and daemons, including the Berkeley r-commands, PPP, telnet, the inetd super-server, and other network-related utilities.

  • cmd/fs.d

Utilities for checking, mounting, unmounting, and analyzing filesystems.

  • cmd/mdb

Contains the sources for the modular debugger, mdb(1), the kernel debugger, kmdb(1), and the majority of the modules.

  • cmd/netfiles

IP port definitions and name service switch configuration files installed into /etc.

  • cmd/ptools

Utilities for manipulating and observing processes; these are based on proc(4) and libproc interfaces.

  • cmd/sgs

Software Generation System. This directory contains binary utilities, such as ld(1), ar(1), and mcs(1), and development tools such as lex(1), yacc(1), and m4(1). Note that this directory also includes several libraries and headers used by these tools.

  • common

Files which are common among cmd, lib, stand, and uts. These typically include headers and sources to basic libraries used by both the kernel and user programs.

  • head

Userland header files (kernel headers are in uts/). Note that only libc headers should be stored here; other libraries should have their headers in their own subdirectories under lib/.

  • lib

Libraries. Most subdirectories are named for the library whose sources they contain or are otherwise self-explanatory.

  • pkg

Contains the manifests and package lists for building ips packages from the ON sources.

  • prototypes

Sample files showing format and copyright notices.

  • psm

Platform-specific modules. Currently this contains only OBP and most of the boot code.

  • stand

Standalone environment code. This is used for booting; for example, code for reading from UFS and the network is here.

  • tools

Development tools and sources. See README.tools for more information about each tool; the file should be updated as tools are added or removed.

  • ucbcmd

Commands and daemons installed into /usr/ucb (for SunOS 4.x compatibility).

  • ucbhead

Header files installed into /usr/ucb (for SunOS 4.x compatibility).

  • ucblib

Libraries installed into /usr/ucb (for SunOS 4.x compatibility).

  • uts

Kernel sources are here (UTS == UNIX Time Sharing). There are numerous subdirectories of uts which are of interest:

  • uts/common

All platform-independent kernel sources. Nearly all of the illumos kernel is here; only a few small parts are architecture-dependent.

  • uts/common/conf

System configuration parameters.

  • uts/common/contract

Code to support process contracts. See contract(4) and libcontract(3LIB) for more information on process contracts.

  • uts/common/cpr

CheckPoint-and-Resume support. This implements suspend and resume functionality.

  • uts/common/crypto

Kernel cryptographic framework. See cryptoadm(1M) for more information.

  • uts/common/ctf

Kernel code for handling Compact C Type Format data.

  • uts/common/des

Implements the old Data Encryption Standard. This is used by KCF.

  • uts/common/disp

Dispatcher, thread handling, and scheduling classes.

  • uts/common/dtrace

CPU-independent dtrace(7D) kernel support.

  • uts/common/exec

Code for handling userland binary executable types (a.out, ELF, etc).

  • uts/common/fs

Filesystems.

  • uts/common/inet

IP networking subsystem, including IPv6.

  • uts/common/io

I/O subsystem. Most of the code in this directory is device drivers (and pseudo-device drivers).

  • uts/common/ipp

IP policy framework; includes QoS and other traffic management.

  • uts/common/kmdb

Kernel modular debugger driver. See kmdb(1).

  • uts/common/krtld

Kernel runtime linker/loader. This is responsible for handling loadable modules and symbol resolution; it is analogous to ld.so.1, and shares code with it.

  • uts/common/ktli

Kernel TLI (Transport Layer Interface).

  • uts/common/net

Header files; most are shipped in /usr/include/net.

  • uts/common/netinet

Header files; most are shipped in /usr/include/netinet.

  • uts/common/nfs

Network File System headers shipped in /usr/include/nfs.

  • uts/common/os

Core operating system implementation. This includes such varied aspects as privileges, zones, timers, the DDI/DKI interfaces, and high-level locking mechanisms.

  • uts/common/pcmcia

PCMCIA I/O subsystem and drivers.

  • uts/common/rpc

Remote Procedure Call subsystem used by services such as NFS and NIS.

  • uts/common/rpcsvc

Generated RPC header files shipped in /usr/include/rpcsvc.

  • uts/common/sys

Header files shipped in /usr/include/sys. These same headers are used to build the kernel as well (if the _KERNEL preprocessor symbol is defined).

  • uts/common/syscall

System call implementations. Most syscalls are implemented in files matching their names. Note that some system calls are implemented in os/ or other subdirectories instead.

  • uts/common/vm

Virtual memory subsystem.

  • uts/common/zmod

Compression/decompression library.

  • uts/i86pc

Architecture-dependent files for x86 machines. The architecture-dependent directories (i86pc, sun, sun4, sun4u) all have a set of subdirectories similar to common/ above.

  • uts/intel

ISA-dependent, architecture-independent files for x86 machines. Note that all architecture-independent source files are built into objects in this hierarchy.

  • uts/sfmmu

Code specific to the SpitFire memory management unit (UltraSPARC).

  • uts/sparc

ISA-dependent, architecture-independent files for SPARC machines. Note that all architecture-independent source files are built into objects in this hierarchy.

  • uts/sun

Sources common to all Sun implementations. Currently this contains a small number of device drivers and some headers shipped in /usr/include/sys.

  • uts/sun4

Sources common to all sun4* machine architectures.

  • uts/sun4u

Architecture-dependent sources for the sun4u architecture. Each system implementation has a subdirectory here.

Exception Lists

You may note that we started inside of usr/src. There is just one other component which is outside of usr/src, the directory exception_lists. The exception_lists contain several files that relate to exceptions to the standard checks that we make. This covers things like packaging and cstyle. Several of these exceptions exist for historical reasons. If you find yourself making additions to these files, you should think twice about what you're doing.

Object File Layouts

There are two basic strategies that can be used in the creation of object files and finished binaries (executables and libraries):

(a) place objects in a dedicated directory hierarchy parallel to the sources

(b) place objects in the same directories as the sources they are built from

ON actually uses each of these approaches in different parts of the tree. Strategy (a) must be used for all kernel code and many libraries, is preferred for new code, and will be described in detail here. There are several legacy variations on strategy (a) as well as instances in which strategy (b) is still used; the majority of these are in the cmd hierarchy. The entire uts hierarchy has been converted to strategy (a) as described below, and you will see this same approach used throughout much of the rest of the tree.

General Strategy for Kernel Objects

First, each platform-independent module has zero or one build directory per architecture. An architecture in this case can be a machine (sun4u, i86pc) or a processor architecture (intel, sparc). The path to this location is always usr/src/uts/<platform>/<module>. The module name in this case is what's found in /kernel/drv or a similar location, and in the case of device drivers or STREAMS modules should always match the name of the man page describing that driver or module.

The only files normally present in this directory for a clean tree are makefiles. After a build, these directories contain one or more of obj32, obj64, debug32, and debug64 directories. These directories contain the object files and finally linked modules that are later installed into the kernel directory and other locations in the prototype.

'Implementation architecture'-independent modules are produced in individual directories (one per module) under the 'instruction-set architecture' directory (i.e.: sparc). Similarly, 'implementation architecture'-dependent modules are produced in individual directories under the 'implementation architecture' directory (i.e.: sun4, sun4u, i86pc).

Platform-dependent modules (including 'unix') may be built more than once in different locations. For example, they'll be built under each of the different implementations of a platform. This is more common on sparc than x86.

The sources are not contained in these build directories.

General Strategy for Command/Library Objects

Most libraries and some commands and daemons are built using makefiles very similar to those used to build the kernel. Accordingly, intermediate objects, shared objects, libraries, and executables are built by small makefile fragments and placed in dedicated ISA-specific subdirectories.

Other commands' build systems place objects in the same directories as the sources. See Component Anatomy, Creation, and Modification for more information on how commands are built.

Exceptions in cmd and lib

Most of the cmd tree is directly based on the original System V Release 4 source, which uses strategy (b) described above. Since most commands and daemons do not need to provide both 32- and 64-bit versions, or do anything special when building for different architectures, this strategy is adequate and appropriate for most commands and has been applied even to new subdirectories. In situations in which architecture-dependent build options are needed or multiple ISA versions of a program must be delivered, this strategy is unworkable and the more general approach of multiple per-ISA object directories must be used instead. This latter approach is similar to the approach used for kernel modules.

The lib hierarchy is somewhat simpler; nearly all subdirectories must use per-ISA object file locations and makefiles.

A few directories do not appear to follow any rule or pattern, such as cmd/cmd-inet and cmd/agents. These are primarily historical artifacts of Sun's internal project organization.

Makefile Layout

This discussion is intended to provide a step-by-step explanation of what targets exist and how they are built by the makefiles. I ignore the platform-specific module architecture because it is unlikely to be of significant interest except to hardware support engineers. The three main subtrees of interest are the kernel (uts), commands and daemons (cmd), and libraries (lib). The next three subsections cover these three subtrees in turn. There are also a handful of makefiles which apply to all builds:

  • usr/src/Makefile

This is the top-level makefile. It drives builds for various targets in each subdirectory. It is aware of the specific targets that need to be built in each subdirectory in order to perform a complete build, and itself knows how to create a skeleton proto area for later use by install and install_h targets.

  • usr/src/Makefile.lint

All linting from the top level is driven by this makefile. It contains long lists of directories known to be lint-clean and contains simple recursive rules for rebuilding each subdirectory's lint target. The actual linting is driven by the lower-level makefiles.

  • usr/src/Makefile.master

  • usr/src/Makefile.master.64

These two makefiles contain generic definitions, such as build and installation tools locations, template macros for compilers, linkers, and other tools to be used by other makefiles in defining rules, and global definitions such as the ISA and machine names that apply to this build. Makefile.master.64 contains definitions specific to 64-bit builds that override the generic definitions.

  • usr/src/Makefile.msg.targ

Common targets for building message catalogues are defined here. Message catalogues provide translations of messages for g11n purposes.

  • usr/src/Makefile.psm

This makefile defines the installation locations for platform-specific modules. These are analogous to the other kernel module install locations /kernel and /usr/kernel.

  • usr/src/Makefile.psm.targ

Installation target definitions for platform-specific modules are defined here. This instructs the build system how to install files into the directories defined by Makefile.psm.

  • usr/src/Targetdirs

This is a set of definitions for the owner, group, and permissions of each directory that will be created by the installation process. It also contains information about special symbolic links to be installed for some 64-bit library versions.

Kernel Makefile Layout

The driving makefile for any module is located in the leaf directory (build directory) where the module and its component objects are built. After a make clobber operation, the makefile should be the only file remaining in that directory. There are two other types of makefiles in the tree: suffixed and non-suffixed. Common definitions and rules needed by all leaf makefiles are contained in the suffixed makefiles; these are included by leaf makefiles. Non-suffixed makefiles generally invoke multiple lower-level makefiles with the same target so that many modules can be built with a single make invocation.

  • uts/Makefile

  • uts/sparc/Makefile

  • uts/sun4u/Makefile

  • uts/intel/Makefile

  • uts/intel/ia32/Makefile

  • uts/i86pc/Makefile

These makefiles generally are cognizant of the components made in subdirectories and invoke makefiles in those sub- directories to perform the actual build. Some targets (or pseudo-targets) may be directly built at this level (such as the cscope databases).

  • uts/Makefile.uts

Contains common definitions for all possible architectures.

  • uts/Makefile.targ

Contains common targets for all possible architectures.

  • uts/common/Makefile.files

  • uts/sun/Makefile.files

  • uts/sparc/Makefile.files

  • uts/sun4/Makefile.files

  • uts/sun4u/Makefile.files

  • uts/intel/Makefile.files

  • uts/intel/ia32/Makefile.files

  • uts/i86pc/Makefile.files

These makefiles are divided into two sections. The first section defines the object lists which comprise each module. The second section defines the appropriate header search paths and other machine-specific global build parameters.

  • uts/common/Makefile.rules

  • uts/sun/Makefile.rules

  • uts/sparc/Makefile.rules

  • uts/sun4/Makefile.rules

  • uts/sun4u/Makefile.rules

  • uts/intel/Makefile.rules

  • uts/intel/ia32/Makefile.rules

  • uts/intel/amd64/Makefile.rules

  • uts/i86pc/Makefile.rules

The files provide build rules (targets) which allow make to function in a multiple directory environment. Each source tree below the directory containing the makefile has a build rule in the file.

  • uts/sun4/Makefile.sun4

  • uts/sun4u/Makefile.sun4u

  • uts/intel/Makefile.intel

    *uts/intel/ia32/Makefile.ia32

  • uts/i86pc/Makefile.i86pc

These makefiles contain the definitions specific (defaults) to the obvious 'implementation architecture'. These rules can be overridden in specific leaf node makefiles if necessary.

  • uts/sun4u/unix/Makefile

  • uts/i86pc/unix/Makefile

Main driving makefile for building unix.

  • uts/sun4u/MODULE/Makefile (for MODULE in cgsix, cpu, kb, ...)

Main driving makefile for building MODULE.

  • uts/sun4u/genunix/Makefile

  • uts/i86pc/genunix/Makefile

Main driving makefile for building genunix.

Issuing the command dmake in the uts directory will cause all supported, modularized kernels and modules to be built.

Issuing the command dmake in a uts/ARCHITECTURE directory (i.e.: uts/sparc) will cause all supported, "implementation architecture"-independent modules for ARCHITECTURE to be built.

Issuing the command dmake in a uts/MACHINE directory (i.e.: uts/sun4u) will cause that kernel and all supported, "implementation architecture"- dependent modules for MACHINE to be built.

The makefiles are verbosely commented. It is desired that they should stay this way.

Command Makefile Layout

Most command and daemon subdirectories follow one of two general layout rules, depending on where object files will be located (see Parts of a Command). For ISA-dependent programs, the layout is similar to that used by the kernel. Programs which do not need ISA-dependent build behavior use a simplified makefile layout. In the description here, we use the example of a generic command called foocmd whose sources are located in usr/src/cmd/foocmd. The makefiles relevant to building foocmd are:

  • usr/src/cmd/Makefile

Top-level driving makefile for all commands/daemons. This is a simple recursive makefile which is aware of which subdirectories should be built and will cause the given target to be rebuilt in each of them.

  • usr/src/cmd/Makefile.cmd

This makefile defines the installation directories and rules for installing executables into them.

  • usr/src/cmd/Makefile.cmd.64

Additional definitions specific to 64-bit builds are provided here.

  • usr/src/cmd/Makefile.cmd.bsm

This specialty makefile is used only by auditstat and dminfo. It provides some generic boilerplate rules.

  • usr/src/cmd/Makefile.targ

Basic target definitions for clobber, lint, and installation.

  • usr/src/cmd/foocmd/Makefile

Driving makefile for foocmd. Normally defines PROG but otherwise contains only boilerplate definitions and targets. This is almost always copied from another similar makefile. If foocmd does not require ISA-dependent build behavior, rules will normally be specified directly, including those for the install target. If foocmd does require ISA-dependent build behavior, this makefile will instead define SUBDIRS to include the ISA-specific versions that must be built, and define targets recursively. This will usually leave the install target definition for each ISA makefile and cause a link to $(ISAEXEC) to be created. See section Handling Multiple Architectures for more information on platform dependencies and $(ISAEXEC).

  • usr/src/cmd/foocmd/Makefile.com

Defines PROG, OBJS, SRCS, and includes Makefile.cmd. May also contain additional flags for compilation or linking. This makefile normally defines targets for all, $(PROG), clean, and lint; this portion is usually generic and would be copied from another similar makefile.

  • usr/src/cmd/foocmd/*/Makefile

ISA-specific makefiles, which may define additional ISA-specific flags or targets, and will generally include its own install target to install the ISA-specific program(s) it builds.

Library Makefile Layout

Most library subdirectories follow the same general layout, which is similar to the command layout. Unlike commands, most libraries are built for both 32- and 64-bit architecture variants, so ISA-specific directories will almost always be present. Therefore, the overall build structure for each library is similar to that of the kernel. We'll give an overview of the general makefiles for libraries as well as briefly explain the makefiles for a specific library. Later on in Component Anatomy, Creation, and Modification we'll go into more detail and show examples of building your own library and other components involved.

  • lib/Makefile.lib

This contains the bulk of the macros for building shared objects.

  • lib/Makefile.lib.64

This contains macros for building 64-bit objects, and should be included in Makefiles for 64-bit native ISAs.

  • lib/Makefile.rootfs

This contains macro overrides for libraries that install into /lib (rather than /usr/lib).

  • lib/Makefile.targ

This contains rules for building shared objects.

  • lib/<library>/Makefile

This is the library's top-level Makefile. It should contain rules for building any ISA-independent targets, such as installing header files and building message catalogs, but should defer all other targets to ISA-specific Makefiles.

  • lib/<library>/Makefile.com

This is the library's common Makefile. It should contain rules and macros which are common to all ISAs. This Makefile should never be built explicitly, but instead should be included (using the make include mechanism) by all of your ISA-specific Makefiles.

  • lib/<library>/<isa>/Makefile

These are the library's ISA-specific Makefiles, one per ISA (usually sparc and i386, and sometimes sparcv9 and amd64). These Makefiles should include your common Makefile and then provide any needed ISA-specific rules and definitions, perhaps overriding those provided in your common Makefile.

Conclusion

We've done a basic tour of what source code is located where inside of the gate as well as covered how we deal with object files. We've introduced our Makefile system and covered the basic layout for these components. In the next section we'll go through and detail all of the components of commands, libraries, and kernel modules, as well as go through examples of adding new ones.