DTrace is a comprehensive dynamic tracing framework for the illumos™ Operating System. DTrace provides a powerful infrastructure to permit administrators, developers, and service personnel to concisely answer arbitrary questions about the behavior of the operating system and user programs. The illumos Dynamic Tracing Guide describes how to use DTrace to observe, debug, and tune system behavior. This book also includes a complete reference for bundled DTrace observability tools and the D programming language.

This illumos release supports systems that use the SPARCTM and x86 families of processor architectures: UltraSPARCTM, SPARC64, AMD64, Pentium, and Xeon EM64T. The supported systems appear in the illumos Hardware Compatibility List. This document cites any implementation differences between the platform types.

In this document the term “x86” refers to 64–bit and 32–bit systems manufactured using processors compatible with the AMD64 or Intel Xeon/Pentium product families. For supported systems, see the illumos Hardware Compatibility List.

Who Should Use This Book

If you have ever wanted to understand the behavior of your system, DTrace is the tool for you. DTrace is a comprehensive dynamic tracing facility that is built into illumos. The DTrace facility can be used to examine the behavior of user programs. The DTrace facility can also be used to examine the behavior of the operating system. DTrace can be used by system administrators or application developers, and is suitable for use with live production systems. DTrace will allow you to explore your system to understand how it works, track down performance problems across many layers of software, or locate the cause of aberrant behavior. As you'll see, DTrace lets you create your own custom programs to dynamically instrument the system and provide immediate, concise answers to arbitrary questions you can formulate using the DTrace D programming language.

DTrace allows all illumos users to:

DTrace allows illumos developers and administrators to:

This guide will teach you everything you need to know about using DTrace. Basic familiarity with a programming language such as C or a scripting language such as awk(1) or perl(1) will help you learn DTrace and the D programming language faster, but you need not be an expert in any of these areas. If you have never written a program or script before in any language, Related Information provides references to other documents you might find useful.

How This Book Is Organized

Introduction provides a whirlwind tour of the entire DTrace facility and introduces readers to the D programming language. Types, Operators, and Expressions, Variables, and D Program Structure then discuss the fundamentals of D in greater detail, and explain how D programs are converted into dynamic instrumentation. This initial group of chapters should be read first by all readers.

Pointers and Arrays, Strings, Structs and Unions, and Type and Constant Definitions discuss the remaining D language features, most of which will be familiar already to C, C++, and JavaTM programmers. Readers who are unfamiliar with any of these languages should read these chapters; more experienced programmers may wish to proceed directly to later chapters.

Aggregations and Actions and Subroutines discuss DTrace's powerful primitive for aggregating data and the set of built-in actions that can be used to build tracing experiments. All readers should carefully read these chapters.

Buffers and Buffering describes the DTrace policies for buffering data and how these can be configured. This chapter should be read by users once they are familiar with constructing and running D programs.

Output Formatting describes the D output formatting actions as well as the default policy for formatting trace data. Readers who are familiar with the C printf function can rapidly skim this chapter. Readers who have never seen printf before should read this chapter carefully.

Speculative Tracing discusses the DTrace facility for speculatively committing data to a trace buffer. This chapter should be read by users who need to use DTrace in a situation where data must be traced prior to understanding whether it is relevant to the question at hand.

dtrace(1M) Utility provides a complete reference for the dtrace command-line utility, similar to the corresponding on-line manual page. Readers may wish to refer to this chapter when various command-line options are presented elsewhere in the book. Scripting then discusses how the dtrace utility can be used to construct executable D scripts and process their command-line arguments, and Options and Tunables describes the options that can be tuned on the command-line or from within a D program itself.

The group of chapters beginning with dtrace Provider and ending with fasttrap Provider discuss the various DTrace providers that can be used to instrument various aspects of the illumos system. All readers should skim these chapters to familiarize themselves with the various providers, and then return back to read particular chapters in detail as needed.

User Process Tracing discusses examples of using DTrace to instrument user processes. Statically Defined Tracing for User Applications describes how application programmers can add customized DTrace providers and probes to user applications. Readers who are user program developers or administrators and wish to use DTrace to investigate user process behavior should read these chapters.

Security and the remaining chapters discuss advanced topics such as security, versioning, and stability attributes of DTrace, and how to perform boot-time and post-mortem tracing with DTrace. These chapters are intended for advanced DTrace users.

Related Information

These books and papers are recommended and related to the tasks that you need to perform with DTrace:

You can share your DTrace experiences and scripts with the rest of the DTrace community on the DTrace discussion mailing list.

Additional Documentation

Additional resources can be found on the illumos website, including the manual pages referenced throughout this book, and additional books and guides for using illumos and its tools.

We Welcome Your Comments

The DTrace and illumos communities are interested in improving this documentation and welcome your comments and suggestions. To share your comments, either join and e-mail the DTrace discussion mailing list or go to the illumos-docbooks repository and click "Issues".

Typographic Conventions

The following table describes the typographic conventions that are used in this book.

Typographic Conventions





The names of commands, files, and directories, and onscreen computer output

Edit your .login file.

Use ls -a to list all files.

machine_name% you have mail.


What you type, contrasted with onscreen computer output

machine_name% su



Placeholder: replace with a real name or value

The command to remove a file is rm filename.


Book titles, new terms, and terms to be emphasized

Read Chapter 6 in the User's Guide.

A cache is a copy that is stored locally.

Do not save the file.

Note: Some emphasized items appear bold online.

Shell Prompts in Command Examples

The following table shows the default UNIXTM system prompt and superuser prompt for the C shell, Bourne shell, and Korn shell.

Shell Prompts



C shell


C shell for superuser


Bourne shell and Korn shell


Bourne shell and Korn shell for superuser