Statically Defined Tracing for User Applications

DTrace provides a facility for user application developers to define customized probes in application code to augment the capabilities of the pid provider. These static probes impose little to no overhead when disabled and are dynamically enabled like all other DTrace probes. You can use static probes to describe application semantics to users of DTrace without exposing or requiring implementation knowledge of your applications. This chapter describes how to define static probes in user applications and how to use DTrace to enable such probes in user processes.

34.1. Choosing the Probe Points

DTrace allows developers to embed static probe points in application code, including both complete applications and shared libraries. These probes can be enabled wherever the application or library is running, either in development or in production. You should define probes that have a semantic meaning that is readily understood by your DTrace user community. For example, you could define query-receive and query-respond probes for a web server that correspond to a client submitting a request and the web server responding to that request. These example probes are easily understood by most DTrace users and correspond to the highest level abstractions for the application, rather than lower level implementation details. DTrace users might use these probes to understand the time distribution of requests. If your query-receive probe presented the URL request strings as an argument, a DTrace user could determine which requests were generating the most disk I/O by combining this probe with the io provider.

You should also consider the stability of the abstractions you describe when choosing probe names and locations. Will this probe persist in future releases of the application, even if the implementation changes? Does the probe make sense on all system architectures or is it specific to a particular instruction set? This chapter will discuss the details of how these decisions guide your static tracing definitions.

34.2. Adding Probes to an Application

DTrace probes for libraries and executables are defined in an ELF section in the corresponding application binary. This section describes how to define your probes, add them to your application source code, and augment your application's build process to include the DTrace probe definitions.

34.2.1. Defining Providers and Probes

You define DTrace probes in a .d source file which is then used when compiling and linking your application. First, select an appropriate name for your user application provider. The provider name you choose will be appended with the process identifier for each process that is executing your application code. For example, if you chose the provider name myserv for a web server that was executing as process ID 1203, the DTrace provider name corresponding to this process would be myserv1203. In your .d source file, add a provider definition similar to the following example:

provider myserv {
	...
};

Next, add a definition for each probe and the corresponding arguments. The following example defines the two probes discussed in Choosing the Probe Points. The first probe has two arguments, both of type string, and the second probe has no arguments. The D compiler converts two consecutive underbars (__) in any probe name to a hyphen (-).

provider myserv {
	probe query__receive(string, string);
	probe query__respond();
};

You should add stability attributes to your provider definition so that consumers of your probes understand the likelihood of change in future versions of your application. See Stability for more information on the DTrace stability attributes. Stability attributes are defined as shown in the following example:

#pragma D attributes Evolving/Evolving/Common provider myserv provider
#pragma D attributes Private/Private/Unknown provider myserv module
#pragma D attributes Private/Private/Unknown provider myserv function
#pragma D attributes Evolving/Evolving/Common provider myserv name
#pragma D attributes Evolving/Evolving/Common provider myserv args

provider myserv {
	probe query__receive(string, string);
	probe query__respond();
};

D scripts that use non-integer arguments from user added probes must use the copyin and copyinstr functions to retrieve those arguments. Please see User Process Tracing for more information.

34.2.2. Adding Probes to Application Code

Now that you have defined your probes in a .d file, you need to augment your source code to indicate the locations that should trigger your probes. Consider the following example C application source code:

void
main_look(void)
{
	...
	query = wait_for_new_query();
	process_query(query)
	...
}

To add a probe site, add a reference to the DTRACE_PROBE() macro defined in <sys/sdt.h> as shown in the following example:

#include <sys/sdt.h>
...

void
main_look(void)
{
	...
	query = wait_for_new_query();
	DTRACE_PROBE2(myserv, query__receive, query->clientname, query->msg);
	process_query(query)
	...
}

The suffix 2 in the macro name DTRACE_PROBE2 refers the number of arguments that are passed to the probe. The first two arguments to the probe macro are the provider name and probe name and must correspond to your D provider and probe definitions. The remaining macro arguments are the arguments assigned to the DTrace arg0..9 variables when the probes fires. Your application source code can contain multiple references to the same provider and probe name. If multiple references to the same probe are present in your source code, any of the macro references will cause the probe to fire.

34.2.3. Building Applications with Probes

You must augment the build process for your application to include the DTrace provider and probe definitions. A typical build process takes each source file and compiles it to create a corresponding object file. The compiled object files are then linked together to create the finished application binary, as shown in the following example:

cc -c src1.c
cc -c src2.c
...
cc -o myserv src1.o src2.o ...

To include DTrace probe definitions in your application, add appropriate Makefile rules to your build process to execute the dtrace command as shown in the following example:

cc -c src1.c
cc -c src2.c
...
dtrace -G -32 -s myserv.d src1.o src2.o ...
cc -o myserv myserv.o src1.o src2.o ...

The dtrace command shown above post-processes the object files generated by the preceding compiler commands and generates the object file myserv.o from myserv.d and the other object files. The dtrace -G option is used to link provider and probe definitions with a user application. The -32 option is used to build 32–bit application binaries. The -64 option is used to build 64–bit application binaries.

34.2.4. Enabling Applications with Probes

When an application with statically defined probes is executed it registers those probes with DTrace. Like the pid provider, a separate provider is registered for each process. If the process launched has the PID 123, then the provider would be for the previously defined myserv would be myserv123. To enable this for all current and future applications with the myserv provider you can use * as the process ID. This results in the following provider myserv*.

If you use the wildcard syntax from inside a zone, then the enabling is limited to just that particular zone. However, if the wildcard syntax is used with an enabling from the global zone, then the enabling is not restricted based on zone. If you are in the global zone and only interested in globally enabling these USDT probes for a single zone then you should use the zone option, specifying the name of the zone to be instrumented.

The following script shows how to use the zone option to only enable applications in the zone named dhcpd.

# pragma D option zone=dhcpd

myserv*:::
{
	@[pid] = count();
}