Modular Debugger Overview

The Modular Debugger (MDB) is a general purpose debugging tool for the illumos operating system whose primary feature is its extensibility. This book describes how to use MDB to debug complex software systems, with a particular emphasis on the facilities available for debugging the illumos kernel and associated device drivers and modules. The book also includes a complete reference for and discussion of the MDB language syntax, debugger features, and MDB Module Programming API.

1.1. Introduction

Debugging is the process of analyzing the execution and state of a software program in order to remove defects. Traditional debugging tools provide facilities for execution control so that programmers can execute programs in a controlled environment and display the current state of program data or evaluate expressions in the source language used to develop the program. Unfortunately, these techniques are often inappropriate for debugging complex software systems such as:

• An operating system, where bugs might not be reproducible and program state is massive and distributed

• Programs that are highly optimized or have had their debug information removed

• Programs that are themselves low-level debugging tools

• Customer situations where the developer can only access post-mortem information

MDB provides a completely customizable environment for debugging these programs and scenarios, including a dynamic module facility that you can use to implement your own debugging commands to perform program-specific analysis. Each MDB module can be used to examine the program in several different contexts, including live and post-mortem. The illumos OS includes a set of MDB modules that help you debug the illumos kernel and related device drivers and kernel modules. Third-party developers might want to develop and deliver their own debugging modules for supervisor or user software.

1.2. MDB Features

MDB provides an extensive collection of features for analyzing the illumos kernel and other target programs. You can do the following tasks in MDB:

• Perform post-mortem analysis of illumos kernel crash dumps and user process core dumps. MDB includes a collection of debugger modules that facilitate sophisticated analysis of kernel and process state, in addition to standard data display and formatting capabilities. These debugger modules enable you to formulate complex queries to investigate kernel and process state in the following ways:

  • Locate all the memory allocated by a particular thread

  • Print a visual picture of a kernel STREAM

  • Determine what type of structure a particular address refers to

  • Locate leaked memory blocks in the kernel

  • Analyze memory to locate stack traces

• Use a first-class programming API to implement your own debugger commands and analysis tools without having to recompile or modify MDB. In MDB, debugging support is implemented as a set of loadable modules (shared libraries that the debugger can open with the dlopen(3C) function), each of which provides a set of commands that extends the capabilities of MDB. MDB provides an API of core services, such as the ability to read and write memory and access symbol table information. MDB provides a framework for you to implement debugging support for your own drivers and modules. Your command and tools can then be made available for everyone to use.

• Learn to use MDB if you are already familiar with the legacy debugging tools adb and crash. MDB provides backward compatibility with these existing debugging solutions. The MDB language is a superset of the adb language. All existing adb macros and commands work within MDB. Thus, developers who use adb can immediately use MDB without knowing any MDB-specific commands. MDB also provides commands that surpass the functionality available from the crash utility.

• Benefit from enhanced usability features. MDB provides many usability features, including the following:

  • Command-line editing

  • Command history

  • Built-in output pager

  • Syntax error checking and handling

  • Online help

  • Interactive session logging

1.3. Using MDB

MDB is available on illumos systems as two commands that share common features: mdb and kmdb. You can use the mdb command interactively or in scripts to debug live user processes, user process core files, kernel crash dumps, the live operating system, object files, and other files. You can use the kmdb command to debug the live operating system kernel and device drivers when you also need to control and halt the execution of the kernel. To start mdb, use the mdb command as described in the mdb(1) manual page. To start kmdb, boot the system as described in the kmdb(1) manual page, or execute the mdb command with the -K option.

1.4. Future Enhancements

MDB provides a stable foundation for developing advanced post-mortem analysis tools. Each illumos release includes additional MDB modules that provide even more sophisticated functionality for debugging the kernel and other software programs. You can use MDB to debug existing software programs, and you can develop your own modules to improve your ability to debug your own illumos drivers and applications.