VMEM(9) Kernel Concepts VMEM(9)


vmem - virtual memory allocator



An address space is divided into a number of logically distinct pieces, or
arenas: text, data, heap, stack, and so on. Within these arenas we often
subdivide further; for example, we use heap addresses not only for the
kernel heap (kmem_alloc() space), but also for DVMA, bp_mapin(), /dev/kmem,
and even some device mappings.

The kernel address space, therefore, is most accurately described as a tree
of arenas in which each node of the tree imports some subset of its parent.
The virtual memory allocator manages these arenas and supports their
natural hierarchical structure.


An arena is nothing more than a set of integers. These integers most
commonly represent virtual addresses, but in fact they can represent
anything at all. For example, we could use an arena containing the
integers minpid through maxpid to allocate process IDs. For uses of this
nature, prefer id_space(9F) instead.

vmem_create() and vmem_destroy() create and destroy vmem arenas. In order
to differentiate between arenas used for addresses and arenas used for
identifiers, the VMC_IDENTIFIER flag is passed to vmem_create(). This
prevents identifier exhaustion from being diagnosed as general memory


We represent the integers in an arena as a collection of spans, or
contiguous ranges of integers. For example, the kernel heap consists of
just one span: [kernelheap, ekernelheap). Spans can be added to an arena
in two ways: explicitly, by vmem_add(); or implicitly, by importing, as
described in Imported Memory below.


Spans are subdivided into segments, each of which is either allocated or
free. A segment, like a span, is a contiguous range of integers. Each
allocated segment [addr, addr + size) represents exactly one
vmem_alloc(size) that returned addr. Free segments represent the space
between allocated segments. If two free segments are adjacent, we coalesce
them into one larger segment; that is, if segments [a, b) and [b, c) are
both free, we merge them into a single segment [a, c). The segments within
a span are linked together in increasing-address order so we can easily
determine whether coalescing is possible.

Segments never cross span boundaries. When all segments within an imported
span become free, we return the span to its source.

Imported Memory

As mentioned in the overview, some arenas are logical subsets of other
arenas. For example, kmem_va_arena (a virtual address cache that satisfies
most kmem_slab_create() requests) is just a subset of heap_arena (the
kernel heap) that provides caching for the most common slab sizes. When
kmem_va_arena runs out of virtual memory, it imports more from the heap; we
say that heap_arena is the vmem source for kmem_va_arena. vmem_create()
allows you to specify any existing vmem arena as the source for your new
arena. Topologically, since every arena is a child of at most one source,
the set of all arenas forms a collection of trees.

Constrained Allocations

Some vmem clients are quite picky about the kind of address they want. For
example, the DVMA code may need an address that is at a particular phase
with respect to some alignment (to get good cache coloring), or that lies
within certain limits (the addressable range of a device), or that doesn't
cross some boundary (a DMA counter restriction) -- or all of the above.
vmem_xalloc() allows the client to specify any or all of these constraints.

The Vmem Quantum

Every arena has a notion of `quantum', specified at vmem_create() time,
that defines the arena's minimum unit of currency. Most commonly the
quantum is either 1 or PAGESIZE, but any power of 2 is legal. All vmem
allocations are guaranteed to be quantum-aligned.

Relationship to the Kernel Memory Allocator

Every kmem cache has a vmem arena as its slab supplier. The kernel memory
allocator uses vmem_alloc() and vmem_free() to create and destroy slabs.


id_space(9F), vmem_add(9F), vmem_alloc(9F), vmem_contains(9F),
vmem_create(9F), vmem_walk(9F)

Jeff Bonwick and Jonathan Adams, "Magazines and vmem: Extending the Slab
Allocator to Many CPUs and Arbitrary Resources.", Proceedings of the 2001
Usenix Conference, http://www.usenix.org/event/usenix01/bonwick.html.

illumos January 18, 2017 illumos