.\" Copyright (c) 2003
.\"	Fraunhofer Institute for Open Communication Systems (FhG Fokus).
.\" 	All rights reserved.
.\"
.\" Redistribution and use in source and binary forms, with or without
.\" modification, are permitted provided that the following conditions
.\" are met:
.\" 1. Redistributions of source code must retain the above copyright
.\"    notice, this list of conditions and the following disclaimer.
.\" 2. Redistributions in binary form must reproduce the above copyright
.\"    notice, this list of conditions and the following disclaimer in the
.\"    documentation and/or other materials provided with the distribution.
.\"
.\" THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND
.\" ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
.\" IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
.\" ARE DISCLAIMED.  IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE
.\" FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
.\" DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
.\" OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
.\" HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
.\" LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
.\" OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
.\" SUCH DAMAGE.
.\"
.\" Author: Hartmut Brandt <harti@FreeBSD.org>
.\"
.\" $FreeBSD$
.\"
.Dd July 15, 2003
.Dt MBPOOL 9
.Os
.Sh NAME
.Nm mbpool
.Nd "buffer pools for network interfaces"
.Sh SYNOPSIS
.In sys/types.h
.In machine/bus.h
.In sys/mbpool.h
.Vt struct mbpool ;
.Ft int
.Fo mbp_create
.Fa "struct mbpool **mbp" "const char *name" "bus_dma_tag_t dmat"
.Fa "u_int max_pages" "size_t page_size" "size_t chunk_size"
.Fc
.Ft void
.Fn mbp_destroy "struct mbpool *mbp"
.Ft "void *"
.Fn mbp_alloc "struct mbpool *mbp" "bus_addr_t *pa" "uint32_t *hp"
.Ft void
.Fn mbp_free "struct mbpool *mbp" "void *p"
.Ft void
.Fn mbp_ext_free "void *" "void *"
.Ft void
.Fn mbp_card_free "struct mbpool *mbp"
.Ft void
.Fn mbp_count "struct mbpool *mbp" "u_int *used" "u_int *card" "u_int *free"
.Ft "void *"
.Fn mbp_get "struct mbpool *mbp" "uint32_t h"
.Ft "void *"
.Fn mbp_get_keep "struct mbpool *mbp" "uint32_t h"
.Ft void
.Fo mbp_sync
.Fa "struct mbpool *mbp" "uint32_t h" "bus_addr_t off" "bus_size_t len"
.Fa "u_int op"
.Fc
.Pp
.Fn MODULE_DEPEND "your_module" "libmbpool" 1 1 1
.Pp
.Cd "options LIBMBPOOL"
.Sh DESCRIPTION
Mbuf pools are intended to help drivers for interface cards that need huge
amounts of receive buffers, and additionally provides a mapping between these
buffers and 32-bit handles.
.Pp
An example of these cards are the Fore/Marconi ForeRunnerHE cards.
These
employ up to 8 receive groups, each with two buffer pools, each of which
can contain up to 8192.
This gives a total maximum number of more than
100000 buffers.
Even with a more moderate configuration the card eats several
thousand buffers.
Each of these buffers must be mapped for DMA.
While for
machines without an IOMMU and with lesser than 4GByte memory this is not
a problem, for other machines this may quickly eat up all available IOMMU
address space and/or bounce buffers.
On sparc64, the default I/O page size
is 16k, so mapping a simple mbuf wastes 31/32 of the address space.
.Pp
Another problem with most of these cards is that they support putting a 32-bit
handle into the buffer descriptor together with the physical address.
This handle is reflected back to the driver when the buffer is filled, and
assists the driver in finding the buffer in host memory.
For 32-bit machines,
the virtual address of the buffer is usually used as the handle.
This does not
work for 64-bit machines for obvious reasons, so a mapping is needed between
these handles and the buffers.
This mapping should be possible without
searching lists and the like.
.Pp
An mbuf pool overcomes both problems by allocating DMA-able memory page wise
with a per-pool configurable page size.
Each page is divided into a number of
equally-sized chunks, the last
.Dv MBPOOL_TRAILER_SIZE
of which are used by the pool code (4 bytes).
The rest of each chunk is
usable as a buffer.
There is a per-pool limit on pages that will be allocated.
.Pp
Additionally, the code manages two flags for each buffer:
.Dq on-card
and
.Dq used .
A buffer may be in one of three states:
.Bl -tag -width "on-card"
.It free
None of the flags is set.
.It on-card
Both flags are set.
The buffer is assumed to be handed over to the card and
waiting to be filled.
.It used
The buffer was returned by the card and is now travelling through the system.
.El
.Pp
A pool is created with
.Fn mbp_create .
This call specifies a DMA tag
.Fa dmat
to be used to create and map the memory pages via
.Xr bus_dmamem_alloc 9 .
The
.Fa chunk_size
includes the pool overhead.
It means that to get buffers for 5 ATM cells
(240 bytes), a chunk size of 256 should be specified.
This results in 12 unused
bytes between the buffer, and the pool overhead of four byte.
The total
maximum number of buffers in a pool is
.Fa max_pages
*
.Fa ( page_size
/
.Fa chunk_size ) .
The maximum value for
.Fa max_pages
is 2^14-1 (16383) and the maximum of
.Fa page_size
/
.Fa chunk_size
is 2^9 (512).
If the call is successful, a pointer to a newly allocated
.Vt "struct mbpool"
is set into the variable pointed to by
.Fa mpb .
.Pp
A pool is destroyed with
.Fn mbp_destroy .
This frees all pages and the pool structure itself.
If compiled with
.Dv DIAGNOSTICS ,
the code checks that all buffers are free.
If not, a warning message is issued
to the console.
.Pp
A buffer is allocated with
.Fn mbp_alloc .
This returns the virtual address of the buffer and stores the physical
address into the variable pointed to by
.Fa pa .
The handle is stored into the variable pointed to by
.Fa hp .
The two most significant bits and the 7 least significant bits of the handle
are unused by the pool code and may be used by the caller.
These are
automatically stripped when passing a handle to one of the other functions.
If a buffer cannot be allocated (either because the maximum number of pages
is reached, no memory is available or the memory cannot be mapped),
.Dv NULL
is returned.
If a buffer could be allocated, it is in the
.Dq on-card
state.
.Pp
When the buffer is returned by the card, the driver calls
.Fn mbp_get
with the handle.
This function returns the virtual address of the buffer
and clears the
.Dq on-card
bit.
The buffer is now in the
.Dq used
state.
The function
.Fn mbp_get_keep
differs from
.Fn mbp_get
in that it does not clear the
.Dq on-card
bit.
This can be used for buffers
that are returned
.Dq partially
by the card.
.Pp
A buffer is freed by calling
.Fn mbp_free
with the virtual address of the buffer.
This clears the
.Dq used
bit, and
puts the buffer on the free list of the pool.
Note that free buffers
are NOT returned to the system.
The function
.Fn mbp_ext_free
can be given to
.Fn m_extadd
as the free function.
The user argument must be the pointer to
the pool.
.Pp
Before using the contents of a buffer returned by the card, the driver
must call
.Fn mbp_sync
with the appropriate parameters.
This results in a call to
.Xr bus_dmamap_sync 9
for the buffer.
.Pp
All buffers in the pool that are currently in the
.Dq on-card
state can be freed
with a call to
.Fn mbp_card_free .
This may be called by the driver when it stops the interface.
Buffers in the
.Dq used
state are not freed by this call.
.Pp
For debugging it is possible to call
.Fn mbp_count .
This returns the number of buffers in the
.Dq used
and
.Dq on-card
states and
the number of buffers on the free list.
.Sh SEE ALSO
.Xr mbuf 9
.Sh AUTHORS
.An Harti Brandt Aq harti@FreeBSD.org
.Sh CAVEATS
The function
.Fn mbp_sync
is currently a no-op because
.Xr bus_dmamap_sync 9
is missing the offset and length parameters.