biodoc.txt

linux 内核源代码

	Notes on the Generic Block Layer Rewrite in Linux 2.5
	=====================================================

Notes Written on Jan 15, 2002:
	Jens Axboe <[e-mail]>
	Suparna Bhattacharya <[e-mail]>

Last Updated May 2, 2002
September 2003: Updated I/O Scheduler portions
	Nick Piggin <[e-mail]>

Introduction:

These are some notes describing some aspects of the 2.5 block layer in the
context of the bio rewrite. The idea is to bring out some of the key
changes and a glimpse of the rationale behind those changes.

Please mail corrections & suggestions to [e-mail].

Credits:
---------

2.5 bio rewrite:
	Jens Axboe <[e-mail]>

Many aspects of the generic block layer redesign were driven by and evolved
over discussions, prior patches and the collective experience of several
people. See sections 8 and 9 for a list of some related references.

The following people helped with review comments and inputs for this
document:
	Christoph Hellwig <[e-mail]>
	Arjan van de Ven <[e-mail]>
	Randy Dunlap <[e-mail]>
	Andre Hedrick <[e-mail]>

The following people helped with fixes/contributions to the bio patches
while it was still work-in-progress:
	David S. Miller <[e-mail]>


Description of Contents:
------------------------

1. Scope for tuning of logic to various needs
  1.1 Tuning based on device or low level driver capabilities
	- Per-queue parameters
	- Highmem I/O support
	- I/O scheduler modularization
  1.2 Tuning based on high level requirements/capabilities
	1.2.1 I/O Barriers
	1.2.2 Request Priority/Latency
  1.3 Direct access/bypass to lower layers for diagnostics and special
      device operations
	1.3.1 Pre-built commands
2. New flexible and generic but minimalist i/o structure or descriptor
   (instead of using buffer heads at the i/o layer)
  2.1 Requirements/Goals addressed
  2.2 The bio struct in detail (multi-page io unit)
  2.3 Changes in the request structure
3. Using bios
  3.1 Setup/teardown (allocation, splitting)
  3.2 Generic bio helper routines
    3.2.1 Traversing segments and completion units in a request
    3.2.2 Setting up DMA scatterlists
    3.2.3 I/O completion
    3.2.4 Implications for drivers that do not interpret bios (don't handle
 	  multiple segments)
    3.2.5 Request command tagging
  3.3 I/O submission
4. The I/O scheduler
5. Scalability related changes
  5.1 Granular locking: Removal of io_request_lock
  5.2 Prepare for transition to 64 bit sector_t
6. Other Changes/Implications
  6.1 Partition re-mapping handled by the generic block layer
7. A few tips on migration of older drivers
8. A list of prior/related/impacted patches/ideas
9. Other References/Discussion Threads

---------------------------------------------------------------------------

Bio Notes
--------

Let us discuss the changes in the context of how some overall goals for the
block layer are addressed.

1. Scope for tuning the generic logic to satisfy various requirements

The block layer design supports adaptable abstractions to handle common
processing with the ability to tune the logic to an appropriate extent
depending on the nature of the device and the requirements of the caller.
One of the objectives of the rewrite was to increase the degree of tunability
and to enable higher level code to utilize underlying device/driver
capabilities to the maximum extent for better i/o performance. This is
important especially in the light of ever improving hardware capabilities
and application/middleware software designed to take advantage of these
capabilities.

1.1 Tuning based on low level device / driver capabilities

Sophisticated devices with large built-in caches, intelligent i/o scheduling
optimizations, high memory DMA support, etc may find some of the
generic processing an overhead, while for less capable devices the
generic functionality is essential for performance or correctness reasons.
Knowledge of some of the capabilities or parameters of the device should be
used at the generic block layer to take the right decisions on
behalf of the driver.

How is this achieved ?

Tuning at a per-queue level:

i. Per-queue limits/values exported to the generic layer by the driver

Various parameters that the generic i/o scheduler logic uses are set at
a per-queue level (e.g maximum request size, maximum number of segments in
a scatter-gather list, hardsect size)

Some parameters that were earlier available as global arrays indexed by
major/minor are now directly associated with the queue. Some of these may
move into the block device structure in the future. Some characteristics
have been incorporated into a queue flags field rather than separate fields
in themselves.  There are blk_queue_xxx functions to set the parameters,
rather than update the fields directly

Some new queue property settings:

	blk_queue_bounce_limit(q, u64 dma_address)
		Enable I/O to highmem pages, dma_address being the
		limit. No highmem default.

	blk_queue_max_sectors(q, max_sectors)
		Sets two variables that limit the size of the request.

		- The request queue's max_sectors, which is a soft size in
		units of 512 byte sectors, and could be dynamically varied
		by the core kernel.

		- The request queue's max_hw_sectors, which is a hard limit
		and reflects the maximum size request a driver can handle
		in units of 512 byte sectors.

		The default for both max_sectors and max_hw_sectors is
		255. The upper limit of max_sectors is 1024.

	blk_queue_max_phys_segments(q, max_segments)
		Maximum physical segments you can handle in a request. 128
		default (driver limit). (See 3.2.2)

	blk_queue_max_hw_segments(q, max_segments)
		Maximum dma segments the hardware can handle in a request. 128
		default (host adapter limit, after dma remapping).
		(See 3.2.2)

	blk_queue_max_segment_size(q, max_seg_size)
		Maximum size of a clustered segment, 64kB default.

	blk_queue_hardsect_size(q, hardsect_size)
		Lowest possible sector size that the hardware can operate
		on, 512 bytes default.

New queue flags:

	QUEUE_FLAG_CLUSTER (see 3.2.2)
	QUEUE_FLAG_QUEUED (see 3.2.4)


ii. High-mem i/o capabilities are now considered the default

The generic bounce buffer logic, present in 2.4, where the block layer would
by default copyin/out i/o requests on high-memory buffers to low-memory buffers
assuming that the driver wouldn't be able to handle it directly, has been
changed in 2.5. The bounce logic is now applied only for memory ranges
for which the device cannot handle i/o. A driver can specify this by
setting the queue bounce limit for the request queue for the device
(blk_queue_bounce_limit()). This avoids the inefficiencies of the copyin/out
where a device is capable of handling high memory i/o.

In order to enable high-memory i/o where the device is capable of supporting
it, the pci dma mapping routines and associated data structures have now been
modified to accomplish a direct page -> bus translation, without requiring
a virtual address mapping (unlike the earlier scheme of virtual address
-> bus translation). So this works uniformly for high-memory pages (which
do not have a corresponding kernel virtual address space mapping) and
low-memory pages.

Note: Please refer to DMA-mapping.txt for a discussion on PCI high mem DMA
aspects and mapping of scatter gather lists, and support for 64 bit PCI.

Special handling is required only for cases where i/o needs to happen on
pages at physical memory addresses beyond what the device can support. In these
cases, a bounce bio representing a buffer from the supported memory range
is used for performing the i/o with copyin/copyout as needed depending on
the type of the operation.  For example, in case of a read operation, the
data read has to be copied to the original buffer on i/o completion, so a
callback routine is set up to do this, while for write, the data is copied
from the original buffer to the bounce buffer prior to issuing the
operation. Since an original buffer may be in a high memory area that's not
mapped in kernel virtual addr, a kmap operation may be required for
performing the copy, and special care may be needed in the completion path
as it may not be in irq context. Special care is also required (by way of
GFP flags) when allocating bounce buffers, to avoid certain highmem
deadlock possibilities.

It is also possible that a bounce buffer may be allocated from high-memory
area that's not mapped in kernel virtual addr, but within the range that the
device can use directly; so the bounce page may need to be kmapped during
copy operations. [Note: This does not hold in the current implementation,
though]

There are some situations when pages from high memory may need to
be kmapped, even if bounce buffers are not necessary. For example a device
may need to abort DMA operations and revert to PIO for the transfer, in
which case a virtual mapping of the page is required. For SCSI it is also
done in some scenarios where the low level driver cannot be trusted to
handle a single sg entry correctly. The driver is expected to perform the
kmaps as needed on such occasions using the __bio_kmap_atomic and bio_kmap_irq
routines as appropriate. A driver could also use the blk_queue_bounce()
routine on its own to bounce highmem i/o to low memory for specific requests
if so desired.

iii. The i/o scheduler algorithm itself can be replaced/set as appropriate

As in 2.4, it is possible to plugin a brand new i/o scheduler for a particular
queue or pick from (copy) existing generic schedulers and replace/override
certain portions of it. The 2.5 rewrite provides improved modularization
of the i/o scheduler. There are more pluggable callbacks, e.g for init,
add request, extract request, which makes it possible to abstract specific
i/o scheduling algorithm aspects and details outside of the generic loop.
It also makes it possible to completely hide the implementation details of
the i/o scheduler from block drivers.

I/O scheduler wrappers are to be used instead of accessing the queue directly.
See section 4. The I/O scheduler for details.

1.2 Tuning Based on High level code capabilities

i. Application capabilities for raw i/o

This comes from some of the high-performance database/middleware
requirements where an application prefers to make its own i/o scheduling
decisions based on an understanding of the access patterns and i/o
characteristics

ii. High performance filesystems or other higher level kernel code's
capabilities

Kernel components like filesystems could also take their own i/o scheduling
decisions for optimizing performance. Journalling filesystems may need
some control over i/o ordering.

What kind of support exists at the generic block layer for this ?

The flags and rw fields in the bio structure can be used for some tuning
from above e.g indicating that an i/o is just a readahead request, or for
marking  barrier requests (discussed next), or priority settings (currently
unused). As far as user applications are concerned they would need an
additional mechanism either via open flags or ioctls, or some other upper
level mechanism to communicate such settings to block.

1.2.1 I/O Barriers

There is a way to enforce strict ordering for i/os through barriers.
All requests before a barrier point must be serviced before the barrier
request and any other requests arriving after the barrier will not be
serviced until after the barrier has completed. This is useful for higher
level control on write ordering, e.g flushing a log of committed updates
to disk before the corresponding updates themselves.

A flag in the bio structure, BIO_BARRIER is used to identify a barrier i/o.
The generic i/o scheduler would make sure that it places the barrier request and
all other requests coming after it after all the previous requests in the
queue. Barriers may be implemented in different ways depending on the
driver. For more details regarding I/O barriers, please read barrier.txt
in this directory.

1.2.2 Request Priority/Latency

Todo/Under discussion:
Arjan's proposed request priority scheme allows higher levels some broad
  control (high/med/low) over the priority  of an i/o request vs other pending
  requests in the queue. For example it allows reads for bringing in an
  executable page on demand to be given a higher priority over pending write
  requests which haven't aged too much on the queue. Potentially this priority
  could even be exposed to applications in some manner, providing higher level
  tunability. Time based aging avoids starvation of lower priority
  requests. Some bits in the bi_rw flags field in the bio structure are
  intended to be used for this priority information.


1.3 Direct Access to Low level Device/Driver Capabilities (Bypass mode)
    (e.g Diagnostics, Systems Management)

There are situations where high-level code needs to have direct access to
the low level device capabilities or requires the ability to issue commands
to the device bypassing some of the intermediate i/o layers.
These could, for example, be special control commands issued through ioctl
interfaces, or could be raw read/write commands that stress the drive's
capabilities for certain kinds of fitness tests. Having direct interfaces at
multiple levels without having to pass through upper layers makes
it possible to perform bottom up validation of the i/o path, layer by
layer, starting from the media.