coss-notes.txt

一个功能非常全面的代理服务器源代码程序,

COSS notes

Adrian Chadd <[e-mail]>

$Id: coss-notes.txt,v 1.4 2006/10/26 19:29:38 serassio Exp $


COSS is a Cyclic Object storage system originally designed by
Eric Stern <[e-mail]>. The idea has been extended
and worked into the current framework by myself.

In these notes I'll discuss the current implementation of COSS
and note where the implementation differed from Eric's original
idea and why the design changes were made.


COSS basics
-----------

COSS works with a single file. Eventually the file may actually be
a raw disk device, but since squid doesn't cache the disk reads
in memory the OS buffer cache will need to be employed for reasonable
performance. For the purposes of this discussion the COSS storage
device will be referred to as a file.

Each stripe is a fixed size an in a fixed position in the file. The
stripe size is a compile-time option.

As objects are written to a COSS stripe, their place is pre-reserved
and data is copied into a memory copy of the stripe. Because of this,
the object size must be known before it can be stored in a COSS
filesystem. (Hence the max-size requirement with a coss cache_dir.)

When a stripe is filled, the stripe is written to disk, and a new
memory stripe is created.

When objects are read back from the COSS file, they can either come
from a stripe in-memory (the current one, or one being written),
or from the disk. If the object is still in a memory stripe, then
it is copied from memory rather than read of disk.

If an object is read from disk, it is re-written to the head of
the current stripe (just as if it were a new object.) This is required
for correct operation of the replacement policy, detailed below.

When the entire COSS file is full, the current stripe again becomes the
fist stripe in the file, and the objects in that stripe are released.
Since the objects on disk are kept in a strict LRU representing the
replacement policy LRU linking the StoreEntry's together, this simply
involves walking the tail of the LRU and freeing entries until we
hit an entry in the next stripe.


COSS implementation details
---------------------------

* The stripe size is fixed. In the original COSS code, Eric optimised
  this a little by allowing the stripes to be truncated to not
  waste disk space at the end of the stripe. This was removed
  to simplify the allocation code slightly and make things easier
  when the store log and checksums are combined in the stripe
  for faster rebuilds.

* COSS currently copies object memory around WAY too much. This needs
  to be fixed eventually.

* It would be nice if the storeRead() interface were a little smarter
  and allowed the filesystem to return as much of an object as possible.
  This would be good for COSS since the read from disk could be simplified
  to use a single OS read() call - this would work really well for
  the object types COSS is designed to cache.

* The original coss code used file_read() and file_write() for disk IO.
  The file_* routines were initially used to implement async disk IO,
  and Eric probably wrote some async disk code for windows.
  I've written a very very simple async_io.c module which uses POSIX
  AIO to implement the async IO. POSIX AIO is well-suited to the
  disk IO COSS performs.

COSS direction
--------------

Eventually, when more of squid is rewritten, I'm going to replace
the replacement policy with something a little more flexible.
A shortcut would be to use a slab allocator and have one slab per
stripe for the StoreEntry's. When it comes time to replace a stripe,
you can just treat the stripe as an array. This would not work
well in the current squid codebase, but it would work well in the
planned rewrite. This would also allow alternate replacement policies
to be used. Oh, it'd cut down the storage requirements per
StoreEntry by two pointers (8 bytes on the i386.)

Notes by DW July 23, 2003
-------------------------

Fixed up swap_filen -> offset implementation.  Now user can use a
block-size setting to determine the maximum COSS cache_dir size.

Fixed bug when cached response is larger than COSS stripe size.
Now require max-size to be less than COSS_MEMBUF_SZ.

Fixed a lockcount bug.  Some aborted requests for cache hits failed
to unlock the CossMemBuf because storeCossReadDone isn't called again.
Solution is to add locked_membuf pointer to CossState structure and
always unlock it if set.  This is probably more reliable than
unlocking based on diskstart/diskend offsets.

I'm worried that COSS is susceptible to a denial-of-service.  If
the user can create N cache misses for responses about as large as
COSS_MEMBUF_SZ, then COSS probably allocates N membufs (stripes)
at the same time.  For large enough values of N, this should cause
a malloc failure.  Solution may be to refuse to allocate new stripes
(thus returning failure for cache misses and hits) after so many
have already been allocated.

Adrian's code has this comment:

    /* Since we're not supporting NOTIFY anymore, lets fail */
    assert(which != COSS_ALLOC_NOTIFY);

However, COSS_ALLOC_NOTIFY was still present in the store_dir_coss.c
rebuild routines.  To avoid assertions during rebuild, I commented
out the storeCossAllocate(SD, e, COSS_ALLOC_NOTIFY) call.

-- Notes: Adrian Chadd, 9/May/2006

* The types used by COSS have been modified to support Large file support,
  at least under Linux. One can compile with --with-large-files to make
  sure the right options have been enabled. no compile or run-time checks
  are currently made to ensure the code has been compiled to support
  large filesystems.. at least yet.

-- Notes: Guido Serassio, 26/October/2006

* When using a regular file as container, COSS storage must be initialized
  once using squid -z like UFS storage.