release-notes-1.1.txt

-

    EXPIRES is the (optional) expiry time from the server reply headers.

These values are compared with the parameters of the 'refresh_pattern'
rules.  The refresh parameters are:

	URL regular expression
	MIN_AGE
	PERCENT
	MAX_AGE

The URL regular expressions are checked in the order listed until a
match is found.  Then this algorithm is applied for determining if an
object is fresh or stale:

    if (CLIENT_MAX_AGE)
        if (AGE > CLIENT_MAX_AGE)
            return STALE
    if (AGE <= MIN_AGE)
        return FRESH
    if (EXPIRES) {
        if (EXPIRES <= NOW)
            return STALE
        else
            return FRESH
    }
    if (AGE > MAX_AGE)
        return STALE
    if (LM_FACTOR < PERCENT)
        return FRESH
    return STALE

Note that the Max-Age in a client request takes the highest precedence.
The 'MIN' value should normally be set to zero since it has higher
precedence than the server's Expires: value.  But if you wish to
override the Expires: headers, you may use the MIN value.

Overriding neighbor refresh rules
==============================================================================
The refresh rules also have an effect on the requests your cache makes
to its neighbors.  Squid uses the MAX_AGE value in the HTTP/1.1
"Cache-Control: Max-age=nnn" request header for outgoing requests.
This solves the problem where neighbors with more relaxed refresh
policies would send you stale objects (by your configuration).


Object Purge Policy
==============================================================================
Squid attempts to keep the size of the disk cache relatively "smooth"
or "flat."  That is, objects are removed at the same rate they are
added.  Earlier versions had a "sawtooth" behavior where objects were
removed only when disk space reached an upper limit.

Squid uses a Least-Recently-Used (LRU) replacement algorithm.  Objects
with large LRU age values are removed before objects with small LRU age
values.  We dynamically calculate the LRU age threshold, above which
objects are removed.  The threshold is calculated as an exponential
function between the high and low water marks.  When the store swap
size is near the low water mark, the LRU threshold is large.  This
encourages more objects to be cached.  When the store swap size is near
the high water mark, the LRU threshold is small, encouraging more
objects to be removed.  The 'reference_age' configuration parameter
specifies the upper limit on the LRU age threshold.

The Squid cache storage is implemented as a hash table with some number
of "hash buckets."  Squid scans one bucket at a time and sorts all the
objects in the bucket by their LRU age.  Objects with an LRU age
over the threshold are removed.  The scan rate is adjusted so that
it takes approximately 24 hours to scan the entire cache.  The 
store buckets are randomized so that we don't always scan the same
buckets at the same time of the day.

If the store swap size somehow exceeds the high water mark, Squid
performs an "emergency" object purge.  We sort up to 256 objects in a
store bucket and remove the eight (8) least recently used ones.  This
continues until the disk space is below the low water mark.

X-Forwarded-For request header
==============================================================================
Squid used to add a request header called "Forwarded" which appeared
in some early HTTP/1.1 draft documents.  This header had the format

    Forwarded: by cache-host for client-address

Current HTTP/1.1 draft documents instead use the "Via" header, but it
does not provide any standard way of indicating the client address
in the request.  Since a number of people missed having the originating
client address in the request, Squid now adds its own request header
called "X-Forwarded-For" which looks like this:

    X-Forwarded-For: 128.138.243.150, unknown, 192.52.106.30

Entries are always IP addresses, or the word "unknown" if the address
could not be determined or if it has been disabled with the
'forwarded_for' configuration option.

We must note that access controls based on this header are extremely
weak and simple to fake.  Anyone may hand-enter a request with any IP
address whatsoever.  This is perhaps the reason why client IP addresses
have been omitted from the HTTP/1.1 specification.


Using ICMP to Measure the Network
==============================================================================
As of version 1.1.9, Squid is able to utilize ICMP Round-Trip-Time (RTT)
measurements to select the optimal location to forward a cache miss.
Previously, cache misses would be forwarded to the parent cache
which returned the first ICP reply message.  These were logged
with FIRST_PARENT_MISS in the access.log file.  Now we can 
select the parent which is closest (RTT-wise) to the origin 
server.  

 1. Supporting ICMP in your Squid cache

    It is more important that your parent caches enable the ICMP
    features.  If you are acting as a parent, then you may want
    to enable ICMP on your cache.  Also, if your cache makes
    RTT measurements, it will fetch objects directly if your 
    cache is closer than any of the parents.

    If you want your Squid cache to measure RTT's to origin servers,
    Squid must be compiled with the USE_ICMP option.  This is easily
    accomplished by uncommenting "-DUSE_ICMP=1" in src/Makefile and
    src/Makefile.in.

    An external program called 'pinger' is responsible for sending and
    receiving ICMP packets.  It must run with root privileges.  After
    Squid has been compiled, the pinger program must be installed
    separately.  A special Makefile target will install 'pinger' with
    appropriate permissions.

        % make install
        % su
        # make install-pinger

    There are three configuration file options for tuning the
    measurement database on your cache.  'netdb_low' and 'netdb_high'
    specify high and low water marks for keeping the database to a
    certain size  (e.g. just like with the IP cache).  The 'netdb_ttl'
    option specifies the minimum rate for pinging a site.  If
    'netdb_ttl' is set to 300 seconds (5 minutes) then an ICMP packet
    will not be sent to the same site more than once every five
    minutes.  Note that a site is only pinged when an HTTP request for
    the site is received.

    Another option, 'minimum_direct_hops' can be used to try finding
    servers which are close to your cache.  If the measured hop count
    to the origin server is less than or equal to minimum_direct_hops,
    the request will be forwarded directly to the origin server.

 2. Utilizing your parents database

    Your parent caches can be asked to include the RTT measurements
    in their ICP replies.  To do this, you must enable 'query_icmp'
    in your config file:

        query_icmp on

    This causes a flag to be set in your outgoing ICP queries. 

    If your parent caches return ICMP RTT measurements then 
    the eighth column of your access.log will have lines 
    similar to:

        CLOSEST_PARENT_MISS/it.cache.nlanr.net

    In this case, it means that 'it.cache.nlanr.net' returned 
    the lowest RTT to the origin server.  If your cache measured
    a lower RTT than any of the parents, the request will
    be logged with

        CLOSEST_DIRECT/www.sample.com


 3. Inspecting the database

    The measurement database can be viewed from the cachemgr by
    selecting "Network Probe Database."  Hostnames are aggregated
    into /24 networks.  All measurements made are averaged over
    time.  Measurements are made to specific hosts, taken from
    the URLs of HTTP requests.  The recv and sent fields are the
    number of ICMP packets sent and received.  At this time they
    are only informational.

    A typical database entry looks something like this:

    Network          recv/sent     RTT  Hops Hostnames
    192.41.10.0        20/  21    82.3   6.0 www.jisedu.org www.dozo.com 
        bo.cache.nlanr.net        42.0   7.0
        uc.cache.nlanr.net        48.0  10.0
        pb.cache.nlanr.net        55.0  10.0
        it.cache.nlanr.net       185.0  13.0

    This means we have sent 21 pings to both www.jisedu.org and
    www.dozo.com.  The average RTT is 82.3 milliseconds.  The 
    next four lines show the measured values from our parent
    caches.  Since 'bo.cache.nlanr.net' has the lowest RTT,
    it would be selected as the location to forward a request
    for a www.jisedu.org or www.dozo.com URL.


Planning for Squid's Memory Usage
==============================================================================
Squid-1.1 has better memory management, although still not ideal.  
Squid uses memory in a variety of ways, but the bulk of memory
usage falls into two categories: per-object metadata and in-transit
objects.

The per-object metadata consists of a StoreEntry data structure, plus
the URL for every object your cache knows about.  This usually averages
out to about 100 bytes per object.  If you assume that the objects
themselves average 20 KB each, then given your disk size ('cache_swap')
you need 1/200th as much for in-memory object metadata.

The other big memory use is due to in-transit objects.  The amount
of memory required for this will depend on your cache's usage patterns.
Obviously a more busy cache will require more memory for in-transit
objects.

The 'cache_mem' parameter places a soft upper limit on the amount of
memory Squid allocates for holding whole objects in VM.  The
'cache_mem' memory is allocated as a pool of 4k blocks.  Objects held
in memory are stored in blocks from this pool.  The 'cache_mem_low' and
'cache_mem_high' values affect the memory reclamation algorithm.

There are two types of in-memory objects: in-transit objects and
completed objects.  The in-transit objects are "locked" in memory until
they are completed.  The completed objects may be either normal or
"negative-cached" objects.

Whenever new memory is needed for in-transit objects and current usage
is above the high water mark, Squid purges some completed objects from
memory.   The in-memory objects are sorted by time of last use and then
removed in order until memory usage is below the low water mark.

Occasionally Squid may need to exceed the maximum number of blocks.
This will happen if all of the in-transit objects will not fit within
the 'cache_mem' pool size.  You will see this warning in your cache.log
file:

    WARNING: Exceeded 'cache_mem' size (4122K > 4096K)

If this warning occurs frequently then you need to consider either
increasing the 'cache_mem' value or decreasing the
'maximum_object_size' value.  If the cache_mem usage is above the low
water mark, then Squid will check for objects larger than
'maximum_object_size.'  Any such objects are put into "delete behind"
mode which means Squid releases the section of the object which has
been delivered to all clients reading from it.

As a rule-of-thumb, you should probably set 'cache_mem' to 1/3 of your
machine's physical memory amount.  You can plan on another 1/3 being
used by the per-object metadata.  And the final 1/3 will be used by
other data structures, unaccounted memory, and malloc() overhead.
Note, this assumes that the machine will be dedicated to running
Squid.  If there are other services on the machine, the memory
estimates should be lowered.

Default Parent
==============================================================================
Squid has the ability to mark parent caches as the 'default' way to 
fetch objects.  This is probably only useful with the 'no-query' option.
For example, the configuration

     cache_host N1 sibling 3128 3130
     cache_host N2 sibling 3128 3130
     cache_host N3 sibling 3128 3130
     cache_host P1 parent 3128 3130 no-query default

will result in ICP queries to sibling caches N1, N2, and N3.  If none
of the siblings has the requested object then it will be retrieved
through parent P1 due to the 'default' designation.  Note that
'default' does not conflict with any 'cache_host_domain' restrictions
which might be placed on a neighbor.

We do not normally recommend use of the default option.  If your
parent cache(s) uses ICP then you should also send them ICP queries.
If your default parent is unreachable then Squid will return error
messages, it will not attempt direct connections to the source.

Cachemgr Passwords
==============================================================================
Squid-1.1 allows cachemgr passwords to be specified in the squid.conf
file (instead of an /etc/passwd entry).  There may be a different
password for each cachemgr operation, but only one password per
operation.  Some sensitive operations require a password, others may be
executed if no passwords are given in the squid.conf file.  Operations
may be disabled by setting the password to "none." See squid.conf for a
full list of cachemgr operations.

Round-Robin IP
==============================================================================
When a hostname resolves to multiple IP addresses, Squid now cycles to
the next address after each connection.  If a connection to an address
fails, it is removed from the list.  If a hostname runs out of
addresses, it is removed from the IP cache.