prog-guide.sgml

-

<sect>Forwarding Selection

<!-- %%%% Chapter : IP/FQDN CACHE %%%% -->
<sect>IP Cache and FQDN Cache

<sect1> Introduction

<P>
The IP cache is a built-in component of squid providing
Hostname to IP-Number translation functionality and managing 
the involved data-structures. Efficiency concerns require
mechanisms that allow non-blocking access to these mappings.
The IP cache usually doesn't block on a request except for special 
cases where this is desired (see below).

<sect1> Data Structures 

<P>
The data structure used for storing name-address mappings
is a small hashtable (<em>static hash_table *ip_table</em>),
where structures of type <em>ipcache_entry</em> whose most
interesting members are:

<verb>
struct _ipcache_entry {
char *name;
time_t lastref;
ipcache_addrs addrs;
struct _ip_pending *pending_head;
char *error_message;
unsigned char locks;
ipcache_status_t status:3;
}
</verb>


<sect1> External overview

<P>
Main functionality
is provided through calls to:
<descrip>
<tag>ipcache_nbgethostbyname(const char *name, IPH *handler, void *handlerdata)
</tag>
where <em/name/ is the name of the host to resolve, <em/handler/ is a 
pointer to the function to be called when the reply from the IP cache (or
the DNS if the IP cache misses) and <em/handlerdata/ is information that
is passed to the handler and does not affect the IP cache.
<tag>ipcache_gethostbyname(const char *name,int flags)</tag>
is different in that
it only checks if an entry exists in it's data-structures and does not by
default contact the DNS, unless this is requested, by setting the <em/flags/ to
<em/IP_BLOCKING_LOOKUP/ or <em/IP_LOOKUP_IF_MISS/.
<tag>ipcache_init()</tag> is called from <em/mainInitialize()/ after disk
initialization and prior to the reverse fqdn cache initialization
<tag>ipcache_restart()</tag> is called to clear the IP cache's data structures,
cancel all pending requests. Currently, it is only called from
<em/mainReconfigure/.
</descrip>

<sect1> Internal Operation 

<P>
Internally, the execution flow is as follows: On a miss, 
 <em/ipcache_getnbhostbyname/ checks whether a request for this name is already
pending, and if positive, it creates a new entry using <em/ipcacheAddNew/ with
the <em/IP_PENDING/ flag set . Then it calls <em/ipcacheAddPending/ to add a
request to the queue together with data and handler.
Else, <em/ipcache_dnsDispatch()/ is called to
directly create a DNS query or to <em/ipcacheEnqueue()/ if all no DNS port is
free.
<em/ipcache_call_pending()/ is called regularly to walk down the pending
list and call handlers. LRU clean-up is performed through <em/ipcache_purgelru()/
according to the <em/ipcache_high/ threshold.

<!-- %%%% Chapter : SERVER PROTOCOLS %%%% -->
<sect>Server Protocols
<sect1>HTTP
<sect1>FTP
<sect1>Gopher
<sect1>Wais
<sect1>SSL
<sect1>Passthrough

<!-- %%%% Chapter : TIMEOUTS %%%% -->
<sect>Timeouts

<!-- %%%% Chapter : EVENTS %%%% -->
<sect>Events

<!-- %%%% Chapter : ACCESS CONTROLS %%%% -->
<sect>Access Controls

<!-- %%%% Chapter : ICP %%%% -->
<sect>ICP

<!-- %%%% Chapter : NETDB %%%% -->
<sect>Network Measurement Database

<!-- %%%% Chapter : Error Pages %%%% -->
<sect>Error Pages

<sect>Callback Data Database

<P>
	Squid's extensive use of callback functions makes it very
	susceptible to memory access errors.  For a blocking operation
	with callback functions, the normal sequence of events is as
	follows:
<verb>
	callback_data = malloc(...);
	...
	fooOperationStart(bar, callback_func, callback_data);
	...
	fooOperationComplete(...);
	callback_func(callback_data, ....);
	...
	free(callback_data);
</verb>
	However, things become more interesting if we want or need
	to free the callback_data, or otherwise cancel the callback,
	before the operation completes.

<P>
	The callback data database lets us do this in a uniform and
	safe manner.  Every callback_data pointer must be added to the
	database.  It is then locked while the blocking operation executes
	elsewhere, and is freed when the operation completes.  The normal
	sequence of events is:
<verb>
	callback_data = malloc(...);
	cbdataAdd(callback_data);
	...
	cbdataLock(callback_data);
	fooOperationStart(bar, callback_func, callback_data);
	...
	fooOperationComplete(...);
	if (cbdataValid(callback_data)) {
		callback_func(callback_data, ....);
	cbdataUnlock(callback_data);
	cbdataFree(callback_data);
</verb>

<P>
	With this scheme, nothing bad happens if <tt/cbdataFree/ gets called
	before <tt/cbdataUnlock/:
<verb>
	callback_data = malloc(...);
	cbdataAdd(callback_data);
	...
	cbdataLock(callback_data);
	fooOperationStart(bar, callback_func, callback_data);
	...
	cbdataFree(callback_data);
	...
	fooOperationComplete(...);
	if (cbdataValid(callback_data)) {
		callback_func(callback_data, ....);
	cbdataUnlock(callback_data);
</verb>
	In this case, when <tt/cbdataFree/ is called before 
	<tt/cbdataUnlock/, the callback_data gets marked as invalid.  Before
	executing the callback function, <tt/cbdataValid/ will return 0
	and callback_func is never executed.  When <tt/cbdataUnlock/ gets
	called, it notices that the callback_data is invalid and will
	then call <tt/cbdataFree/.

<!-- %%%% Chapter : CACHE MANAGER %%%% -->
<sect>Cache Manager

<!-- %%%% Chapter : HTTP Headers %%%% -->
<sect>HTTP Headers
<P>
<em/Files:/
	<tt/HttpHeader.c/,
	<tt/HttpHeaderTools.c/,
	<tt/HttpHdrCc.c/,
	<tt/HttpHdrContRange.c/,
	<tt/HttpHdrExtField.c/,
	<tt/HttpHdrRange.c/

<P> 
	<tt/HttpHeader/ class encapsulates methods and data for HTTP header
	manipulation.  <tt/HttpHeader/ can be viewed as a collection of HTTP
	header-fields with such common operations as add, delete, and find.
	Compared to an ascii "string" representation, <tt/HttpHeader/ performs
	those operations without rebuilding the underlying structures from
	scratch or searching through the entire "string".

<sect1>General remarks

<P>
	<tt/HttpHeader/ is a collection (or array) of HTTP header-fields. A header
	field is represented by an <tt/HttpHeaderEntry/ object. <tt/HttpHeaderEntry/ is
	an (id, name, value) triplet.  Meaningful "Id"s are defined for
	"well-known" header-fields like "Connection" or "Content-Length".
	When Squid fails to recognize a field, it uses special "id",
	<em/HDR_OTHER/.  Ids are formed by capitalizing the corresponding HTTP
	header-field name and replacing dashes ('-') with underscores ('_').

<P>
	Most operations on <tt/HttpHeader/ require a "known" id as a parameter. The
	rationale behind the later restriction is that Squid programmer should
	operate on "known" fields only. If a new field is being added to
	header processing, it must be given an id.
 
<sect1>Life cycle

<P> 
	<tt/HttpHeader/ follows a common pattern for object initialization and
	cleaning:

<verb>
    /* declare */
    HttpHeader hdr;
    
    /* initialize (as an HTTP Request header) */
    httpHeaderInit(&amp;hdr, hoRequest);

    /* do something */
    ...

    /* cleanup */
    httpHeaderClean(&amp;hdr);
</verb>

<P> 
	Prior to use, an <tt/HttpHeader/ must be initialized. A programmer must
	specify if a header belongs to a request or reply message. The
	"ownership" information is used mostly for statistical purposes.

<P>
	Once initialized, the <tt/HttpHeader/ object <em/must/ be, eventually,
	cleaned.  Failure to do so will result in a memory leak.

<P>
	Note that there are no methods for "creating" or "destroying" a
	"dynamic" <tt/HttpHeader/ object. Looks like headers are always stored as a
	part of another object or as a temporary variable. Thus, dynamic
	allocation of headers is not needed.


<sect1>Header Manipulation.

<P>
	The mostly common operations on HTTP headers are testing for a particular
	header-field (<tt/httpHeaderHas()/), extracting field-values (<tt/httpHeaderGet*()/), and
	adding new fields (<tt/httpHeaderPut*()/).

<P>
	<tt/httpHeaderHas(hdr, id)/ returns true if at least one header-field specified by
	"id" is present in the header. Note that using <em/HDR_OTHER/ as an id is
	prohibited.  There is usually no reason to know if there are "other"
	header-fields in a header.

<P>
	<tt/httpHeaderGet&lt;Type&gt;(hdr, id)/ returns the value of the specified header-field.
	The "Type" must match header-field type. If a header is not present a "null"
	value is returned. "Null" values depend on field-type, of course.

<P>
	Special care must be taken when several header-fields with the same id are
	preset in the header. If HTTP protocol allows only one copy of the specified
	field per header (e.g. "Content-Length"), <tt/httpHeaderGet&lt;Type&gt;()/ will return
	one of the field-values (chosen semi-randomly). If HTTP protocol allows for
	several values (e.g. "Accept"), a "String List" will be returned.

<P>
	It is prohibited to ask for a List of values when only one value is permitted,
	and visa-versa. This restriction prevents a programmer from processing one
	value of an header-field while ignoring other valid values.

<P>
	<tt/httpHeaderPut&lt;Type&gt;(hdr, id, value)/ will add an header-field with a specified
	field-name (based on "id") and field_value. The location of the newly added
	field in the header array is undefined, but it is guaranteed to be after all
	fields with the same "id" if any. Note that old header-fields with the same id
	(if any) are not altered in any way.

<P>
	The value being put using one of the <tt/httpHeaderPut()/ methods is converted to
	and stored as a String object.

<P>
	Example:

<verb>
	    /* add our own Age field if none was added before */
	    int age = ...
	    if (!httpHeaderHas(hdr, HDR_AGE))
		httpHeaderPutInt(hdr, HDR_AGE, age);
</verb>

<P>
	There are two ways to delete a field from a header. To delete a "known" field
	(a field with "id" other than <em/HDR_OTHER/), use <tt/httpHeaderDelById()/ function.
	Sometimes, it is convenient to delete all fields with a given name ("known" or
	not) using <tt/httpHeaderDelByName()/ method. Both methods will delete <em/all/ fields
	specified.


<P>

	The <em/httpHeaderGetEntry(hdr, pos)/ function can be used for
	iterating through all fields in a given header. Iteration is
	controlled by the <em/pos/ parameter. Thus, several concurrent
	iterations over one <em/hdr/ are possible. It is also safe to
	delete/add fields from/to <em/hdr/ while iteration is in progress.

<verb>
	/* delete all fields with a given name */
	HttpHeaderPos pos = HttpHeaderInitPos;
	HttpHeaderEntry *e;
	while ((e = httpHeaderGetEntry(hdr, &amp;pos))) {
		if (!strCaseCmp(e->name, name))
			... /* delete entry */
	}
</verb>

	Note that <em/httpHeaderGetEntry()/ is a low level function and must
	not be used if high level alternatives are available. For example, to
	delete an entry with a given name, use the <em/httpHeaderDelByName()/
	function rather than the loop above.

<sect1>I/O and Headers.

<P>
	To store a header in a file or socket, pack it using <tt/httpHeaderPackInto()/
	method and a corresponding "Packer". Note that <tt/httpHeaderPackInto/ will pack
	only header-fields; request-lines and status-lines are not prepended, and
	CRLF is not appended. Remember that neither of them is a part of HTTP
	message header as defined by the HTTP protocol.


<sect1>Adding new header-field ids.

<P> 
	Adding new ids is simple. First add new HDR_ entry to the http_hdr_type
	enumeration in enums.h. Then describe a new header-field attributes in
	the HeadersAttrs array located in <tt/HttpHeader.c/. The last
	attribute specifies field type. Five types are supported: integer
	(<em/ftInt/), string (<em/ftStr/), date in RFC 1123 format
	(<em/ftDate_1123/), cache control field (<em/ftPCc/), range field
	(<em/ftPRange/), and content range field (<em/ftPContRange/).  Squid
	uses type information to convert internal binary representation of
	fields to their string representation (<tt/httpHeaderPut/ functions)
	and visa-versa (<tt/httpHeaderGet/ functions).

<P>
	Finally, add new id to one of the following arrays:
	<em/GeneralHeadersArr/, <em/EntityHeadersArr/, <em/ReplyHeadersArr/,
	<em/RequestHeadersArr/.  Use HTTP specs to determine the applicable
	array.  If your header-field is an "extension-header", its place is in
	<em/ReplyHeadersArr/ and/or in <em/RequestHeadersArr/. You can also
	use <em/EntityHeadersArr/ for "extension-header"s that can be used
	both in replies and requests.  Header fields other than
	"extension-header"s must go to one and only one of the arrays
	mentioned above.

<P>
	Also, if the new field is a "list" header, add it to the
	<em/ListHeadersArr/ array.  A "list" field-header is the one that is
	defined (or can be defined) using "#" BNF construct described in the
	HTTP specs. Essentially, a field that may have more than one valid
	field-value in a single header is a "list" field.

<P>
	In most cases, if you forget to include a new field id in one of the required
	arrays, you will get a run-time assertion. For rarely used fields, however, it
	may take a long time for an assertion to be triggered.

<P>
	There is virtually no limit on the number of fields supported by Squid. If
	current mask sizes cannot fit all the ids (you will get an assertion if that
	happens), simply enlarge HttpHeaderMask type in <tt/typedefs.h/.


<sect1>A Word on Efficiency.

<P>
	<tt/httpHeaderHas()/ is a very cheap (fast) operation implemented using a bit mask
	lookup.

<P>
	Adding new fields is somewhat expensive if they require complex conversions to
	a string.

<P>
	Deleting existing fields requires scan of all the entries and comparing their
	"id"s (faster) or "names" (slower) with the one specified for deletion.

<P>
	Most of the operations are faster than their "ascii string" equivalents.

</article>