metakit-fileformat - metakit database system.mht

TCL的数据库处理支撑库及一些示例

  0x00 at its beginning. This definition is necessary to allow =
unambiguous=20
  separation of <STRONG>bpiSignByte</STRONG> and =
<STRONG>bpiDataByte</STRONG>.=20
  <STRONG>bpInt</STRONG>'s are always stored in big-endian format. In =
other=20
  words, the MSB is stored first, followed by the lesser bits. <BR><BR>
  <DL>
    <DT><STRONG>bpiSignByte</STRONG>
    <DD>::=3D byte /=3D 0x00 <BR><BR>The sign byte is an optional part =
of instances=20
    of <STRONG>bpInt</STRONG>. Its presence indicates that the following =

    <STRONG>bpiDataByte</STRONG> and <STRONG>bpiStopByte</STRONG> =
contain a=20
    negative number and that the actual value in 2-complement =
representation is=20
    obtained by negating all bits in the expanded number (1-complement). =

<BR><BR>
    <DT><STRONG>bpiDataByte</STRONG>
    <DD>::=3D byte /in 0x00 .. 0x7f <BR><BR>The meat of any instance of=20
    <STRONG>bpInt</STRONG>. Bit 7 is set to 0, indicating that more =
bytes=20
    follows. Bit 6 to 0 are the payload, a fragment of the 2-complement=20
    representation of the number stored in the <STRONG>bpInt</STRONG>. =
<BR><BR>
    <DT><STRONG>bpiStopByte</STRONG>
    <DD>::=3D byte /in 0x80 .. 0xff <BR><BR>The last (and possibly the =
only) byte=20
    in any instance of <STRONG>bpInt</STRONG>. Bit 7 is set to 1, as =
indicator=20
    that this is the last byte in <STRONG>bpInt</STRONG>. Bit 6 to 0 are =
the=20
    payload, the last fragment of the 2-complement representation of the =
number=20
    stored in the instance of <STRONG>bpInt</STRONG>. =
</DD></DL><BR><BR>Examples:=20
  <BR><BR>
  <P>
  <TABLE>
    <TBODY>
    <TR>
      <TD bgColor=3Dblack>&nbsp;</TD>
      <TD><PRE class=3Dsample>        0x80 : First is last byte,
               payload =3D 00000000/2 =3D 0/10

        0x00 0x80 : Negative number
                    payload      =3D 0000000/2
                    1-complement =3D 1111111/0 =3D -1/10

        0x94 : First is last byte.
               payload 0011000/2 =3D 20/10

        0x00 0x94 : Negative number
                    payload      =3D 0011000/2
                    1-complement =3D 1100111/2 =3D -21/10
</PRE></TD></TR></TBODY></TABLE></P><BR><BR>
  <DT><STRONG>Char</STRONG>
  <DD>::=3D { byte }{1,4} <BR><BR>A character in UTF-8 encoding, taking =
up one to=20
  four bytes. <BR><BR>
  <DT><STRONG>pString</STRONG>
  <DD>::=3D bpInt { Char } <BR><BR>A string of characters, with prefixed =
length=20
  information. This is the Pascal notation, hence the =
<STRONG>p</STRONG>.=20
  <EM>Note</EM> that the length is specified as the number of bytes in =
the=20
  string, and not the number of characters. However as the contents have =
to be a=20
  sequence of valid UTF-8 characters this symbol was used in the =
definition=20
  instead of the more generic <STRONG>byte</STRONG>. <BR><BR>
  <DT><STRONG>cString</STRONG>
  <DD>::=3D { Char } '\0' <BR><BR>A string of characters, terminated by =
a=20
  null-byte. This is C notation. <BR><BR>
  <DT><STRONG>UnknownLong</STRONG>
  <DD>::=3D Long <BR><BR>Usage in a definition implies that the purpose =
of the=20
  element is not known. <BR><BR>
  <DT><STRONG>Header</STRONG>
  <DD>::=3D Magic Valid HeaderType FooterLocation <BR><BR>
  <DL>
    <DT><STRONG>Magic</STRONG>
    <DD>::=3D byte byte /=3D (0x4a 0x4c =3D "JL" | 0x4c 0x4a =3D "LJ") =
<BR><BR>These two=20
    magic bytes signal the beginning of a metakit database, and =
additionally the=20
    endianess of <STRONG>Word</STRONG> and <STRONG>Long</STRONG> values. =
The=20
    first form, "JL", indicates a little endian format, the other one =
indicates=20
    a big endian format. <BR><BR>
    <DT><STRONG>Valid</STRONG>
    <DD>::=3D byte /=3D 0x1a Indicates that the header is valid. No =
other value is=20
    known. <BR><BR>
    <DT><STRONG>HeaderType</STRONG>
    <DD>::=3D byte /in { 0x00, 0x80 } <BR><BR>The value 0x00 indicates a =
new-style=20
    header whereas 0x80 signals usage of an old-style header. The format =
of=20
    old-style headers is not known, therefore the presence of 0x80 can =
be=20
    treated as an error. <BR><BR>
    <DT><STRONG>FooterLocation</STRONG>
    <DD>::=3D Long <BR><BR>A pointer to the <STRONG>Footer</STRONG> of =
the=20
    database. Through that the <STRONG>TableOfContents</STRONG> can be =
reached=20
    and from there all of the data stored in the file or serialization. =
</DD></DL>
  <DT><STRONG>Footer</STRONG>
  <DD>::=3D UnknownLong HeaderLocation UnknownLong TOCLocation <BR><BR>
  <DL>
    <DT><STRONG>HeaderLocation</STRONG>
    <DD>::=3D Long <BR><BR>Offset to the beginning of the metakit=20
    <STRONG>Header</STRONG> in the file. The offset is counted backwards =
from=20
    the beginning of the footer. As this is 16 bytes from the end of the =

    <STRONG>Serialization</STRONG>, the actual location of the beginning =
of the=20
    header is "<STRONG>HeaderLocation</STRONG> + 16" bytes backward from =
the end=20
    of the serialization. <BR><BR>
    <DT><STRONG>TOCLocation</STRONG>
    <DD>::=3D Long Pointer to the beginning of the=20
    <STRONG>TableOfContents</STRONG> in the metakit file. Usable only =
after=20
    <STRONG>HeaderLocation</STRONG> was decoded into the absolute =
location of=20
    the beginning of the metakit header in the file. It is expected that =
the end=20
    of the <STRONG>TableOfContents</STRONG> coincides with the beginning =
of the=20
    <STRONG>Footer</STRONG>. </DD></DL><BR><BR>Because of the two =
offsets in the=20
  footer it is possible to append a metakit database to any file and =
still be=20
  able to access all of its contents. It is this property which makes =
the usage=20
  of metakit databases as a file system for starkits and starpacks =
possible.=20
  <BR><BR>
  <DT><STRONG>IVecData</STRONG>
  <DD>::=3D IVecIntData | IVecFloatData | IVecDoubleData | IVecLongData =
|=20
  IVecBinaryData | IVecCatalogData | IVecSubviewData <BR><BR>
  <DL>
    <DT><STRONG>IVecIntData</STRONG>
    <DD>::=3D { byte } <BR><BR>The itemvector is used to compactly store =
integer=20
    numbers in 2-complement representation. Depending on the maximal =
absolute=20
    value over all numbers in the itemvector the metakit library will =
allocate=20
    1, 2, 4, 8, 16, or 32 bits per item, for all items. The latter means =
that=20
    all items in this itemvector will have the same size. <BR><BR>How =
many bits=20
    per item were allocated can be computed from the number of rows in =
the view=20
    containing the associated column and the size of the itemvector in =
bytes.=20
    <BR><BR>The data of all items is stored back to back as usual, with =
a little=20
    twist. Multiple items are packed into a single byte if the number of =
bits=20
    per item is less than eight. <BR><BR>This format is called =
<EM>adaptive=20
    integers</EM>. It should not be confused with octet-packed integers=20
    (<STRONG>bpInt</STRONG>). <BR><BR><STRONG>IVecIntData</STRONG> =
itemvectors=20
    are used to store the data of integer columns (type =
<STRONG>I</STRONG>), and=20
    also, in conjunction with <STRONG>IVecCatalogData</STRONG> and=20
    <STRONG>IVecBinaryData</STRONG>, to store the data of binary and =
string=20
    columns (types <STRONG>S</STRONG> and <STRONG>B</STRONG>). See =
section <A=20
    =
href=3D"http://www.equi4.com/metakit/metakit-ff.html#variablesizeddata">V=
ARIABLE=20
    SIZED DATA</A>. <BR><BR>
    <DT><STRONG>IVecFloatData</STRONG>
    <DD>::=3D { Long } <BR><BR>Each item is a single precision floating =
point=20
    number stored in 4 bytes. The items are stored back to back. The =
vectors are=20
    used to store the data for columns of type <STRONG>F</STRONG>. =
<BR><BR>
    <DT><STRONG>IVecDoubleData</STRONG>
    <DD>::=3D { {Long}{2} } <BR><BR>Each item is a double precision =
floating point=20
    number stored in 8 bytes. The items are stored back to back. The =
vectors are=20
    used to store the data for columns of type <STRONG>D</STRONG>. =
<BR><BR>
    <DT><STRONG>IVecLongData</STRONG>
    <DD>::=3D { {Long}{2} } <BR><BR>Each item is a 64-bit integer number =
in=20
    2-complement representation stored in 8 bytes. The item are stored =
back to=20
    back. The vectors are used to store the data for columns of type=20
    <STRONG>L</STRONG>. <BR><BR>
    <DT><STRONG>IVecBinaryData</STRONG>
    <DD>::=3D { byte } | cString <BR><BR>Each item is a variable-sized =
block of=20
    bytes. The length of each item is stored in a separate itemvector, =
an=20
    instance of <STRONG>IVecIntData</STRONG>. The connection between the =
two=20
    itemvectors is made through an instance of =
<STRONG>VariableMap</STRONG>.=20
    <BR><BR>The items are always stored back to back. However in the =
case of=20
    strings each item is actually a <STRONG>cString</STRONG>. <BR><BR>
    <DT><STRONG>IVecCatalogData</STRONG>
    <DD>::=3D { SkipCount OutlineRef } { IVecBinaryData } <BR><BR>See =
also section=20
    <A=20
    =
href=3D"http://www.equi4.com/metakit/metakit-ff.html#variablesizeddata">V=
ARIABLE=20
    SIZED DATA</A>. <BR><BR>The part containing additional binary data =
is=20
    present only if one or more of the <EM>indirect</EM> data-blocks =
listed in=20
    the catalog coming before it are marked as having location 0. =
<EM>Note</EM>=20
    that the size of this binary section is <EM>never</EM> counted in =
the size=20
    of the itemvector itself. In that respect the binary data is not =
part of the=20
    itemvector, although when reading them it is easier to pretend that =
they are=20
    part of the itemvector. This makes it possible to determine the =
border=20
    between the catalog items and the associated binary data without =
complicated=20
    calculations. <BR><BR>Each such itemvector is associated with two =
other=20
    itemvectors holding the information for the directly stored data of =
their=20
    column (size and the data itself)). This association is made through =
an=20
    instance of <STRONG>VariableMap</STRONG>. <BR><BR>
    <DL>
      <DT><STRONG>SkipCount</STRONG>
      <DD>::=3D bpInt <BR><BR>Declares how many items to skip in the =
associated=20
      itemvector for the directly stored data to reach this item. =
<BR><BR>
      <DT><STRONG>OutlineRef</STRONG>
      <DD>::=3D IVecRef <BR><BR>This refers to the block of bytes =
actually holding=20
      the data for the described cell. This block is an instance of=20
      <STRONG>IVecBinaryData</STRONG>. The referred itemvector contains =
only the=20
      data for this item and nothing else. If the location is 0, then =
the data=20
      is <EM>not</EM> stored in a separate itemvector, but inside of the =

      <STRONG>IVecBinaryData</STRONG> section of the catalog itemvector =
itself.=20
      <BR><BR></DD></DL>
    <DT><STRONG>IVecSubviewData</STRONG>
    <DD>::=3D { SubviewMap } <BR><BR>Each item in the vector represents =
a whole=20
    subview and does so by containing a mapping from the columns the =
subview=20
    consists of to the itemvectors storing the subview data for these =
columns.=20
    See <STRONG>Mapping</STRONG> for the definition of symbol=20
    <STRONG>SubviewMap</STRONG>. <BR><BR></DD></DL>
  <DT><STRONG>IVecRef</STRONG>
  <DD>::=3D IVecSize [ IVecLocation ] <BR><BR>
  <DL>
    <DT><STRONG>IVecSize</STRONG>
    <DD>::=3D bpInt <BR><BR>Size of the referenced itemvector, in bytes. =
If the=20
    value is zero a location is irrelevant and not stored. <BR><BR>
    <DT><STRONG>IVecLocation</STRONG>
    <DD>::=3D bpInt <BR><BR>Pointer to the beginning of the itemvector. =
Not=20
    present if the itemvector is empty (size =3D=3D 0). See=20
    <STRONG>IVecData</STRONG> for the legal internal structures of =
itemvectors.=20
    <BR><BR></DD></DL>
  <DT><STRONG>ColumnMap</STRONG>
  <DD>::=3D FixedMap | VariableMap <BR><BR>See also section <A=20
  =
href=3D"http://www.equi4.com/metakit/metakit-ff.html#columnmapping">COLUM=
N=20