bat5.mx

一个内存数据库的源代码这是服务器端还有客户端

@' The contents of this file are subject to the MonetDB Public License
@' Version 1.1 (the "License"); you may not use this file except in
@' compliance with the License. You may obtain a copy of the License at
@' http://monetdb.cwi.nl/Legal/MonetDBLicense-1.1.html
@'
@' Software distributed under the License is distributed on an "AS IS"
@' basis, WITHOUT WARRANTY OF ANY KIND, either express or implied. See the
@' License for the specific language governing rights and limitations
@' under the License.
@'
@' The Original Code is the MonetDB Database System.
@'
@' The Initial Developer of the Original Code is CWI.
@' Portions created by CWI are Copyright (C) 1997-2007 CWI.
@' All Rights Reserved.

@f bat5
@v 2.0
@a Peter Boncz, M.L. Kersten
@+ Binary Association Tables
This module contains the commands and patterns to manage Binary
Association Tables (BATs). The relational operations you can execute
on BATs have the form of a neat algebra, described in algebra.mx

But a database system needs more that just this algebra, since often it
is crucial to do table-updates (this would not be permitted in a strict
algebra).

All commands needed for BAT updates, property management, basic I/O, 
persistence, and storage options can be found in this module.

All parameters to the modules are passed by reference.
In particular, this means that string values are passed to the module
layer as (str *)
and we have to de-reference them before entering the gdk library.
(Actual a design error in gdk to differentiate passing int/str)
This calls for knowledge on the underlying BAT types`s
@-
@{
@= derefStr
{ if( @1->@2type >= TYPE_str  && ATOMstorage(@1->@2type) >= TYPE_str)
 { if(@3== 0 || *(str*)@3==0) @3 = (str)str_nil;
   else @3 = *(str *)@3; 
}}
@-
The code speaks for itself
@mal


#command bat(ht:int, tt:int) :bat[:any_1,:any_2] 
#address BKCnewBat
#comment "Creates a new empty transient BAT with head- and tail-types 
#        as indicated.";
#
#command bat(ht:int, tt:int, size:int) :bat[:any_1,:any_2] 
#address BKCnewBATint
#comment "Creates a new empty BAT, allocating 'size' elements.";
#
#command bat(ht:int, tt:int, size:lng) :bat[:any_1,:any_2] 
#address BKCnewBATlng
#comment "Creates a new empty BAT, allocating 'size' elements.";

module bat;
command reverse(b:bat[:any_1,:any_2]) :bat[:any_2,:any_1] 
address BKCreverse
comment "Returns the reverse view of a BAT (head is tail and tail is head).
        BEWARE  no copying is involved; input and output refer to the 
        same object!";

command mirror(b:bat[:any_1,:any_2]) :bat[:any_1,:any_1] 
address BKCmirror
comment "Returns the head-mirror image of a BAT (two head columns).";

#command convert(b:bat[:any_1,:any_2] ) :bat[:any_1,:any_2] 
#address BKCconvert
#comment "Convert the contents of a BAT from little-endian to big-endian 
#        and vice versa. THIS command SHOULD NOT BE USED FROM MAL!
#	DANGEROUS! DEBUGGING PURPOSES ONLY!";

command order(b:bat[:any_1,:any_2]) :bat[:any_1,:any_2] 
address BKCorder
comment "Sorts the BAT itself on the head, in place. ";
command orderReverse(b:bat[:any_1,:any_2]) :bat[:any_1,:any_2] 
address BKCorder_rev
comment "Reverse sorts the BAT itself on the head, in place. ";

command revert(b:bat[:any_1,:any_2]) :bat[:any_1,:any_2] 
address BKCrevert
comment "Puts all BUNs in a BAT in reverse order.  (Belongs to the 
        BAT sequence module)";

@+ BAT properties
Properties of BATs not necessarily require loading the BAT completely.
The BAT identifier can be used to access the descriptors.
These properties can be inspected with the 
@emph{info(BAT[:any_1,:any_2]) :bat[str,str] } command:
@verbatim
> car_age.info.print;
#---------------------------------#
# BAT:               tmp_31       #
# (str)              (str)        #
#---------------------------------#
[ "batId",           "car_age"    ] # logical bat name
[ "batCacheid",      "26"         ] # BBP index
[ "batParentid",     "0"          ] # set if a BAT is a view
[ "head",            "void"       ] # physical head-type
[ "tail",            "int"        ] # physical tail-type
[ "batPersistence",  "persistent" ] # transient/session/persistent
[ "batRestricted",   "updatable"  ] # read-only/append-only/updatable
[ "batDirty",        "clean"      ] # clean/dirty
[ "batRefcnt",       "1"          ] # physical refcount
[ "batLRefcnt",      "1"          ] # logical refcount (total)
[ "batPlevel",       "1"          ] # logical refcount (persistent part)
[ "batSet",          "0"          ] # [head,tail] combinations are unique
[ "batCopiedtodisk", "1"          ] # has been saved or not
@end verbatim

Per column, a number of properties are kept. We only show the head-properties; the tail properties
have the first character of their named replaced by 't' (@emph{sorted}, @emph{tdense}, etc.):
@verbatim
[ "hsorted",         "1"          ] # column is known to be sorted
[ "hdense",          "1"          ] # column is known to be densely ascended
[ "hseqbase",        "0@0"        ] # if densely ascending first value
@end verbatim

As described in the @[<a href="www/gdk.html#mod_1_3_0">GDK Technical Reference</a>@,
BATs store their data in one to five heaps, of which at least one called @emph{batBuns}
is present always. For each column type that is @emph{variable-sized} - like @emph{str} - another
heap might be present (@emph{hheap} and @emph{theap}). 

@verbatim
[ "batBuns.free",    "400004"     ] # occupied size in bytes
[ "batBuns.size",    "400012"     ] # allocated size in bytes
[ "batBuns.maxsize", "400012"     ] # reserver virtual memory in bytes
[ "batBuns.storage", "malloced"   ] # malloced/mmap/priv
@end verbatim

Properties steer the algorithms that Monet uses to execute algebra command. For instance,
the @emph{find(bat[:any_1,:any_2] b, :any_1) :any_2} that looks up a tail value by head,
uses binary search if and only if the head column is sorted (in other cases, hash-lookup
and scan are execution options).

Sometimes new (extension) code contains bugs in the property management, leaving false
properties on produced BATs. You can imagine that later on, this leads to additional
bugs, as e,g, the binary search algorithms will yield erroneous results on a non-sorted
sequence.

@mal
command info ( b:bat[:any_1,:any_2]) :bat[:str,:str] 
address BKCinfo
comment "Produce a BAT containing info about a BAT in [attribute,value] format. 
        It contains all properties of the BAT record. See the BAT documentation 
        in GDK for more information.";

command getCapacity(b:bat[:any_1,:any_2]):lng 
address BKCgetCapacity
comment "Returns the current allocation size (in max number of elements) of a BAT.";

command getHeadType(b:bat[:any_1,:any_2] ) :str 
address BKCgetHeadType
comment "Returns the type of the head column of a BAT, as an integer type number.";

command getTailType( b:bat[:any_1,:any_2] ) :str 
address BKCgetTailType 
comment "Returns the type of the tail column of a BAT, as an integer type number.";

command getRole ( bid:bat[:any_1,:any_2] ) :str 
address BKCgetRole
comment "Returns the rolename of the head column of a BAT.";

command setKey( b:bat[:any_1,:any_2], mode:bit) :bat[:any_1,:any_2] 
address BKCsetkey
comment "Sets the 'key' property of the head column to 'mode'. In 'key' mode, 
        the kernel will silently block insertions that cause a duplicate 
        entries in the head column. KNOWN BUG:when 'key' is set to TRUE, this 
	function does not automatically eliminate duplicates. 
        Use b := b.kunique;";

command isaKey( b:bat[:any_1,:any_2]) :bit 
address BKCgetKey
comment "return whether the head column of a BAT is unique (key).";

command setSet( b:bat[:any_1,:any_2], mode:bit) :bat[:any_1,:any_2] 
address BKCsetSet
comment "Sets the 'set' property on this BAT to 'mode'. In 'set' mode, 
        the kernel will silently block insertions that cause a duplicate 
        BUN [head,tail] entries in the BAT.  KNOWN BUG:when 'set' is set 
        to TRUE, this function does not automatically eliminate duplicates. 
        Use b := b.sunique; Returns the BAT itself.";

command isaSet( b:bat[:any_1,:any_1]) :bit 
address BKCisaSet
comment "return whether the BAT mode is set to unique.";

command setAccess( b:bat[:any_1,:any_2], mode:str) :bat[:any_1,:any_2]
address BKCsetAccess
comment "Try to change the update access priviliges 
	to this BAT. Mode:
	 r[ead-only]      - allow only read access.
	 a[append-only]   - allow reads and update.
	 w[riteable]      - allow all operations.
	BATs are updatable by default. On making a BAT read-only, 
        all subsequent updates fail with an error message.Returns 
        the BAT itself.";
#command access( b:bat[:any_1,:any_2], mode:int) :bat[:any_1,:any_2]
#address BKCaccess;

command setAppendMode( b:bat[:any_1,:any_2]) :bat[:any_1,:any_2]
address BKCsetAppendMode
comment "Change access privilige of BAT to append only";

command setReadMode( b:bat[:any_1,:any_2]) :bat[:any_1,:any_2]
address BKCsetReadMode
comment "Change access privilige of BAT to read only";

command setWriteMode( b:bat[:any_1,:any_2]) :bat[:any_1,:any_2]
address BKCsetWriteMode
comment "Change access privilige of BAT to read and write";

command getAccess( b:bat[:any_1,:any_2]):str 
address BKCgetAccess
comment "return the access mode attached to this BAT as a character.";

command hasAppendMode( b:bat[:any_1,:any_2]):bit 
address BKChasAppendMode
comment "return true if to this BAT is append only.";

command hasWriteMode( b:bat[:any_1,:any_2]):bit 
address BKChasWriteMode
comment "return true if to this BAT is read and write.";

command hasReadMode( b:bat[:any_1,:any_2]):bit 
address BKChasReadMode
comment "return true if to this BAT is read only.";


command getSequenceBase( b:bat[:oid,:any_1]):oid 
address BKCgetSequenceBase
comment "Get the sequence base for the void column of a BAT.";

#command setSequenceBase( b:bat[:oid,:any_1], base:oid):void
#address BKCsetSequenceBase
#comment "Set the sequence base for the void column of a BAT.";

command setSorted(b:bat[:any_1,:any_2]) :bit 
address BKCsetSorted
comment "Assure BAT is ordered on the head.";

command isSorted(b:bat[:any_1,:any_2]) :bit 
address BKCisSorted
comment "Returns whether a BAT is ordered on head or not.";
command isSortedReverse(b:bat[:any_1,:any_2]) :bit 
address BKCisSortedReverse
comment "Returns whether a BAT is ordered on head or not.";

command getBatSize(b:bat[:any_1,:any_2]) :lng 
address BKCbatsize
comment "A version of BATsize that does not require loading the BAT.";

@- BAT updates
Update commands come in many disguises.  Note that we don;t return
the BAT id, but merely a success/failure code.
@mal
command insert(b:bat[:any_1,:any_2], src:bat[:any_1,:any_2]):void
address BKCinsert_bat
comment "Insert all BUNs of the second BAT into the first.";

command insert(b:bat[:any_1,:any_2], src:bat[:any_1,:any_2], force:bit):void
address BKCinsert_bat_force
comment "Insert all BUNs of the second BAT into the first.";

command insert(b:bat[:any_1,:any_2], h:any_1, t:any_2) :void
address BKCinsert_bun
comment "Insert one BUN[h,t] in a BAT.";

command insert(b:bat[:any_1,:any_2], h:any_1, t:any_2, force:bit) :void
address BKCinsert_bun_force
comment "Insert one BUN[h,t] in a BAT.";

@+
@mal
command replace(b:bat[:any_1, :any_2], src:bat[:any_1,:any_2]) :void
address BKCreplace_bat
comment "Perform replace for all BUNs of the second BAT into the first.";

command replace(b:bat[:any_1, :any_2], src:bat[:any_1,:any_2], force:bit) :void
address BKCreplace_bat_force
comment "Perform replace for all BUNs of the second BAT into the first.";

command replace(b:bat[:any_1, :any_2], h:any_1, t:any_2) :void
address BKCreplace_bun
comment "Replace the tail value of one BUN that has some head value.";

command replace(b:bat[:any_1, :any_2], h:any_1, t:any_2, force:bit) :void
address BKCreplace_bun_force
comment "Replace the tail value of one BUN that has some head value.";
@-
The SQL append/inplace commands
@mal
command append( i:bat[:any_1,:any_2], u:bat[:any_1,:any_2] ) :void
address BKCappend_wrap
comment "append the content of u to i";

command append( i:bat[:any_1,:any_2], u:bat[:any_1,:any_2], force:bit ) :void
address BKCappend_force_wrap
comment "append the content of u to i";

command append(i:bat[:oid,:any_1], u:any_1):void
address BKCappend_val_wrap
comment "append the value u to i";

command append(i:bat[:void,:any_1], u:any_1):void
address BKCappend_val_wrap
comment "append the value u to i";

command append(i:bat[:any_1,:void], u:any_1):void
address BKCappend_reverse_val_wrap
comment "append the value u to i";

command append(i:bat[:any_1,:any_2], u:any_2, force:bit):void
address BKCappend_val_force_wrap
comment "append the value u to i";

command inplace( o:bat[:any_1,:any_2], id:any_1, t:any_2) :void
address BKCbun_inplace
comment "inplace replace values on the given locations";

command inplace( o:bat[:any_1,:any_2], d:bat[:any_1,:any_2]) :void
address BKCbat_inplace
comment "inplace replace values on the given locations";

command inplace( o:bat[:any_1,:any_2], id:any_1, t:any_2, force:bit) :void
address BKCbun_inplace_force
comment "inplace replace values on the given locations";

command inplace( o:bat[:any_1,:any_2], d:bat[:any_1,:any_2], force:bit) :void
address BKCbat_inplace_force
comment "inplace replace values on the given locations";

@mal
command delete(b:bat[:any_1, :any_2], h:any_1, t:any_2) :void
address BKCdelete_bun
comment "Delete one specific BUN.";

command delete(b:bat[:any_1, :any_2], h:any_1) :void
address BKCdelete
comment "Delete all BUNs with a certain tail value.";

command delete(b:bat[:any_1, :any_2]) :void
address BKCdelete_all
comment "Delete all BUNs in a BAT.";

@mal
command delete(b:bat[:any_1, :any_2], src:bat[:any_1,:any_2]) :void
address BKCdelete_bat_bun
comment "Delete from the first BAT all BUNs with a corresponding BUN 
        in the second.";

command getAlpha(b:bat[:any_1,:any_2]) :bat[:any_1,:any_2] 
address BKCgetAlpha
comment "Obtain the list of BUNs added";
command getDelta(b:bat[:any_1,:any_2]) :bat[:any_1,:any_2] 
address BKCgetDelta
comment "Obtain the list of BUNs deleted";
@- BAT I/O, Persistency
The BAT Buffer Pool (BBP) manages all known BATs. It administers
their logical and physical names and a reference count. BATs can
either be @emph{persistent} or @emph{transient}. The BBP also manages
swapping on a BAT level:a BAT is either loaded entirely or not.
MAL variables of type @emph{bat} can either be loaded or not. When
the Monet server is started, all BATs are swapped out. If an unloaded
bat-variable is used as an operand in a command, it is automatically
loaded.  The BBP applies a simple but effective LRU based swapping
algorithm. BATs have a @emph{heat}, which drops over time, and is
increased when a BAT is used by some command.  If the size of the
allocated arena gets to be large, BATs may be swapped out.

Note, we should move old-fashioned mil operator definitions
into a separate module.

@mal
command setName ( b:bat[:any_1,:any_2] , s:str) :void
address BKCsetName
comment "Give a logical name to a BAT. ";

command getName ( b:bat[:any_1,:any_2]) :str 
address BKCgetBBPname
comment "Gives back the logical name of a BAT.";

command setRole( b:bat[:any_1,:any_2], h:str, t:str) :void
address BKCsetRole
comment "Give a logical name to the columns of a BAT.";

command setColumn( b:bat[:any_1,:any_2], t:str) :void
address BKCsetColumn
comment "Give a logical name to the tail column of a BAT.";
command setColumn( b:bat[:any_1,:any_2], h:str, t:str) :void
address BKCsetColumns
comment "Give both columns of a BAT a new name.";


command isTransient( b:bat[:any_1,:any_2]) :bit 
address BKCisTransient;
command setTransient( b:bat[:any_1,:any_2]) :void
address BKCsetTransient
comment "Make the BAT transient.  Returns 
	boolean which indicates if the
BAT administration has indeed changed.";

command isPersistent( b:bat[:any_1,:any_2]) :bit 
address BKCisPersistent;
command setPersistent( b:bat[:any_1,:any_2],f:bit) :void
address BKCpersists
comment "Backward compatibility";
command setPersistent( b:bat[:any_1,:any_2]) :void
address BKCsetPersistent
comment "Make the BAT persistent.  Returns boolean which indicates 
        if the BAT administration has indeed changed.";

command save(nme:bat[:any_1,:any_2]) :void
address BKCsave2;

command save(nme:str) :bit 
address BKCsave
comment "Save a BAT to storage, if it was loaded and dirty.  
        Returns whether IO was necessary.  Please realize that 
	calling this function violates the atomic commit protocol!!";


command load(name:str) :bat[:any_1,:any_2] 
address BKCload
comment "Load a particular BAT from disk";

command unload(name:str) :bit 
address BKCunload
comment "Swapout a BAT to disk. Transient BATs can also be swapped out.     
        Returns whether the unload indeed happened. ";

command isCached(b:bat[:any_1,:any_2]):bit 
address BKCisCached
comment "Bat is stored in main memory.";

command getHeat(b:bat[:any_1,:any_2]) :lng 
address BKCheat
comment "Return the current BBP heat (LRU stamp)";

command setCold(b:bat[:any_1,:any_1]) :void 
address BKCcoldBAT
comment "Makes a BAT very cold for the BBP. The chance of being choses 
        for swapout is big, afterwards.";
command setHot(b:bat[:any_1,:any_2]) :void 
address BKChotBAT
comment "Makes a BAT very hot for the BBP. The chance of being chosen for 
        swapout is small, afterwards.";

@- Heap Specific Commands
BATs are stored in memory in a number of Heap objects. A heap is nothing
more than a contiguous range of memory. Bats are saved to disk by just writing
away their image. This approach without pointer swizzling makes it possible
to either load an image into an alloced range of memory (STORE_MEM), or
memory-map (STORE_MMAP) an image into virtual memory.

The heap images of a BAT are stored in the @emph{$MONETHOME/dbfarm/$DB/bat/}
directory.  For each bat X, the following heaps are stored:
@table @code
@item[X.buns] 
    an array with all the fixed-size parts of all BUNs.
@item[X.hheap] 
    if the head column contains a variable sized atoms (e.g. str),
then the fixed-size part of a BUN contains an integer byte-offset into
the heap. String values themself are stored in this the X.hheap.
@item[X.theap] 
    similar to X.hheap, but for the tail column.
@item[X.desc] 
the BAT descriptor. Stores most of the properties of a BAT.
@end table

Each of these heaps can be compressed using the Unix @emph{compress}
utility forming a X.ext.Z file. Monet will automatically decompress it
upon load. Compressed heaps cannot be memory mapped.

@
For more technical information on BATs, we refer to the
@[<a href="http://www.cwi.nl/~monet/www/scw/gdk/470_pseudo.html">GDK</a>@
documentation.
@mal
command setMemoryMap(b:bat[:any_1,:any_2], buns_mode:int, hheap_mode:int, 
		theap_mode:int) :bit 
address BKCmmap
comment "For each individual heap, you can change the allocation mode 
        to either STORE_MEM or STORE_MMAP. Passing an int(nil) means:
        no change.  Changing a dirty STORE_MEM heap into STORE_MMAP, 
	will cause a BAT save (this has to happen before the heap can 
        be mapped into virtual memory). These modes are persistent. ";

command setMemoryMap(b:bat[:any_1,:any_2], mode:int):bit 
address BKCmmap2