gdk.mx

来自「这个是内存数据库中的一个管理工具」· MX 代码 · 共 1,875 行 · 第 1/5 页

@item BAT *
@tab BATsave (BAT *b)
@item int
@tab BATmmap (BAT *b, int bn, int hh, int th )
@item int
@tab BATmadvise (BAT *b, int bn, int hh, int th )
@item int
@tab BATmmap_pin (BAT *b)
@item int
@tab BATmmap_unpin (BAT *b)
@item int
@tab BATdelete (BAT *b)
@item BAT *
@tab BATconvert (str name, int direction)
@end multitable
@-
A BAT created by @%BATnew@ is considered temporary until one calls the
routine @%BATsave@ or @%BATmode@.  This routine reserves disk space
and checks for name clashes in the BAT directory. It also makes the
BAT persistent. The empty BAT is initially marked as ordered on both
columns.
@
Failure to read or write the BAT results in a NULL, otherwise it
returns the BAT pointer.
@
The @%BATconvert@ converts a BAT that contains data saved on
hardware with a different byte order (little endian vs big endian),
into the correct format. Direction is either CONV_NTOH or CONV_HTON,
indication conversion from network to host or host to network format.
@
MonetDB now has a mmap trim thread that takes care of flushing the
memory mapped regions when MonetDB starts to consume too much
main memory. Heaps (that are randomly accessed) can be excluded
from this mechanism, by pinning them. BATmmap\_pin/unpin do
this for all heaps of a BAT.

@- Heap Storage Modes
The discriminative storage modes are memory-mapped, compressed, or 
loaded in memory.
The @strong{BATmmap()} changes the storage mode of each heap associated to 
a BAT.
As can be seen in the bat record, each BAT has one BUN-heap (@emph{bn}), and
possibly two heaps (@emph{hh} and @emph{th}) for variable-sized atoms.
@
The @%BATmadvise@ call works in the same way. Using the @%madvise()@ system
call it issues buffer management advise to the OS kernel, as for the
expected usage pattern of the memory in a heap.
@{
@h
/* Heap storage modes */
#define STORE_MEM	0	/* load into GDKmalloced memory */
#define STORE_MMAP	1	/* mmap() into virtual memory */
#define STORE_PRIV	2	/* BAT copy of copy-on-write mmap */
#define STORE_UNSET	((char)-1)/* no mode specified */

gdk_export size_t    GDK_mem_pagebits; /* page size for non-linear mmaps */
#define REMAP_PAGE_BITS    GDK_mem_pagebits
#define REMAP_PAGE_SIZE    (1 << REMAP_PAGE_BITS)
#define REMAP_PAGE_MASK    (REMAP_PAGE_SIZE - 1)
#define REMAP_PAGE_MAXBITS (REMAP_PAGE_BITS+3)
#define REMAP_PAGE_MAXSIZE (1<<REMAP_PAGE_MAXBITS) /* max page bytesize of unary BUN heap (8-byte atom) */

/* Buffer management advise for heaps */
#define BUF_NORMAL	0	/* No further special treatment */
#define BUF_RANDOM	1	/* Expect random page references */
#define BUF_SEQUENTIAL	2	/* Expect sequential page references */
#define BUF_WILLNEED	3	/* Will need these pages */
#define BUF_DONTNEED	4	/* Don't need these pages */

gdk_export BAT *BATload(str nme);
gdk_export BAT *BATload_intern(bat bid);
gdk_export BAT *BATsave(BAT *b);
gdk_export int BATmmap(BAT *b, int bn, int hh, int th );
gdk_export int BATmmap_pin(BAT *b);
gdk_export int BATmmap_unpin(BAT *b);
gdk_export int BATmadvise(BAT *b, int bn, int hh, int th );
gdk_export int BATdelete(BAT *b);
gdk_export size_t BATvmsize(BAT *b, int dirty);
gdk_export size_t BATmemsize(BAT *b, int dirty);

gdk_export int GDKfdlocate(str nme, str mode, str ext);
gdk_export FILE *GDKfilelocate(str nme, str mode, str ext);
gdk_export void GDKfilepath(str path, str nme, str mode, str ext);
gdk_export int GDKcreatedir(str nme);
gdk_export int GDKremovedir(str nme);

gdk_export char *GDKload(str nme, str ext, size_t size, size_t chunk, int mode);
gdk_export int GDKsave(str nme, str ext, void *buf, size_t size, int mode);
gdk_export int GDKunlink(str dir, str nme, str extension);
gdk_export int GDKmove(str dir1, str nme1, str ext1, str dir2, str nme2, str ext2);
@
@}

@- Printing
@multitable @columnfractions 0.08 0.7
@item int
@tab BATprintf (stream *f, BAT *b)
@item int
@tab BATmultiprintf (stream *f, int argc, BAT *b[], int printoid, int order, int printorderby)
@end multitable

The functions to convert BATs into ASCII and the reverse use
internally defined formats. They are primarily meant for ease of
debugging and to a lesser extent for output processing.
Printing a BAT is done essentially by looping through its components,
printing each association.
If an index is available, it will be used.
@
The @%BATmultiprintf@ command assumes a set of BATs with
corresponding oid-s in the head columns. It performs the multijoin
over them, and prints the multi-column result on the file.

@{
@h
gdk_export int BATprint(BAT *b);
gdk_export int BATprintf(stream *f, BAT *b);
gdk_export int BATmultiprintf(stream *f, int argc, BAT *argv[], int printoid, int order, int printorderby);
@
@}

@- BAT clustering
@multitable @columnfractions 0.08 0.7
@item BAT *
@tab BATsort (BAT *b)
@item BAT *
@tab BATsort_rev (BAT *b)
@item BAT *
@tab BATorder (BAT *b)
@item BAT *
@tab BATorder_rev (BAT *b)
@item BAT *
@tab BATrevert (BAT *b)
@item int
@tab BATordered (BAT *b)
@end multitable
@
When working in a main-memory situation, clustering of data on disk-pages
is not important. Whenever mmap()-ed data is used intensively, reducing
the number of page faults is a hot issue.
@
The above functions rearrange data in MonetDB heaps (used for storing
BUNs var-sized atoms, or accelerators). Applying these clusterings will
allow that MonetDB's main-memory oriented algorithms work efficiently  also
in a disk-oriented context.
@
The @%BATsort@ functions return a copy of the input BAT, sorted
in ascending order on the head column. @%BATordered@ starts a
check on the head values to see if they are ordered. The result
is returned and stored in the @%hsorted@ field of the BAT.
@%BATorder@ is similar to @%BATsort@, but sorts the BAT itself, rather
than returning a copy (BEWARE: this operation destroys the
delta information. TODO:fix). The @%BATrevert@ puts all the live
BUNs of a BAT in reverse order. It just reverses the sequence, so
this does not necessarily mean that they are sorted in reverse order!
@{
@h
#define GDK_SORTED_REV	128	/* reversely sorted */
#define GDK_SORTED	65	/* 65 = (32 bits radix_clustered)<<1 + 1 */

gdk_export BAT *BATsort(BAT *b);
gdk_export BAT *BATsort_rev(BAT *b);
gdk_export BAT *BATorder(BAT *b);
gdk_export BAT *BATorder_rev(BAT *b);
gdk_export BAT *BATrevert(BAT *b);
gdk_export int BATordered(BAT *b);
gdk_export int BATordered_rev(BAT *b);
gdk_export BAT *BATssort(BAT *b);
gdk_export BAT *BATssort_rev(BAT *b);
gdk_export BAT *BATsorder(BAT *b);
gdk_export BAT *BATsorder_rev(BAT *b);

gdk_export void GDKqsort(void *a, void *base, size_t n, int width, int tpe, int loc);
gdk_export void GDKqsort_rev(void *a, void *base, size_t n, int width, int tpe, int loc);
gdk_export int GDKssort(void *a, void *base, size_t n, int width, int tpe, int loc);
gdk_export int GDKssort_rev(void *a, void *base, size_t n, int width, int tpe, int loc);

#define BAThordered(b)	(((b)->htype == TYPE_void)?GDK_SORTED:(b)->hsorted)
#define BATtordered(b)	(((b)->ttype == TYPE_void)?GDK_SORTED:(b)->tsorted)
#define BAThdense(b)	(BAThvoid(b) && (b)->hseqbase != oid_nil)
#define BATtdense(b)	(BATtvoid(b) && (b)->tseqbase != oid_nil)
#define BAThvoid(b)	(((b)->hdense&(b)->hsorted&1) || (b)->htype==TYPE_void)
#define BATtvoid(b)	(((b)->tdense&(b)->tsorted&1) || (b)->ttype==TYPE_void)
#define BAThkey(b)	(b->hkey != FALSE || BAThdense(b))
#define BATtkey(b)	(b->tkey != FALSE || BATtdense(b))

#define BATsetcount(b,cnt)	((b)->batCount = (cnt),(b)->batDirtydesc = TRUE)
@
@}

@+ BAT Buffer Pool
@multitable @columnfractions 0.08 0.7
@item int
@tab BBPfix (bat bi)
@item int
@tab BBPunfix (bat bi)
@item int
@tab BBPincref (bat bi, int logical)
@item int
@tab BBPdecref (bat bi, int logical)
@item void
@tab BBPhot (bat bi)
@item void
@tab BBPcold (bat bi)
@item str
@tab BBPname (bat bi)
@item bat
@tab BBPindex  (str nme)
@item BAT*
@tab BATdescriptor (bat bi)
@item bat
@tab BBPcacheid (BAT *b)
@end multitable
@
The BAT Buffer Pool module contains the code to manage the storage
location of BATs. It uses two tables @%BBPlogical@ and @%BBphysical@
to relate the BAT name with its corresponding file system name.
This information is retained in an ASCII file within the database home
directory for ease of inspection. It is loaded upon restart
of the server and saved upon transaction commit (if necessary).
@
The remaining BBP tables contain status information to load, swap and
migrate the BATs. The core table is
@%BBPcache@ which contains a pointer to the BAT descriptor with its heaps.
A zero entry means that the file resides on disk. Otherwise it has been read
or mapped into memory.
@
BATs loaded into memory are retained in a BAT buffer pool.  They
retain their position within the cache during their life cycle, which
make indexing BATs a stable operation.  Their descriptor can be
obtained using @%BBPcacheid@.
@
The @%BBPindex@ routine checks if a BAT with a certain name is
registered in the buffer pools. If so, it returns its BAT id.
The @%BATdescriptor@ routine has a  BAT id parameter, and returns
a pointer to the corresponding BAT record (after incrementing the reference
count). The BAT will be loaded into memory, if necessary.
@
@{
The structure of the BBP file obeys the tuple format for GDK.

The status and BAT persistency information is encoded in the status field.
@h
typedef struct {
	BAT *b[2];		/* if loaded: BAT* handle + reverse */
	str nme[2];		/* logical name + reverse */
	str bak[2];		/* logical name + reverse backups */
	bat next[2];		/* next BBP slot in link list */
	BATstore *cache;	/* cached header info */
	str path;		/* dir + basename for storage */
	int refs;		/* in-memory references on which the loaded status of a BAT relies */
	int lrefs;		/* logical references on which the existence of a BAT relies */
	int lastused;		/* BBP LRU stamp */
	volatile int status;	/* status mask used for spin locking */
} BBPrec;

gdk_export bat BBPmaxsize;
gdk_export bat BBPlimit;
gdk_export bat BBP_free;
gdk_export int BBP_dirty;	/* BBP table dirty? */
gdk_export BBPrec *BBP;

/* fast defines without checks; internal use only  */
#define BBP_cache(i)    BBP[ABS(i)].b[(i)<0]
#define BBP_logical(i)  BBP[ABS(i)].nme[(i)<0]
#define BBP_next(i)     BBP[ABS(i)].next[(i)<0]
#define BBP_physical(i) BBP[ABS(i)].path
#define BBP_desc(i)	BBP[ABS(i)].cache
#define BBP_refs(i)     BBP[ABS(i)].refs
#define BBP_lrefs(i)    BBP[ABS(i)].lrefs
#define BBP_lastused(i) BBP[ABS(i)].lastused
#define BBP_status(i)   BBP[ABS(i)].status

/* macros that nicely check parameters */
#define BBPdirty(x)	 (BBP_dirty=(x))
#define BBPcacheid(b)	 ((b)->batCacheid)
#define BBPstatus(i)	 (BBPcheck((i),"BBPstatus")?BBP_status(i):-1)
#define BBPcurstamp()	 BBP_curstamp
#define BBPrefs(i)	 (BBPcheck((i),"BBPrefs")?BBP_refs(i):-1)
#define BBPcache(i)	 (BBPcheck((i),"BBPcache")?BBP_cache(i):(BAT*) NULL)
#define BBPname(i)	 (BBPcheck((i),"BBPname")?((i) > 0 || BBP_logical(i))?BBP_logical(i):BBP_logical(-(i)):"")
#define BBPvalid(i)      (BBP_logical(i) != NULL && *BBP_logical(i) != '.')
#define BATgetId(b)	 BBPname((b)->batCacheid)
#define BBPfix(i)	 BBPincref(i, FALSE)
#define BBPunfix(i)	 BBPdecref(i, FALSE)

#define BBPRENAME_ALREADY	-1
#define BBPRENAME_ILLEGAL	-2
#define BBPRENAME_LONG		-3

gdk_export void BBPlock(str s);

gdk_export void BBPhot(bat b);
gdk_export void BBPcold(bat b);
gdk_export void BBPtrim(size_t memdelta, size_t vmdelta);
gdk_export void BBPunlock(str s);

gdk_export str BBPlogical(bat b, str buf);
gdk_export str BBPphysical(bat b, str buf);
gdk_export int BBP_curstamp;
gdk_export BATstore *BBPgetdesc(bat i);
gdk_export BAT *BBPquickdesc(bat b, int delaccess);
@
@}

@+ GDK Extensibility
GDK can be extended with new atoms, search accelerators and storage
modes.

@- Atomic Type Descriptors
The atomic types over which the binary associations are maintained
are described by an atom descriptor.
 @multitable @columnfractions 0.08 0.7
@item void
@tab ATOMproperty    (str   nme, char *property, int (*fcn)()));
@item int
@tab ATOMindex       (char *nme);
@item int
@tab ATOMdump        ();
@item void
@tab ATOMdelete      (int id);
@item str
@tab ATOMname        (int id);
@item int
@tab ATOMsize        (int id);
@item int
@tab ATOMalign       (int id);
@item int
@tab ATOMvarsized    (int id);
@item ptr
@tab ATOMnilptr      (int id);
@item int
@tab ATOMfromstr     (int id, str s, int* len, ptr* v_dst);
@item int
@tab ATOMtostr       (int id, str s, int* len, ptr* v_dst);
@item int
@tab ATOMhash        (int id, ptr val, in mask);
@item int
@tab ATOMcmp         (int id, ptr val_1, ptr val_2);
@item int
@tab ATOMconvert     (int id, ptr v, int direction);
@item int
@tab ATOMfix         (int id, ptr v);
@item int
@tab ATOMunfix       (int id, ptr v);
@item int
@tab ATOMheap        (int id, Heap *hp, size_t cap);
@item void
@tab ATOMheapconvert (int id, Heap *hp, int direction);
@item int
@tab ATOMheapcheck   (int id, Heap *hp, HeapRepair *hr);
@item int
@tab ATOMput         (int id, Heap *hp, BUN pos_dst, ptr val_src);
@item int
@tab ATOMdel         (int id, Heap *hp, BUN v_src);
@item int
@tab ATOMlen         (int id, ptr val);
@item ptr
@tab ATOMnil         (int id);
@item int
@tab ATOMformat      (int id, ptr val, char** buf);
@item int
@tab ATOMprint       (int id, ptr val, stream *fd);
@item ptr
@tab ATOMdup         (int id, ptr val );
@end multitable
@
@- Atom Definition
User defined atomic types can be added to a running system with
the following interface:.

@itemize
@item @emph{ATOMproperty()} registers a new atom definition, if there is no 
atom registered yet under that name. 
It then installs the attribute of the named property.
Valid names are "size", "align", "null"