tablet.mx

一个内存数据库的源代码这是服务器端还有客户端

@' The contents of this file are subject to the MonetDB Public License
@' Version 1.1 (the "License"); you may not use this file except in
@' compliance with the License. You may obtain a copy of the License at
@' http://monetdb.cwi.nl/Legal/MonetDBLicense-1.1.html
@'
@' Software distributed under the License is distributed on an "AS IS"
@' basis, WITHOUT WARRANTY OF ANY KIND, either express or implied. See the
@' License for the specific language governing rights and limitations
@' under the License.
@'
@' The Original Code is the MonetDB Database System.
@'
@' The Initial Developer of the Original Code is CWI.
@' Portions created by CWI are Copyright (C) 1997-2007 CWI.
@' All Rights Reserved.

@f tablet
@a Niels Nes, Martin Kersten
@d 29/07/2003
@+ The table interface

A database cannot live without ASCII tabular print/dump/load operations. 
It is needed to produce reasonable listings, to exchange answers
with a client, and to keep a database version for backup.
This is precisely where the tablet module comes in handy.
[This module should replace all other table dump/load functions]

We start with a simple example to illustrate the plain ASCII
representation and the features provided. Consider the
relational table answer(name:str, age:int, sex:chr, address:str, dob:date)
obtained by calling the routine tablet.page(B1,...,Bn) where the Bi represent
BATS.
@verbatim
[ "John Doe",		25,	'M',	"Parklane 5",	"25-12-1978" ]
[ "Maril Streep",	23,	'F',	"Church 5",	"12-07-1980" ]
[ "Mr. Smith",		53,	'M',	"Church 1",	"03-01-1950" ]
@end verbatim
@-
The lines contain the representation of a list in Monet tuple format.
This format has been chosen to ease parsing by any front-end. The scalar values
are represented according to their type. For visual display, the columns
are aligned by placing enough tabs between columns based on sampling the
underlying bat to determine a maximal column width.
(Note,actual commas are superfluous)

The arguments to the command can be any sequence of BATs, but which are
assumed to be aligned. That is, they all should have the same number of
tuples and the j-th tuple tail of Bi is printed along-side the j-th tuple
tail of Bi+1. 

Printing both columns of a single bat is handled by tablet as a
print of two columns. This slight inconvenience is catch-ed by
the io.print(b) command, which resolves most back-ward compatibility issues.
@-
In many cases this output would suffice for communication with a front-end.
However, for visual inspection the user should be provided also some meta
information derived from the database schema. Likewise, when reading a
table this information is needed to prepare a first approximation of
the schema namings. This information is produced by the command
tablet.header(B1,...,Bn), which lists the column role name.
If no role name is give, a default is generated based on the
BAT name, e.g. B1_tail.

@verbatim
#------------------------------------------------------#
# name,           age, sex, address,       dob         #
#------------------------------------------------------#
[ "John Doe",      25, 'M', "Parklane 5", "25-12-1978" ]
[ "Maril Streep",  23, 'F', "Church 5",   "12-07-1980" ]
[ "Mr. Smith",     53, 'M', "Church 1",   "03-01-1950" ]
@end verbatim
@-

The command tablet.display(B1,...,Bn) is a contraction of tablet.header();
tablet.page().  

In many cases the tablet produced may be too long to consume completely
by the front end. In that case, the user needs page size control, much
like the more/less utilities under Linux. However, no guarantee
is given for arbitrarily going back and forth. 
[but works as long as we materialize results first ]
A portion of the tablet can be printed by identifying the rows of interest as
the first parameter(s) in the page command, e.g.


@verbatim
tablet.page(nil,10,B1,...,Bn);	#prints first 10 rows
tablet.page(10,20,B1,...,Bn);	#prints next 10 rows
tablet.page(100,nil,B1,...,Bn);	#starts printing at tuple 100 until end
@end verbatim

A paging system also provides the commands tablet.firstPage(), 
tablet.nextPage(), tablet.prevPage(), and tablet.lastPage() using 
a user controlled tablet size tablet.setPagesize(L). 

The tablet display operations use a client (thread) specific formatting
structure. This structure is initialized using either 
tablet.setFormat(B1,...,Bn) or tablet.setFormat(S1,...,Sn) (Bi is a BAT, Si a scalar).
Subsequently, some additional properties can be set/modified, 
column width and brackets. 
After printing/paging the BAT resources should be freed using
the command tablet.finish().

Any access outside the page-range leads to removal of the report structure.
Subsequent access will generate an error.
To illustrate, the following code fragment would be generated by
the SQL compiler

@verbatim
	tablet.setFormat(B1,B2);
	tablet.setDelimiters("|","\t","|\n");
	tablet.setName(0, "Name");
	tablet.setNull(0, "?");
	tablet.setWidth(0, 15);
	tablet.setBracket(0, " ", ",");
	tablet.setName(1, "Age");
	tablet.setNull(1, "-");
	tablet.setDecimal(1, 9,2);
	tablet.SQLtitle("Query: select * from tables");
	tablet.page();
	tablet.SQLfooter(count(B1),cpuTicks);
@end verbatim

@-
This table is printed with tab separator(s) between elements
and the bar (|) to mark begin and end of the string.
The column parameters give a new title, 
a null replacement value, and the preferred column width.
Each column value is optionally surrounded by brackets.
Note, scale and precision can be applied to integer values only.
A negative scale leads to a right adjusted value.

The title and footer operations are SQL specific routines to
decorate the output.

Another example involves printing a two column table in XML format.
[Alternative, tablet.XMLformat(B1,B2) is a shorthand for the following:]

@verbatim
	tablet.setFormat(B1,B2);
	tablet.setTableBracket("<rowset>","</rowset>");
	tablet.setRowBracket("<row>","</row>");
	tablet.setBracket(0, "<name>", "</name>");
	tablet.setBracket(1, "<age>", "</age>");
	tablet.page();
@end verbatim
@- Tablet properties
More detailed header information can be obtained with the command 
tablet.setProperties(S), where S
is a comma separated list of properties of interest,
followed by the tablet.header().
The properties to choose from are: bat, name, type, width, 
sorted, dense, key, base, min, max, card,....

@verbatim
#--------------------------------------#
# B1,   B2,     B3,     B4,     B5     # BAT
# str,  int,    chr,    str,    date   # type
# true, false,  false,  false,  false  # sorted
# true, true,   false,  false,  false  # key
# ,     23,     'F',    ,              # min
# ,     53,     'M',	,              # max
# 4,     4,     4,      4,      4      # count
# 4,i    3,     2,      2,      3      # card
# name,	age,    sex,   address, dob    # name
#--------------------------------------#
@end verbatim

@- Scalar tablets
In line with the 10-year experience of Monet, printing scalar values
follow the tuple layout structure. This means that the header() command 
is also applicable.
For example, the sequence "i:=0.2;v:=sin(i); tablet.display(i,v);"
produces the answer:
@verbatim
#----------------#
# i,	v	 #
#----------------#
[ 0.2,	0.198669 ]
#----------------#
@end verbatim
@-

All other formatted printing should be done with the printf() operations
contained in the module io.

@- Tablet dump/restore

Dump and restore operations are abstractions over sequence of tablet commands.
The command tablet.dump(stream,B1,...,Bn) is a contraction of the sequence
tablet.setStream(stream);
tablet.setProperties("name,type,dense,sorted,key,min,max");
tablet.header(B1,..,Bn); tablet.page(B1,..,Bn). 
The result can be read by tablet.load(stream,B1,..,Bn) command. 
If loading is successful, e.g. no parsing
errors occurred, the tuples are appended to the corresponding BATs.

@- Front-end extension
A general bulk loading of foreign tables, e.g. CSV-files and fixed position 
records, is not provided. Instead, we extend the list upon need. 
Currently, the routines tablet.SQLload(stream,delim1,delim2, B1,..,Bn) 
reads the files using the Oracle(?) storage. The counterpart for
dumping is tablet.SQLdump(stream,delim1,delim2);

@- The commands

The load operation is for bulk loading a table, each column will be loaded
into its own bat. The arguments are void-aligned bats describing the
input, ie the name of the column, the tuple separator and the type.
The nr argument can be -1 (The input (datafile) is read until the end)
or a maximum.

The dump operation is for dumping a set of bats, which are aligned.
Again with void-aligned arguments, with name (currently not used),
tuple separator (the last is the record separator) and bat to be dumped.
With the nr argument the dump can be limited (-1 for unlimited).

The output operation is for ordered output. A bat (possibly form the collection)
gives the order. For each element in the order bat the values in the bats are
searched, if all are found they are output in the datafile, with the given
separators.

The scripts from the tablet.mil file are all there too for backward
compatibility with the old Mload format files.

The load_format loads the format file, since the old format file was
in a table format it can be loaded with the load command.

The result from load_format can be used with load_data to load the data
into a set of new bats.

These bats can be made persistent with the make_persistent script or
merge with existing bats with the merge_data script.

The dump_format scripts dump a format file for a given set of
to be dumped bats. These bats can be dumped with dump_data.
@mal
# the major tablet shuffling routines
module tablet;

command load( names:bat[:oid,:str], seps:bat[:oid,:str], 
		types:bat[:oid,:str], datafile:str, nr:int ) :bat[:str,:bat] 
address CMDtablet_load
comment "Load a bat using specific format.";

command input( names:bat[:oid,:str], seps:bat[:oid,:str], 
		types:bat[:oid,:str], s:streams, nr:int ) :bat[:str,:bat] 
address CMDtablet_input
comment "Load a bat using specific format.";

command dump(names:bat[:oid,:str], seps:bat[:oid,:str], 
		bats:bat[:oid,:bat], datafile:str, nr:int) :void 
address CMDtablet_dump 
comment "Dump the bat in ASCII format";

command output(order:bat[:any_1,:any_2], seps:bat[:oid,:str], 
		bats:bat[:oid,:bat], s:streams) :void 
address CMDtablet_output 
comment "Send the bat to an output stream.";

pattern display(v:any...):int 
address TABdisplayRow
comment "Display a formatted row";
pattern display(v:bat[:any_1,:any]...):int 
address TABdisplayTable
comment "Display a formatted table";

pattern page(b:bat[:any_1,:any]...):int 
address TABpage
comment "Display all pages at once without header";
pattern header(b:any...):int 
address TABheader
comment "Display the minimal header for the table";
command setProperties(prop:str):int 
address TABsetProperties
comment "Define the set of properties";
pattern dump(s:streams,b:bat[:any,:any]...):int 
address TABdump
comment "Print all pages with header to a stream";

pattern setFormat(b:any...):void 
address TABsetFormat
comment "Initialize a new reporting structure.";
command finish():void 
address TABfinishReport
comment "Free the storage space of the report descriptor";
command setStream(s:streams):void 
address TABsetStream
comment "Redirect the output to a stream.";
command setPivot(b:bat[:void,:oid]) :void
address TABsetPivot
comment "The pivot bat identifies the tuples of interest. The only requirement 
	    is that all keys mentioned in the pivot tail exist in all BAT parameters 
	    of the print comment. The pivot also provides control over the order 
	    in which the tuples are produced.";
command setDelimiter(sep:str):void 
address TABsetDelimiter
comment "Set the column separator.";
command setTableBracket(lbrk:str,rbrk:str)
address TABsetTableBracket
comment "Format the brackets around a table";
command setRowBracket(lbrk:str,rbrk:str)
address TABsetRowBracket
comment "Format the brackets around a row";

# Set the column properties 
pattern setColumn(idx:int, v:any_1)
address TABsetColumn
comment "Bind i-th output column to a variable";
command setName(idx:int, nme:str)
address TABsetColumnName
comment "Set the display name for a given column";
command setBracket(idx:int,lbrk:str,rbrk:str)
address TABsetColumnBracket
comment "Format the brackets around a field";
command setNull(idx:int, fmt:str)
address TABsetColumnNull
comment "Set the display format for a null value for a given column";
command setWidth(idx:int, maxwidth:int)
address TABsetColumnWidth
comment "Set the maximal display witdh for a given column. All values exceeding 
	the length are simple shortened without any notice.";
command setPosition(idx:int,f:int,i:int)
address TABsetColumnPosition
comment "Set the character position to use for this field when loading according to 
	fixed (punch-card) layout.";
command setDecimal(idx:int,s:int,p:int)
address TABsetColumnDecimal
comment "Set the scale and precision for numeric values";

command firstPage():void 
address TABfirstPage
comment "Produce the first page of output";
command lastPage():void 
address TABlastPage
comment "Produce the last page of output";
command nextPage():void 
address TABnextPage
comment "Produce the next page of output";
command prevPage():void 
address TABprevPage
comment "Produce the prev page of output";
command getPageCnt():void 
address TABgetPageCnt
comment "Return the size in number of pages";
command getPage(i:int):void 
address TABgetPage
comment "Produce the i-th page of output";
@-
@{
@include prelude.mx
@+ Implementation
The implementation needs the stream abstraction, which also provides
primitives to compress/decompress files on the fly.
The file can plain ASCII, gzipped or bzipped, decided by the extention 
(none, gz or bz2). The default is plain ASCII, which is formatted to
pre presented on the screen directly.

@h
#ifndef _TABLET_IO2_H_
#define _TABLET_IO2_H_

#include <gdk.h>
#include "streams.h"
#include <mal_exception.h>
#include <mal_client.h>
#include <mal_interpreter.h>

#ifdef WIN32
#ifndef LIBTABLET
#define tablet_export extern __declspec(dllimport)
#else
#define tablet_export extern __declspec(dllexport)
#endif
#else
#define tablet_export extern
#endif

typedef ptr *(*frStr) (void *extra, int type, char *s, char *e, char quote);
typedef int (*toStr) (void *extra, char **buf, int *len, int type, ptr a);

typedef struct Column_t {
	char *batname;
	char *name;		/* column title */
	char *sep;
	int seplen;
	char *type;
	int adt;		/* type index */
	BAT *c;			/* set to NULL when scalar is meant */
	ptr p;
	int base;		/* of first tuple */
	unsigned int tabs;	/* field size in tab positions */
	str lbrk, rbrk;		/* column brackets */
	str nullstr;		/* null representation */
	int quoted;		/* escape charaters */
	unsigned int width;	/* actual column width */
	unsigned int maxwidth;	/* permissible width */
	int fieldstart;		/* Fixed character field load positions */
	int fieldwidth;
	int scale, precision;
	toStr tostr;
	frStr frstr;
	void *extra;
	void *data;
	int len;
} Column;

@-
All table printing is based on building a report structure first.
This table structure is private to a client, which made us to
keep it in an ADT.
@h

typedef struct Table_t {
	int signature;
	char *sep;		/* default separator */
	str ttopbrk, tbotbrk;	/* table brackets */
	str rlbrk, rrbrk;	/* row brackets */
	str properties;		/* of header to display */
	str title, footer;	/* alternatives */
	size_t offset;
	size_t nr;		/* allocated space for table loads */
	size_t pageLimit;
	size_t firstrow, lastrow;	/* last window to print */
	size_t nr_attrs;	/* attributes found sofar */
	size_t max_attrs;
	Column *format;		/* remove later */
	stream *fd;
	BAT *pivot;
	unsigned int rowwidth;		/* sum of columns used for mallocs */
	Column columns[1];
} Tablet;

tablet_export BAT *TABLETload(BAT *names, BAT *seps, BAT *types, char *datafile, size_t nr);
tablet_export BAT *TABLETinput(BAT *names, BAT *seps, BAT *types, bstream *s, stream *out, size_t nr);
tablet_export size_t TABLEToutput(BAT *order, BAT *seps, BAT *bats, stream *s);
tablet_export void TABLETdump(BAT *names, BAT *seps, BAT *bats, char *datafile, size_t nr);

/* The low level routines are primarilly used by the SQL front-end.*/
tablet_export int TABLETcreate_bats(Tablet * as);
tablet_export size_t TABLETassign_BATs(Tablet * as, BAT *bats);
tablet_export ssize_t TABLETload_file(Tablet * as, bstream *b, stream *out);
tablet_export BAT *TABLETcollect_bats(Tablet * as);
tablet_export void TABLETdestroy_format(Tablet * as);
tablet_export int TABLEToutput_file(Tablet * as, BAT *order, stream *s);

tablet_export ptr *TABLETstrFrStr(Column *c, char *s, char *e);
tablet_export ptr *TABLETadt_frStr(void *extra, int type, char *s, char *e, char quote);
tablet_export int TABLETadt_toStr(void *extra, char **buf, int *len, int type, ptr a);

#endif
@c
#include "mal_config.h"
#include "tablet.h"
#include "algebra.h"

#include <string.h>
#include <ctype.h>

#define CLEAR(X) if(X) {GDKfree(X);X = NULL;}
#define isScalar(C)  C->adt != TYPE_bat

tablet_export str TABsetFormat(MalBlkPtr mb, MalStkPtr stk, InstrPtr pci);
tablet_export str TABheader(MalBlkPtr mb, MalStkPtr stk, InstrPtr pci);
tablet_export str TABdisplayTable(MalBlkPtr mb, MalStkPtr stk, InstrPtr pci);
tablet_export str TABdisplayRow(MalBlkPtr mb, MalStkPtr stk, InstrPtr pci);
tablet_export str TABpage(MalBlkPtr mb, MalStkPtr stk, InstrPtr pci);
tablet_export str TABsetProperties(int *ret, str *prop);
tablet_export str TABdump(MalBlkPtr mb, MalStkPtr stk, InstrPtr pci);
tablet_export str TABfinishReport(int *ret);
tablet_export str TABsetStream(int *ret, stream **s);
tablet_export str TABsetPivot(int *ret, int *bid);
tablet_export str TABsetDelimiter(int *ret, str *sep);
tablet_export str TABsetColumn(MalBlkPtr mb, MalStkPtr stk, InstrPtr pci);
tablet_export str TABsetColumnName(int *ret, int *idx, str *s);
tablet_export str TABsetTableBracket(int *ret, str *lbrk, str *rbrk);
tablet_export str TABsetRowBracket(int *ret, str *lbrk, str *rbrk);
tablet_export str TABsetColumnBracket(int *ret, int *idx, str *lbrk, str *rbrk);
tablet_export str TABsetColumnNull(int *ret, int *idx, str *nullstr);
tablet_export str TABsetColumnWidth(int *ret, int *idx, int *width);
tablet_export str TABsetColumnPosition(int *ret, int *idx, int *first, int *width);

tablet_export str TABsetColumnDecimal(int *ret, int *idx, int *scale, int *prec);
tablet_export str TABfirstPage(int *ret);
tablet_export str TABlastPage(int *ret);
tablet_export str TABnextPage(int *ret);
tablet_export str TABprevPage(int *ret);
tablet_export str TABgetPage(int *ret, int *pnr);
tablet_export str TABgetPageCnt(int *ret);
tablet_export str CMDtablet_load(int *ret, int *nameid, int *sepid, int *typeid, str *filename, int *nr);
tablet_export str CMDtablet_dump(int *ret, int *nameid, int *sepid, int *bids, str
*filename, int *nr);
tablet_export str CMDtablet_input(int *ret, int *nameid, int *sepid, int *typeid, stream *s, int *nr);
tablet_export str CMDtablet_output(int *ret, int *nameid, int *sepid, int *bids, void **s);
tablet_export void TABshowHeader(Tablet * t);
tablet_export void TABshowRow(Tablet * t);
tablet_export void TABshowRange(Tablet * t, lng first, lng last);

static void makeTableSpace(int rnr, unsigned int acnt);
static str bindVariable(Tablet * t, unsigned int anr, str nme, int tpe, ptr val, int *k);
static void clearTable(Tablet * t);
static int isScalarVector(Tablet * t);
static int isBATVector(Tablet * t);

static void TABshowPage(Tablet * t);
static int setTabwidth(Column * c);

#define LINE(s, X)     { int n=(int)(X)-1; stream_write(s, "#", 1, 1); while(--n>0) \
	                            stream_write(s, "-", 1, 1); \
	                            stream_printf(s, "#\n"); }
#define TABS(s, X)     { int n=(int)(X); while(n-->0) stream_printf(s, "\t");  }
@+
The table formatting information is stored in a system wide table.
Access is granted to a single client thread only.
The table structure depends on the columns to be printed,
it will be dynamically extended to accommodate the space.
@c
static int tablet_debug = 0;
static Tablet *tableReports[MAL_MAXCLIENTS];


static void
TABformatPrepare(MalBlkPtr mb, MalStkPtr stk, InstrPtr pci)
{
	Client c = MCgetClient();
	int rnr = c - mal_clients;
	Tablet *t;
	int anr = 0, i, tpe, k=0;
	ptr val;

	makeTableSpace(rnr, pci->argc - pci->retc);
	t = tableReports[rnr];
	if (t->rlbrk == 0)
		t->rlbrk = GDKstrdup("[ ");
	if (t->rrbrk == 0)
		t->rrbrk = GDKstrdup("]");
	if (t->sep == 0)
		t->sep = GDKstrdup("\t");
	t->rowwidth = strlen(t->rlbrk) + strlen(t->rrbrk) - 2;

	for (i = pci->retc; i < pci->argc; anr++, i++) {
		/* The type should be taken from the stack ! */
		tpe = stk->stk[pci->argv[i]].vtype;
		val = (ptr) getArgReference(stk, pci, i);
		bindVariable(t, anr, getArgName(mb, pci, i), tpe, val, &k);
	}
	t->nr_attrs = anr;
	t->fd = c->fdout;
	t->pivot = 0;
}

str
TABsetFormat(MalBlkPtr mb, MalStkPtr stk, InstrPtr pci)
{
	TABformatPrepare(mb, stk, pci);
	return MAL_SUCCEED;
}

str
TABheader(MalBlkPtr mb, MalStkPtr stk, InstrPtr pci)
{
	Client c = MCgetClient();
	int rnr = c - mal_clients;

	TABformatPrepare(mb, stk, pci);
	TABshowHeader(tableReports[rnr]);
	return MAL_SUCCEED;
}

str
TABdisplayTable(MalBlkPtr mb, MalStkPtr stk, InstrPtr pci)
{
	Client c = MCgetClient();
	int rnr = c - mal_clients;
	Tablet *t;

	TABheader(mb, stk, pci);
	t = tableReports[rnr];
	if (!isBATVector(t))
		throw(MAL, "tablet.print", "Only aligned BATs expected");
	else {
		t->pageLimit = 20;
		t->firstrow = t->lastrow = 0;
		TABshowPage(t);
	}
	return MAL_SUCCEED;
}

str
TABdisplayRow(MalBlkPtr mb, MalStkPtr stk, InstrPtr pci)
{
	Client c = MCgetClient();
	int rnr = c - mal_clients;
	Tablet *t;