db_file.pm

关于Berkelay数据库的共享源码

    $hash{"abc"} = "def" ;
    my $a = $hash{"ABC"} ;
    # ...
    undef $db ;
    untie %hash ;

Hopefully the contents of each of the filters should be
self-explanatory. Both "fetch" filters remove the terminating NULL,
and both "store" filters add a terminating NULL.


=head2 Another Example -- Key is a C int.

Here is another real-life example. By default, whenever Perl writes to
a DBM database it always writes the key and value as strings. So when
you use this:

    $hash{12345} = "something" ;

the key 12345 will get stored in the DBM database as the 5 byte string
"12345". If you actually want the key to be stored in the DBM database
as a C int, you will have to use C<pack> when writing, and C<unpack>
when reading.

Here is a DBM Filter that does it:

    use warnings ;
    use strict ;
    use DB_File ;
    my %hash ;
    my $filename = "filt" ;
    unlink $filename ;


    my $db = tie %hash, 'DB_File', $filename, O_CREAT|O_RDWR, 0666, $DB_HASH 
      or die "Cannot open $filename: $!\n" ;

    $db->filter_fetch_key  ( sub { $_ = unpack("i", $_) } ) ;
    $db->filter_store_key  ( sub { $_ = pack ("i", $_) } ) ;
    $hash{123} = "def" ;
    # ...
    undef $db ;
    untie %hash ;

This time only two filters have been used -- we only need to manipulate
the contents of the key, so it wasn't necessary to install any value
filters.

=head1 HINTS AND TIPS 


=head2 Locking: The Trouble with fd

Until version 1.72 of this module, the recommended technique for locking
B<DB_File> databases was to flock the filehandle returned from the "fd"
function. Unfortunately this technique has been shown to be fundamentally
flawed (Kudos to David Harris for tracking this down). Use it at your own
peril!

The locking technique went like this. 

    $db = tie(%db, 'DB_File', 'foo.db', O_CREAT|O_RDWR, 0644)
        || die "dbcreat foo.db $!";
    $fd = $db->fd;
    open(DB_FH, "+<&=$fd") || die "dup $!";
    flock (DB_FH, LOCK_EX) || die "flock: $!";
    ...
    $db{"Tom"} = "Jerry" ;
    ...
    flock(DB_FH, LOCK_UN);
    undef $db;
    untie %db;
    close(DB_FH);

In simple terms, this is what happens:

=over 5

=item 1.

Use "tie" to open the database.

=item 2.

Lock the database with fd & flock.

=item 3.

Read & Write to the database.

=item 4.

Unlock and close the database.

=back

Here is the crux of the problem. A side-effect of opening the B<DB_File>
database in step 2 is that an initial block from the database will get
read from disk and cached in memory.

To see why this is a problem, consider what can happen when two processes,
say "A" and "B", both want to update the same B<DB_File> database
using the locking steps outlined above. Assume process "A" has already
opened the database and has a write lock, but it hasn't actually updated
the database yet (it has finished step 2, but not started step 3 yet). Now
process "B" tries to open the same database - step 1 will succeed,
but it will block on step 2 until process "A" releases the lock. The
important thing to notice here is that at this point in time both
processes will have cached identical initial blocks from the database.

Now process "A" updates the database and happens to change some of the
data held in the initial buffer. Process "A" terminates, flushing
all cached data to disk and releasing the database lock. At this point
the database on disk will correctly reflect the changes made by process
"A".

With the lock released, process "B" can now continue. It also updates the
database and unfortunately it too modifies the data that was in its
initial buffer. Once that data gets flushed to disk it will overwrite
some/all of the changes process "A" made to the database.

The result of this scenario is at best a database that doesn't contain
what you expect. At worst the database will corrupt.

The above won't happen every time competing process update the same
B<DB_File> database, but it does illustrate why the technique should
not be used.

=head2 Safe ways to lock a database

Starting with version 2.x, Berkeley DB  has internal support for locking.
The companion module to this one, B<BerkeleyDB>, provides an interface
to this locking functionality. If you are serious about locking
Berkeley DB databases, I strongly recommend using B<BerkeleyDB>.

If using B<BerkeleyDB> isn't an option, there are a number of modules
available on CPAN that can be used to implement locking. Each one
implements locking differently and has different goals in mind. It is
therefore worth knowing the difference, so that you can pick the right
one for your application. Here are the three locking wrappers:

=over 5

=item B<Tie::DB_Lock>

A B<DB_File> wrapper which creates copies of the database file for
read access, so that you have a kind of a multiversioning concurrent read
system. However, updates are still serial. Use for databases where reads
may be lengthy and consistency problems may occur.

=item B<Tie::DB_LockFile> 

A B<DB_File> wrapper that has the ability to lock and unlock the database
while it is being used. Avoids the tie-before-flock problem by simply
re-tie-ing the database when you get or drop a lock.  Because of the
flexibility in dropping and re-acquiring the lock in the middle of a
session, this can be massaged into a system that will work with long
updates and/or reads if the application follows the hints in the POD
documentation.

=item B<DB_File::Lock> 

An extremely lightweight B<DB_File> wrapper that simply flocks a lockfile
before tie-ing the database and drops the lock after the untie. Allows
one to use the same lockfile for multiple databases to avoid deadlock
problems, if desired. Use for databases where updates are reads are
quick and simple flock locking semantics are enough.

=back

=head2 Sharing Databases With C Applications

There is no technical reason why a Berkeley DB database cannot be
shared by both a Perl and a C application.

The vast majority of problems that are reported in this area boil down
to the fact that C strings are NULL terminated, whilst Perl strings are
not. See L<DBM FILTERS> for a generic way to work around this problem.

Here is a real example. Netscape 2.0 keeps a record of the locations you
visit along with the time you last visited them in a DB_HASH database.
This is usually stored in the file F<~/.netscape/history.db>. The key
field in the database is the location string and the value field is the
time the location was last visited stored as a 4 byte binary value.

If you haven't already guessed, the location string is stored with a
terminating NULL. This means you need to be careful when accessing the
database.

Here is a snippet of code that is loosely based on Tom Christiansen's
I<ggh> script (available from your nearest CPAN archive in
F<authors/id/TOMC/scripts/nshist.gz>).

    use warnings ;
    use strict ;
    use DB_File ;
    use Fcntl ;

    my ($dotdir, $HISTORY, %hist_db, $href, $binary_time, $date) ;
    $dotdir = $ENV{HOME} || $ENV{LOGNAME};

    $HISTORY = "$dotdir/.netscape/history.db";

    tie %hist_db, 'DB_File', $HISTORY
        or die "Cannot open $HISTORY: $!\n" ;;

    # Dump the complete database
    while ( ($href, $binary_time) = each %hist_db ) {

        # remove the terminating NULL
        $href =~ s/\x00$// ;

        # convert the binary time into a user friendly string
        $date = localtime unpack("V", $binary_time);
        print "$date $href\n" ;
    }

    # check for the existence of a specific key
    # remember to add the NULL
    if ( $binary_time = $hist_db{"http://mox.perl.com/\x00"} ) {
        $date = localtime unpack("V", $binary_time) ;
        print "Last visited mox.perl.com on $date\n" ;
    }
    else {
        print "Never visited mox.perl.com\n"
    }

    untie %hist_db ;

=head2 The untie() Gotcha

If you make use of the Berkeley DB API, it is I<very> strongly
recommended that you read L<perltie/The untie Gotcha>. 

Even if you don't currently make use of the API interface, it is still
worth reading it.

Here is an example which illustrates the problem from a B<DB_File>
perspective:

    use DB_File ;
    use Fcntl ;

    my %x ;
    my $X ;

    $X = tie %x, 'DB_File', 'tst.fil' , O_RDWR|O_TRUNC
        or die "Cannot tie first time: $!" ;

    $x{123} = 456 ;

    untie %x ;

    tie %x, 'DB_File', 'tst.fil' , O_RDWR|O_CREAT
        or die "Cannot tie second time: $!" ;

    untie %x ;

When run, the script will produce this error message:

    Cannot tie second time: Invalid argument at bad.file line 14.

Although the error message above refers to the second tie() statement
in the script, the source of the problem is really with the untie()
statement that precedes it.

Having read L<perltie> you will probably have already guessed that the
error is caused by the extra copy of the tied object stored in C<$X>.
If you haven't, then the problem boils down to the fact that the
B<DB_File> destructor, DESTROY, will not be called until I<all>
references to the tied object are destroyed. Both the tied variable,
C<%x>, and C<$X> above hold a reference to the object. The call to
untie() will destroy the first, but C<$X> still holds a valid
reference, so the destructor will not get called and the database file
F<tst.fil> will remain open. The fact that Berkeley DB then reports the
attempt to open a database that is already open via the catch-all
"Invalid argument" doesn't help.

If you run the script with the C<-w> flag the error message becomes:

    untie attempted while 1 inner references still exist at bad.file line 12.
    Cannot tie second time: Invalid argument at bad.file line 14.

which pinpoints the real problem. Finally the script can now be
modified to fix the original problem by destroying the API object
before the untie:

    ...
    $x{123} = 456 ;

    undef $X ;
    untie %x ;

    $X = tie %x, 'DB_File', 'tst.fil' , O_RDWR|O_CREAT
    ...


=head1 COMMON QUESTIONS

=head2 Why is there Perl source in my database?

If you look at the contents of a database file created by DB_File,
there can sometimes be part of a Perl script included in it.

This happens because Berkeley DB uses dynamic memory to allocate
buffers which will subsequently be written to the database file. Being
dynamic, the memory could have been used for anything before DB
malloced it. As Berkeley DB doesn't clear the memory once it has been
allocated, the unused portions will contain random junk. In the case
where a Perl script gets written to the database, the random junk will
correspond to an area of dynamic memory that happened to be used during
the compilation of the script.

Unless you don't like the possibility of there being part of your Perl
scripts embedded in a database file, this is nothing to worry about.

=head2 How do I store complex data structures with DB_File?

Although B<DB_File> cannot do this directly, there is a module which
can layer transparently over B<DB_File> to accomplish this feat.

Check out the MLDBM module, available on CPAN in the directory
F<modules/by-module/MLDBM>.

=head2 What does "Invalid Argument" mean?

You will get this error message when one of the parameters in the
C<tie> call is wrong. Unfortunately there are quite a few parameters to
get wrong, so it can be difficult to figure out which one it is.

Here are a couple of possibilities:

=over 5

=item 1.

Attempting to reopen a database without closing it. 

=item 2.

Using the O_WRONLY flag.

=back

=head2 What does "Bareword 'DB_File' not allowed" mean? 

You will encounter this particular error message when you have the
C<strict 'subs'> pragma (or the full strict pragma) in your script.
Consider this script:

    use warnings ;
    use strict ;
    use DB_File ;
    my %x ;
    tie %x, DB_File, "filename" ;

Running it produces the error in question:

    Bareword "DB_File" not allowed while "strict subs" in use 

To get around the error, place the word C<DB_File> in either single or
double quotes, like this:

    tie %x, "DB_File", "filename" ;

Although it might seem like a real pain, it is really worth the effort
of having a C<use strict> in all your scripts.

=head1 REFERENCES

Articles that are either about B<DB_File> or make use of it.

=over 5

=item 1.

I<Full-Text Searching in Perl>, Tim Kientzle ([e-mail]),
Dr. Dobb's Journal, Issue 295, January 1999, pp 34-41

=back

=head1 HISTORY

Moved to the Changes file.

=head1 BUGS

Some older versions of Berkeley DB had problems with fixed length
records using the RECNO file format. This problem has been fixed since
version 1.85 of Berkeley DB.

I am sure there are bugs in the code. If you do find any, or can
suggest any enhancements, I would welcome your comments.

=head1 AVAILABILITY

B<DB_File> comes with the standard Perl source distribution. Look in
the directory F<ext/DB_File>. Given the amount of time between releases
of Perl the version that ships with Perl is quite likely to be out of
date, so the most recent version can always be found on CPAN (see
L<perlmodlib/CPAN> for details), in the directory
F<modules/by-module/DB_File>.

This version of B<DB_File> will work with either version 1.x, 2.x or
3.x of Berkeley DB, but is limited to the functionality provided by
version 1.

The official web site for Berkeley DB is F<http://www.sleepycat.com>.
All versions of Berkeley DB are available there.

Alternatively, Berkeley DB version 1 is available at your nearest CPAN
archive in F<src/misc/db.1.85.tar.gz>.

If you are running IRIX, then get Berkeley DB version 1 from
F<http://reality.sgi.com/ariel>. It has the patches necessary to
compile properly on IRIX 5.3.

=head1 COPYRIGHT

Copyright (c) 1995-2005 Paul Marquess. All rights reserved. This program
is free software; you can redistribute it and/or modify it under the
same terms as Perl itself.

Although B<DB_File> is covered by the Perl license, the library it
makes use of, namely Berkeley DB, is not. Berkeley DB has its own
copyright and its own license. Please take the time to read it.

Here are are few words taken from the Berkeley DB FAQ (at
F<http://www.sleepycat.com>) regarding the license:

    Do I have to license DB to use it in Perl scripts? 

    No. The Berkeley DB license requires that software that uses
    Berkeley DB be freely redistributable. In the case of Perl, that
    software is Perl, and not your scripts. Any Perl scripts that you
    write are your property, including scripts that make use of
    Berkeley DB. Neither the Perl license nor the Berkeley DB license
    place any restriction on what you may do with them.

If you are in any doubt about the license situation, contact either the
Berkeley DB authors or the author of DB_File. See L<"AUTHOR"> for details.


=head1 SEE ALSO

L<perl>, L<dbopen(3)>, L<hash(3)>, L<recno(3)>, L<btree(3)>,
L<perldbmfilter>

=head1 AUTHOR

The DB_File interface was written by Paul Marquess
E<lt>[e-mail]<gt>.
Questions about the DB system itself may be addressed to
E<lt>[e-mail]<gt>.

=cut