changelog

网络爬虫程序

  for storing port numbers (thanks to Raun Nohavitza)
* fixed checking of HTML document types with added encoding after MIME type
  (thanks to Brunie-Taton Alain)
* repaired broken site level computing on sites with moved starting documents
  in -site_level option
* implemented functions for launching commands on WIN32 with system()-like
  function when cygwin not installed (thanks to Thierry R間nier)
* added support for loading files from MSIE cache on Win32, and added options
  -ie_cache/-noie_cache to enable/disable this feature
* backported improvements to gaccel code from chbg. Now it is much more
  reliable.
* added new macro %q to -fnrules option, which will be replaced with urlencoded
  query string from POS/GET request specification
* fixed big memory leak in old style fnrules evaluation function caused by bad
  block nesting
* added two new functions (sif, !, &, |) to -fnrules option. ! is logical NOT
  for numeric values. & is logical AND for num. values, | is logical OR for
  numeric values. sif is decision between two strings by condition.
  (sif (cond) (val_if_cond_true) (val_if_cond_false)) is eqivalent for C
  expression (cond) ? (val_if_cond_true) : (val_if_cond_false)
* added checks to reject compilation of NS cache reading code with BerkeleyDB
  2.0 and above because of incompatible database format. NScache uses 1.8x hash.
* corected support for reading NS cache on big endian platforms based on patch
  for my NScache program from ...
* made HTTP/1.1 default (still possible to switch to HTTP/1.0 with option
  -nouse_http11)
* changed handling of parent urls in URL structure. Now is used linked list
  instead of nul terminated array. It is much safer for handling in MT.
* fixed segfault on redirection of robots.txt when HTTP/1.1 enabled cased by
  bad handling of persistant connections
* fixed bug in robots.txt file parsing code which causes infinite loops with
  some robots.txt files
* fixed memory leaks on robots.txt redirections
* fixed segfault when using -mode dontstore in multithreaded mode, caused by
  allocating shorter buffer for storing temporary unique name :-(
* fix to be able to compile with gtk-1.3 (aka gtk-2.0)
* added support for HTTP redirection on 307 response code
* added description messages for all HTTP/1.1 response codes which may occure
  and cause unknow errors just with numeric description
* fixed bug in processing of HTTP/1.1 chunked transfer encoding types
  after moved URLs because of oddly initialized trailer reading flags :-(
* it is possible now enter on commandline otions unsupported in current
  compile time configuration, pavuk now only displays warning instead of
  raising error and exiting (thanks to Bjorn R. Bjornsson)
* fixed compilation when threads are enabled support for regular exprssions
  is disabled or not present
* added locking of robots.txt info structure to prevent downloading it 
  concurrently with multiple threads when compiled with MT support
* ---------- released testing version 0.9pl27b
* fixed compilation bug when compiling without SSL support (thanks to 
  Le Faucheur Frederic)
* fixed bug made in previous testing release which causes segfault always
  when opening Limits config dialog because of use of unitialized pointer
* added support for long/short commandline options with GNU getopt like syntax
  and compatibility with old format of pavuk options (no short options defined
  yet)
* changed handling of scenarios from commandline. Scenario is now loaded at time
  when is --scenario option processed by commandline parser instead of prior to
  commandline parsing as before.
* now it is not mandatory to specify --scndir option before loading scenario.
* ---------- released testing version 0.9pl27c
* more reliable implementation of asynchronous DNS client/server for GUI
  version. Now guarantees atomicity of reads/writes, so no possible of
  protocol inconsistence after user break in middle of communication.
* internal restructuralization of code (hope not, but may lead to problems)
* fixed bug in preserving of presistant connections on robot.txt redirects
* fixed unnecessary closures of persistant connections in sync mode after
  304 response code
* added new options -dump_after/-nodump_after for use with -dumpfd option.
  this option control when will be document dumped to output (immediatly or
  after download&processing)
* added new options -dump_response/-nodump_response for dumping also HTTP
  responses to -dumpfd
* fixed bug in parsing CSS inside HTML tags
* removed support for extracting destination URL from HTML after HTTP 
  redirects. It must be broken server which doesn't send Location: header
  after redirect ... not worth to add workarounds for this problem
* rewrote from scratch the HTML parser (this means I'v got rid of the
  oldest, worsest writen code in pavuk). It seemds it should be bit faster
  and is much better extensible an maintainable.
* removed few small memory leaks
* added simple support for javascript patterns in DOM event attributes of tags,
  based on regular expressions
* ---------- released testing version 0.9pl27d
* fixed several memory leaks
* fixed bug in base64 encoding routine which was failing with non ASCII
  characters above 127
* changed way how is handled Digest authorization
* implemented NTLM authorization
* implemented NTLM proxy authorization
* now -auth_scheme & -http_proxy_auth options accept also textual parameters
  "user" "Basic" "Digest" "NTLM" besides numeric 1 2 3 4
* total restructuralization and cleanup of HTTP handling code. I was carefull,
  but it may lead to problems.
* now works NTLM and Digest authorization also with CONNECT requests
* minor changes in common settings dialog
* fixed bug in processing js patterns caused by bad tag attributes
* added new option -js_patterns to allow parsing of custom javascript patterns
  inside HTML documents
* added support for parsing also script body and look for patterns line by line
  (works also for files referenced by <SCRIPT SRC=...>
* implemented handling of proxy redirects (305 HTTP response)
* fixed compilation bug caused by undeclared _mt_dumpfd_lock_ mutex (thanks
  to Le Faucheur Frederic)
* fixed bug in handling locales in national environment (thanks to Milan
  Kerslager)
* added Czech translation to Gnome desktop entry for pavuk (thanks to Milan
  Kerslager)
* ---------- released testing version 0.9pl27e
* implemented detection of broken HTTP/1.0 proxies which don't handle properly
  downgrading to HTTP/1.0 when communicating with server which use newer HTTP
  protocol version (this causes bug when trying to use persistent connections)
* more paranoia checkings of reading/writing sockets in HTTP code
* automatic request repeat after premature closure of persistent HTTP 
  connection
* added support for robots excluding with <META NAME="robots" content="...">
  (thanks to Markus Mayer)
* fixed compilation bug with OpenSSL-0.9.6 because of new MD4 implementation
  int this OpenSSL version (thanks to Le Faucheur Frederic)
* fixed bug in new html parsing engine which fails to parse properly rest of 
  document after <script>...</script>
* added support for HTTP/1.0 Keep-Alive proxy connections
* ---------- released testing version 0.9pl27f
* added install script for NSIS win32 installer
* fixed compilation bugs when building without GUI
* portability fixes to QNX RtP
* updated auth info edit dialog for NTLM support
* fixed possible MT race condition in gopher directory persing routine
* fixed confusion of ftp code with -remove_old & -ftplist when in sync mode
  files disapeared from server were processed like directories which failed 
  (thanks to galanga)
* ported to BeOS 5 PE (works fine except file locking)
* added support for javascrip parsing in javascript:... URLs inside any
  supported HTML attribute
* fixed ftp directory listing when using active ftp data connections
* added option -follow_cmd which allows you to execute some script which
  can decide if pavuk should follow links from current document (thanks to
  Georg Rehm and hashao)
* adjusted establishment of active ftp data connections to be able to handle
  properly states, when server is unable or don't want to connect before 
  sending response
* leading/trailing spaces are removed from attributes before processing it 
  as URL to support broken sites ...
* ---------- released testing version 0.9pl27g
* fixed segfault when Location: contains relative URL after redirect
* fixed broken timestamping of HTML files in sync mode (thanks to Le Faucheur
  Frederic)
* fixed segfault on broken HTML tags with leading spaces and unclosed quotes
* if -store_info is active also rejected URLs contain stored MIME header 
  (thanks to Georg Rehm)
* don't apply limiting conditions (minsize/maxsize/mimet) on robots.txt
  documents
* fixed segfault when -norelocate option is activated (thanks to Markus Mayer)
* added O_BINARY to several open calls to prevent possible problems on Win32
* added new options -retrieve_symlink/-noretrieve_symlink to enable
  downloading of symbolic links from FTP server as regular files (thanks to 
  Petr Cech & Andras Korn)
* fixed segfault in robots info cleanup code
* implemented new -js_transform option to allow bit more powerfull support
  for js patterns. No rewriting supported now (thanks to Mark D. Anderson)
* fixed problems when compiling with PCRE support
* ---------- released testing version 0.9pl27h
* fixed segfault on broken meta refresh tag (thanks to Georg Rehm)
* fixed bug in removing of trailing spaces from URLs (thanks to Le Faucheur
  Frederic)
* added support for access authorization to FTP proxy server (thanks to Beno 
  Kardel)
* added GUI config for -js_transform option
* fixed bug in processing javascript bodies enclosed between <script></script>,
  which causes breakin of ending </script> tag
* -js_pattern patterns without substrings are now omited
* fixed broken behaviour of pavuk when while regeting file receives empty
  response, it will process it as proper HTTP/0.9 response and stops regeting
  file (thanks to Christian Axbrink)
* simplified that horrible dialogs for adding preffered languages,charsets and
  mime types
* added new debug level "limits" for debuging limiting conditions
* updated manual page
* fixed deadlock on closing log file
* ---------- released testing version 0.9pl27i
* updated Czech message catalog (thanks to Petr Cech)
* added initialization of GTK locales
* added posibility to generate massage catalogs in UTF-8 encoding for
  use with future versions of GTK+
* fixed problems with switching languge multiple times in GUI window
* updated documentation
* updated German message catalog (thanks to Colin Marquardt)
* fixed retrieving of URLs from selection and via DND to omit illegal CRLF
  characters (thanks to Aleksander Adamowski)
* adjusted win32 installer script to support installing message catalogs
* added support for setting message catalog path on WIN32 to install directory
* better handling of WIN32 paths in GUI
* added window icon to WIN32 version

version 0.9pl28 (Aug ?? 2001)
---------------
* added new option (-limit_inlines/-dont_limit_inlines) to disable checking
  of limiting options for inline objects (thanks to Olivier Sirol)
* fixed bug with special characters in filenames on FTP servers (thanks to
  Jo雔 GRONDIN), same for Gopher directories
* FTP directory listings are now transfered in ASCII mode (thanks to Jo雔
  GRONDIN)
* removed MT race condition in calling inet_ntoa()
* added new option -ftp_list_options to allow passing options to FTP LIST/NLST
  commands
* support for multiple WWW-Authenticate: and Proxy-Authenticate: in HTTP
  response (thanks to Monika Nowotnik)
* ported to AtheOS
* fixed improperly handled rewriting of links in HTML documents pointing to
  itself (thanks to Nicolay Mausz)
* added new function (getval) to -fnrules option extened syntax rule for
  getting values of query parameters of URL (thanks to Nicolay Mausz)
* added initialization of OpenSSL PRNG randomizer to prevent message
  "PRNG not seeded" on some platforms (thanks to Albert Chin)
* ---------- released testing version 0.9pl28a
* compilation fixes for nongcc compilers and bigendian architectures (thanks
  to Albert Chin)
* fixed segfault which occured always when used unknown long option
* added forgoten gdk options to option table
* fixed compilation without NTLM support enabled (thanks to Georg Rehm)
* added option --disable-ntlm to configure script to be able to compile
  pavuk without NTLM authorization support (thanks to Albert Chin)
* fixed segfault which occurs when closing Common config dialog (thanks to
  Georg Rehm)
* fixed all notworking options using regular patters when pavuk is compiled
  as multithreaded program (thanks to Mirko)
* fixed NTLM implementation to be able to work properly on bigendian machines,
  with non GCC compilers and on 64bit platforms
* fixed leaking of file descriptors after "File redirect" when have before
  persistent connection opened
* improved URL queue handling and downloading threads management
* changed internaly handling of filename assignemnts (not well tested yet,
  can cause instability or deadlocks in MT)
* fixed segfault when no URL is specified in -request or -formdata options
  (thanks to Andrew Price)
* fixed segfault when using -formdata option caused by freeing already freed
  memory chunk (thanks to Andrew Price)
* removed several minor memory leaks
* added checking of BerkeleyDB implementation in libc in configure script
* updated French message catalog (thanks to Le Faucheur Frederic and
  Pascal Adoux)
* added new option -fix_wuftpd, to fix broken wuftpd behaviour, when it
  doesn't raise error when listing not exixting directory (thanks to
  Jo雔 GRONDIN)
* ---------- released testing version 0.9pl28b
* added new option -post_update/-nopost_update to force pavuks URL updating
  engine to update in parents documents only URL currently downloaded
* %o macro is supporte now also in simple -fnrules macros
* added two new macros to -fnrules option - %M == mime type of document,
  %E == standard extension of document MIME type. This two new macros work
  properly only when used with -post_update options. (thanks to Majkel
  Kretschmar)
* in sync mode are now processed at first links from direcory scan (if -subdir
  was specified) and than just other links.
* added two new f