io.txt

著名的手机浏览器开源代码

This is the base of a paper I wrote with Horst.
It provides a good introduction to Dillo's internals.
(Highly recommended if you plan to patch or develop in Dillo)
--Jcid


-----------------------------------------------------
Paralell network programming of the Dillo web browser
-----------------------------------------------------

 Jorge Arellano-Cid <[e-mail]>
 Horst H. von Brand <[e-mail]>


--------
Abstract
--------

   Network  programs  face  several delay sources when sending or
retrieving  data.  This  is  particularly problematic in programs
which interact directly with the user, most notably web browsers.
We  present  a hybrid approach using threads communicated through
pipes  and  signal  driven  I/O, which allows a non-blocking main
thread and overlapping waiting times.


------------
Introduction
------------

   The Dillo project didn't start from scratch but mainly working
on  the  code base of gzilla (a light web browser written by Raph
Levien). As the project went by, the code of the whole source was
standardized,  and the networking engine was replaced with a new,
faster design. The source code is currently in alpha test, and is
available at <http://dillo.sourceforge.net> under the GNU General
Public License.

   This  paper covers basic design aspects of the hybrid approach
that  the  Dillo  web  browser  uses  to  solve  several  latency
problems.  After  introducing  the  main  delay-sources, the main
points of the hybrid design will be addressed.


-------------
Delay sources
-------------

   Network  programs  face several delay-sources while sending or
retrieving  data.  In  the particular case of a web browser, they
are found in:

  DNS querying:
    The time required to solve a name.

  Initiating the TCP connection:
    The three way handshake of the TCP protocol.

  Sending the query:
    The time spent uploading queries to the remote server.

  Retrieving data:
    The time spent expecting and receiving the query answer.

  Closing the TCP connection:
    The four packet-sending closing sequence of the TCP protocol.
 
   In  a  WAN  context,  every  single  item  of this list has an
associated  delay that is non deterministic and often measured in
seconds. If we add several connections per browsed page (each one
requiring  at  least  the 4 last steps), the total latency can be
considerable.


-----------------------------------
The traditional (blocking) approach
-----------------------------------

   The main problems with the blocking approach are:

     When issuing an operation that can't be completed
     immediately, the process is put to sleep waiting for
     completion, and the program doesn't do any other
     processing in the meantime.

     When waiting for a specific socket operation to complete,
     packets that belong to other connections may be arriving,
     and have to wait for service.

     Web browsers handle many small transactions, 
     if waiting times are not overlapped
     the latency perceived by the user can be very annoying.

     If the user interface is just put to sleep during network
     operations, the program becomes unresponsive, confusing
     and perhaps alarming the user.

     Not overlapping waiting times and processing makes
     graphical rendering (which is arguably the central function
     of a browser) unnecessarily slow.


---------------------
Dillo's hybrid design
---------------------

   Dillo  uses  threads  and  signal  driven  I/O  extensively to
overlap   waiting   times  and  computation.  Handling  the  user
interface  in a thread that never blocks gives a good interactive
``feel.''  The  use of GTK+, a sophisticated widget framework for
graphical  user  interfaces,  helped very much to accomplish this
goal.  All the interface, rendering and I/O engine was built upon
its facilities.

   The  design  is  said to be ``hybrid'' because it uses threads
for  DNS  querying and reading local files, and signal driven I/O
for  TCP  connections.  The  threaded  DNS  scheme is potentially
concurrent  (this  depends on underlying hardware), while the I/O
handling   (both   local   files   and   remote  connections)  is
definitively parallel.

   To  simplify  the  structure  of  the browser, local files are
encapsulated  into  HTTP streams and presented to the rest of the
browser  as  such, in exactly the same way a remote connection is
handled.  To  create  this  illusion,  a thread is launched. This
thread  opens  a  pipe  to  the  browser,  it then synthesizes an
appropriate  HTTP  header, sends it together with the file to the
browser  proper.  In  this way, all the browser sees is a handle,
the  data on it can come from a remote connection or from a local
file.

   To  handle  a remote connection is more complex. In this case,
the  browser  asks the cache manager for the URL. The name in the
URL  has  to  be  resolved  through  the DNS engine, a socket TCP
connection  must be established, the HTTP request has to be sent,
and  finally  the  result  retrieved. Each of the steps mentioned
could  give  rise to errors, which have to be handled and somehow
communicated to the rest of the program. For performance reasons,
it  is  critical that responses are cached locally, so the remote
connection  doesn't  directly  hand over the data to the browser;
the  response is passed to the cache manager which then relays it
to  the rest of the browser. The DNS engine caches DNS responses,
and  either  answers  them from the cache or by querying the DNS.
Querying  is  done  in a separate thread, so that the rest of the
browser isn't blocked by long waits here.

   The  activities  mentioned do not happen strictly in the order
stated  above.  It  is  even possible that several URLs are being
handled  at  the  same  time,  in  order  to  overlap waiting and
downloading.   The   functions  called  directly  from  the  user
interface   have   to  return  quickly  to  maintain  interactive
response.  Sometimes they return connection handlers that haven't
been completely set up yet. As stated, I/O is signal-driven, when
one  of  the  descriptors  is ready for data transfer (reading or
writing), it wakes up the I/O engine.

   Data transfer between threads inside the browser is handled by
pipes,  shared  memory  is  little used. This almost obviates the
need for explicit synchronization, which is one of the main areas
of  complexity and bugs in concurrent programs. Dillo handles its
threads  in  a way that its developers can think of it as running
on a single thread of control. This is accomplished by making the
DNS  engine  call-backs  happen  within  the  main thread, and by
isolating file loading with pipes.

   Using threads in this way has three big advantages:

     The browser doesn't block when one of its child threads
     blocks. In particular, the user interface is responsive
     even while resolving a name or downloading a file.

     Developers don't need to deal with complex concurrent
     concerns. Concurrency is hard to handle,  and few developers
     are adept at this. This gives access a much larger pool of
     potential developers, something which can be critical
     in an open-source development project.

     By making the code mostly sequential, debugging the code
     with traditional tools like gdb is possible. Debugging
     parallel programs is very hard, and appropriate tools are
     hard to come by.

   Because  of  simplicity and portability concerns, DNS querying
is  done  in  a  separate  thread. The standard C library doesn't
provide  a  function for making DNS queries that don't block. The
alternative  is  to implement a new, custom DNS querying function
that doesn't block. This is certainly a complex task, integrating
this  mechanism  into the thread structure of the program is much
simpler.

   Using  a  thread  and  a  pipe  to  read  a  local file adds a
buffering step to the process (and a certain latency), but it has
a couple of significative advantages:

     By handling local files in the same way as remote
     connections, a significant amount of code is reused.

     A preprocessing step of the file data can be added easily,
     if needed. In fact, the file is encapsulated into an HTTP
     data stream.


-----------
DNS queries
-----------

   Dillo handles DNS queries with threads, letting a child thread
wait  until  the  DNS server answers the request. When the answer
arrives,  a call-back function is called, and the program resumes
what  it  was doing at DNS-request time. The interesting thing is
that  the  call-back  happens in the main thread, while the child
thread  simply  exits  when  done.  This is implemented through a
server-channel design. 


The server channel
------------------

   There  is  one  thread  for each channel, and each channel can
have  multiple  clients. When the program requests an IP address,
the server first looks for a cached match; if it hits, the client
call-back  is  invoked immediately, but if not, the client is put
into  a  queue,  a thread is spawned to query the DNS, and a GTK+
idle  client  is  set  to poll the channel 5~times per second for
completion,  and  when  it finally succeeds, every client of that
channel is serviced.

   This  scheme  allows all the further processing to continue on
the same thread it began: the main thread.