libcurl-tutorial.3

THIS IS HTTP CURL Example

.\" **************************************************************************
.\" *                                  _   _ ____  _
.\" *  Project                     ___| | | |  _ \| |
.\" *                             / __| | | | |_) | |
.\" *                            | (__| |_| |  _ <| |___
.\" *                             \___|\___/|_| \_\_____|
.\" *
.\" * Copyright (C) 1998 - 2007, Daniel Stenberg, <[e-mail]>, et al.
.\" *
.\" * This software is licensed as described in the file COPYING, which
.\" * you should have received as part of this distribution. The terms
.\" * are also available at http://curl.haxx.se/docs/copyright.html.
.\" *
.\" * You may opt to use, copy, modify, merge, publish, distribute and/or sell
.\" * copies of the Software, and permit persons to whom the Software is
.\" * furnished to do so, under the terms of the COPYING file.
.\" *
.\" * This software is distributed on an "AS IS" basis, WITHOUT WARRANTY OF ANY
.\" * KIND, either express or implied.
.\" *
.\" * $Id: libcurl-tutorial.3,v 1.14 2007-08-30 20:34:57 danf Exp $
.\" **************************************************************************
.\"
.TH libcurl-tutorial 3 "27 Feb 2007" "libcurl" "libcurl programming"
.SH NAME
libcurl-tutorial \- libcurl programming tutorial
.SH "Objective"
This document attempts to describe the general principles and some basic
approaches to consider when programming with libcurl. The text will focus
mainly on the C interface but might apply fairly well on other interfaces as
well as they usually follow the C one pretty closely.

This document will refer to 'the user' as the person writing the source code
that uses libcurl. That would probably be you or someone in your position.
What will be generally referred to as 'the program' will be the collected
source code that you write that is using libcurl for transfers. The program
is outside libcurl and libcurl is outside of the program.

To get the more details on all options and functions described herein, please
refer to their respective man pages.

.SH "Building"
There are many different ways to build C programs. This chapter will assume a
unix-style build process. If you use a different build system, you can still
read this to get general information that may apply to your environment as
well.
.IP "Compiling the Program"
Your compiler needs to know where the libcurl headers are located. Therefore
you must set your compiler's include path to point to the directory where you
installed them. The 'curl-config'[3] tool can be used to get this information:

$ curl-config --cflags

.IP "Linking the Program with libcurl"
When having compiled the program, you need to link your object files to create
a single executable. For that to succeed, you need to link with libcurl and
possibly also with other libraries that libcurl itself depends on. Like the
OpenSSL libraries, but even some standard OS libraries may be needed on the
command line. To figure out which flags to use, once again the 'curl-config'
tool comes to the rescue:

$ curl-config --libs

.IP "SSL or Not"
libcurl can be built and customized in many ways. One of the things that
varies from different libraries and builds is the support for SSL-based
transfers, like HTTPS and FTPS. If a supported SSL library was detected
properly at build-time, libcurl will be built with SSL support. To figure out
if an installed libcurl has been built with SSL support enabled, use
\&'curl-config' like this:

$ curl-config --feature

And if SSL is supported, the keyword 'SSL' will be written to stdout,
possibly together with a few other features that can be on and off on
different libcurls.

See also the "Features libcurl Provides" further down.
.IP "autoconf macro"
When you write your configure script to detect libcurl and setup variables
accordingly, we offer a prewritten macro that probably does everything you
need in this area. See docs/libcurl/libcurl.m4 file - it includes docs on how
to use it.

.SH "Portable Code in a Portable World"
The people behind libcurl have put a considerable effort to make libcurl work
on a large amount of different operating systems and environments.

You program libcurl the same way on all platforms that libcurl runs on. There
are only very few minor considerations that differs. If you just make sure to
write your code portable enough, you may very well create yourself a very
portable program. libcurl shouldn't stop you from that.

.SH "Global Preparation"
The program must initialize some of the libcurl functionality globally. That
means it should be done exactly once, no matter how many times you intend to
use the library. Once for your program's entire life time. This is done using

 curl_global_init()

and it takes one parameter which is a bit pattern that tells libcurl what to
initialize. Using \fICURL_GLOBAL_ALL\fP will make it initialize all known
internal sub modules, and might be a good default option. The current two bits
that are specified are:
.RS
.IP "CURL_GLOBAL_WIN32"
which only does anything on Windows machines. When used on
a Windows machine, it'll make libcurl initialize the win32 socket
stuff. Without having that initialized properly, your program cannot use
sockets properly. You should only do this once for each application, so if
your program already does this or of another library in use does it, you
should not tell libcurl to do this as well.
.IP CURL_GLOBAL_SSL
which only does anything on libcurls compiled and built SSL-enabled. On these
systems, this will make libcurl initialize the SSL library properly for this
application. This is only needed to do once for each application so if your
program or another library already does this, this bit should not be needed.
.RE

libcurl has a default protection mechanism that detects if
\fIcurl_global_init(3)\fP hasn't been called by the time
\fIcurl_easy_perform(3)\fP is called and if that is the case, libcurl runs the
function itself with a guessed bit pattern. Please note that depending solely
on this is not considered nice nor very good.

When the program no longer uses libcurl, it should call
\fIcurl_global_cleanup(3)\fP, which is the opposite of the init call. It will
then do the reversed operations to cleanup the resources the
\fIcurl_global_init(3)\fP call initialized.

Repeated calls to \fIcurl_global_init(3)\fP and \fIcurl_global_cleanup(3)\fP
should be avoided. They should only be called once each.

.SH "Features libcurl Provides"
It is considered best-practice to determine libcurl features at run-time
rather than at build-time (if possible of course). By calling
\fIcurl_version_info(3)\fP and checking out the details of the returned
struct, your program can figure out exactly what the currently running libcurl
supports.

.SH "Handle the Easy libcurl"
libcurl first introduced the so called easy interface. All operations in the
easy interface are prefixed with 'curl_easy'.

Recent libcurl versions also offer the multi interface. More about that
interface, what it is targeted for and how to use it is detailed in a separate
chapter further down. You still need to understand the easy interface first,
so please continue reading for better understanding.

To use the easy interface, you must first create yourself an easy handle. You
need one handle for each easy session you want to perform. Basically, you
should use one handle for every thread you plan to use for transferring. You
must never share the same handle in multiple threads.

Get an easy handle with

 easyhandle = curl_easy_init();

It returns an easy handle. Using that you proceed to the next step: setting
up your preferred actions. A handle is just a logic entity for the upcoming
transfer or series of transfers.

You set properties and options for this handle using
\fIcurl_easy_setopt(3)\fP. They control how the subsequent transfer or
transfers will be made. Options remain set in the handle until set again to
something different. Alas, multiple requests using the same handle will use
the same options.

Many of the options you set in libcurl are "strings", pointers to data
terminated with a zero byte. Keep in mind that when you set strings with
\fIcurl_easy_setopt(3)\fP, libcurl will not copy the data. It will merely
point to the data. You MUST make sure that the data remains available for
libcurl to use until finished or until you use the same option again to point
to something else.

One of the most basic properties to set in the handle is the URL. You set
your preferred URL to transfer with CURLOPT_URL in a manner similar to:

.nf
 curl_easy_setopt(handle, CURLOPT_URL, "http://domain.com/");
.fi

Let's assume for a while that you want to receive data as the URL identifies a
remote resource you want to get here. Since you write a sort of application
that needs this transfer, I assume that you would like to get the data passed
to you directly instead of simply getting it passed to stdout. So, you write
your own function that matches this prototype:

 size_t write_data(void *buffer, size_t size, size_t nmemb, void *userp);

You tell libcurl to pass all data to this function by issuing a function
similar to this:

 curl_easy_setopt(easyhandle, CURLOPT_WRITEFUNCTION, write_data);

You can control what data your function get in the forth argument by setting
another property:

 curl_easy_setopt(easyhandle, CURLOPT_WRITEDATA, &internal_struct);

Using that property, you can easily pass local data between your application
and the function that gets invoked by libcurl. libcurl itself won't touch the
data you pass with \fICURLOPT_WRITEDATA\fP.

libcurl offers its own default internal callback that'll take care of the data
if you don't set the callback with \fICURLOPT_WRITEFUNCTION\fP. It will then
simply output the received data to stdout. You can have the default callback
write the data to a different file handle by passing a 'FILE *' to a file
opened for writing with the \fICURLOPT_WRITEDATA\fP option.

Now, we need to take a step back and have a deep breath. Here's one of those
rare platform-dependent nitpicks. Did you spot it? On some platforms[2],
libcurl won't be able to operate on files opened by the program. Thus, if you
use the default callback and pass in an open file with
\fICURLOPT_WRITEDATA\fP, it will crash. You should therefore avoid this to
make your program run fine virtually everywhere.

(\fICURLOPT_WRITEDATA\fP was formerly known as \fICURLOPT_FILE\fP. Both names
still work and do the same thing).

If you're using libcurl as a win32 DLL, you MUST use the
\fICURLOPT_WRITEFUNCTION\fP if you set \fICURLOPT_WRITEDATA\fP - or you will
experience crashes.

There are of course many more options you can set, and we'll get back to a few
of them later. Let's instead continue to the actual transfer:

 success = curl_easy_perform(easyhandle);

\fIcurl_easy_perform(3)\fP will connect to the remote site, do the necessary
commands and receive the transfer. Whenever it receives data, it calls the
callback function we previously set. The function may get one byte at a time,
or it may get many kilobytes at once. libcurl delivers as much as possible as
often as possible. Your callback function should return the number of bytes it
\&"took care of". If that is not the exact same amount of bytes that was
passed to it, libcurl will abort the operation and return with an error code.

When the transfer is complete, the function returns a return code that informs
you if it succeeded in its mission or not. If a return code isn't enough for
you, you can use the CURLOPT_ERRORBUFFER to point libcurl to a buffer of yours
where it'll store a human readable error message as well.

If you then want to transfer another file, the handle is ready to be used
again. Mind you, it is even preferred that you re-use an existing handle if
you intend to make another transfer. libcurl will then attempt to re-use the
previous connection.

.SH "Multi-threading Issues"
The first basic rule is that you must \fBnever\fP share a libcurl handle (be
it easy or multi or whatever) between multiple threads. Only use one handle in
one thread at a time.

libcurl is completely thread safe, except for two issues: signals and SSL/TLS
handlers. Signals are used timeouting name resolves (during DNS lookup) - when
built without c-ares support and not on Windows..

If you are accessing HTTPS or FTPS URLs in a multi-threaded manner, you are
then of course using the underlying SSL library multi-threaded and those libs
might have their own requirements on this issue. Basically, you need to
provide one or two functions to allow it to function properly. For all
details, see this:

OpenSSL

 http://www.openssl.org/docs/crypto/threads.html#DESCRIPTION

GnuTLS

 http://www.gnu.org/software/gnutls/manual/html_node/Multi_002dthreaded-applications.html

NSS
 
 is claimed to be thread-safe already without anything required

yassl

 Required actions unknown

When using multiple threads you should set the CURLOPT_NOSIGNAL option to TRUE
for all handles. Everything will or might work fine except that timeouts are
not honored during the DNS lookup - which you can work around by building
libcurl with c-ares support. c-ares is a library that provides asynchronous
name resolves. Unfortunately, c-ares does not yet fully support IPv6. On some
platforms, libcurl simply will not function properly multi-threaded unless
this option is set.

Also, note that CURLOPT_DNS_USE_GLOBAL_CACHE is not thread-safe.

.SH "When It Doesn't Work"
There will always be times when the transfer fails for some reason. You might
have set the wrong libcurl option or misunderstood what the libcurl option
actually does, or the remote server might return non-standard replies that
confuse the library which then confuses your program.

There's one golden rule when these things occur: set the CURLOPT_VERBOSE
option to TRUE. It'll cause the library to spew out the entire protocol
details it sends, some internal info and some received protocol data as well
(especially when using FTP). If you're using HTTP, adding the headers in the
received output to study is also a clever way to get a better understanding