📄 netflow2mysql.html
字号:
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.0 Transitional//EN">
<html>
<head>
<title>NetFlow2MySQL version 0.24 manual</title>
</head>
<body>
<h1>NetFlow2MySQL version 0.24 manual</h1>
<h1>0. INDEX</h1>
<A href="#1.">1. What is NetFlow2MySQL</A><br>
<A href="#2.">2. Compatibility</A><br>
<A href="#2.1.">2.1. NetFlow versions</A><br>
<A href="#2.2.">2.2. Plathomes</A><br>
<A href="#3.">3. Requirement</A><br>
<A href="#3.1.">3.1. Required things to compile</A><br>
<A href="#3.2.">3.2. Required things to run</A><br>
<A href="#3.3.">3.3. Required things to collect NetFlow packets.</A><br>
<A href="#4.">4. Installation</A><br>
<A href="#4.1.">4.1. Compile</A><br>
<A href="#4.2.">4.2. Preparing Database</A><br>
<A href="#4.2.1.">4.2.1. Creating a database</A><br>
<A href="#4.2.2.">4.2.2. Creating a user</A><br>
<A href="#4.2.3.">4.2.3. Creating tables.</A><br>
<A href="#4.3.">4.3. Router Configuration</A><br>
<A href="#4.3.1.">4.3.1. Cisco Examples</A><br>
<A href="#4.3.1.1.">4.3.1.1. Basic configuration.</A><br>
<A href="#4.3.1.2.">4.3.1.2. Watch only some sub interfaces</A><br>
<A href="#4.3.1.3.">4.3.1.3. Setting the sampling rate</A><br>
<A href="#4.3.1.3.1.">4.3.1.3.1. For Cisco 12000, watching one of 1000 packets.</A><br>
<A href="#4.3.1.3.2.">4.3.1.3.2. For other Cisco routers, watching one of 1000 packets.</A><br>
<A href="#4.3.2.">4.3.2. Juniper Examples</A><br>
<A href="#4.3.3.">4.3.3. pcNetFlow example</A><br>
<A href="#4.4.">4.4. Storing Exporter Addresses(Optional)</A><br>
<A href="#5.">5. Usage</A><br>
<A href="#6.">6. License</A><br>
<A href="#7.">7. Contact</A><br>
<A href="#8.">8. Changes</A><br>
<H1><A name="1.">1. What is NetFlow2MySQL</A></H1>
<P>NetFlow2MySQL is software to store contents of NetFlow packets into MySQL databases.
<br>
After using NetFlow2MySQL, you can analyze your traffic by using SQL queries.</P>
<h1><A name="2.">2. Compatibility</A></h1>
<h2><A name="2.1.">2.1. NetFlow versions</A></h2>
<P>NetFlow2MySQL can handle:</P>
<UL>
<LI>NetFlow version 5 packets.</LI>
<LI>NetFlow version 9 packets.</LI>
</UL>
<STRONG>NetFlow2MySQL cannot handle other versions of NetFlow!</STRONG>
<h2><A name="2.2.">2.2. Plathomes</A></h2>
<p>We confirmed that this software worked on Linux and FreeBSD plathomes.
You need libgnugetopt and uncommenting of the Makefile on FreeBSD plathomes.</p>
<H1><A name="3.">3. Requirement</A></H1>
<H2><A name="3.1.">3.1. Required things to compile</A></H2>
<ul>
<li>MySQL Shared libraries (libmysql)</li>
<li>MySQL Development header files and libraries (libmysql-devel)</li>
</ul>
<h2><A name="3.2.">3.2. Required things to run</A></h2>
<ul>
<li>MySQL server (can run on another hosts)</li>
</ul>
<h2><A name="3.3.">3.3. Required things to collect NetFlow packets.</A></h2>
<ul>
<li>NetFlow capable routers.<br>
We tested with Cisco, Juniper or our <a href="http://cluster19.aist-nara.ac.jp/public/#pcNetFlow">pcNetFlow</a>.</li>
</ul>
<h1><A name="4.">4. Installation</A></h1>
<h2><A name="4.1.">4.1. Compile</A></h2>
<pre>$ tar xvzf NetFlow2MySQL-version.tar.gz
$ cd NetFlow2MySQL
$ make</pre>
<p>If make says 'mysql.h: No such file or directory' or '/usr/bin/ld: cannot find -lmysqlclient', change the paths of mysql_lib and mysql_include in Makefile to where you installed MySQL client library.
mysql_lib is where 'libmysqlclient.a' exists. mysql_include is where 'mysql.h' exists.
</p>
<p>If you are not lucky, you may need to uncomment or modify paths in the Makefile, install some libraries or hack it.</p>
<h2><A name="4.2.">4.2. Preparing Database</A></h2>
<h3><A name="4.2.1.">4.2.1. Creating a database</A></h3>
<p>You need to create at least one database for NetFlow collection.
By default, NetFlow2MySQL use 'flow' as the name of database.
However, any database names are OK if you give the name as the argument.
<br>
To create database, you can use mysqladmin or mysql_setpermission program.</p>
<h3><A name="4.2.2.">4.2.2. Creating a user</A></h3>
<p>You also need to add a suitable user to MySQL, the user must have INSERT privilege of the database.
You can use mysql_setpermission program.</p>
<h3><A name="4.2.3.">4.2.3. Creating tables.</A></h3>
<p>You need two tables for NetFlow collections.
One is named 'header' and the other is named 'record'.
<br>You can add these tables by executing following command.</p>
<pre>$ mysql -u [user_name] -p [database_name] < tables.sql</pre>
<p>The user must have CREATE privilege.</p>
<p>If you want to store NetFlow exporter's addresses, See <a href="#4.4.">section 4.4.</a></p>
<p>You can also create protocol and service tables by using 'protocol.sql' and 'service.sql'.
These tables represent '/etc/protocols' and '/etc/services'</p>
<h2><A name="4.3.">4.3. Router Configuration</A></h2>
<h3><A name="4.3.1.">4.3.1. Cisco Examples</A></h3>
<p>Here are some example configurations of Cisco routers.</p>
<h4><A name="4.3.1.1.">4.3.1.1. Basic configuration.</A></h4>
<p>Watching all incoming packets of GigabitEthernet 1/0 to make NetFlow statistics,
and will export them to 10.10.10.10:5000 as NetFlow version 9 packets.
<br>You may need IOS 12.0(24)S or higher to use NetFlow version 9. If your IOS does not support NetFlow version 9, it is OK to use version 5.</p>
<pre>
Rotuer>enable
Router#config terminal
Router(config)#ip flow-export version 9
Router(config)#ip flow-export destination 10.10.10.10 5000
Router(config)#int GigabitEthernet 1/0
Router(config-if)#ip route-cache flow</pre>
<h4><A name="4.3.1.2.">4.3.1.2. Watch only some sub interfaces</A></h4>
<p>If you only want to watch some sub interfaces.
<br>You may need IOS 12.2(14)S or higher to use 'ip flow ingress' command. See
<a href="http://www.cisco.com/en/US/products/sw/iosswrel/ps5187/products_command_reference_chapter09186a008017cf29.html#wp1101595">http://www.cisco.com/en/US/products/sw/iosswrel/ps5187/products_command_reference_chapter09186a008017cf29.html#wp1101595</A>
</p>
<pre>
Rotuer>enable
Router#config terminal
Router(config)#ip flow-export version 9
Router(config)#ip flow-export destination 10.10.10.10 5000
Router(config)#int GigabitEthernet 1/0.1
Router(config-if)#ip flow ingress</pre>
<h4><A name="4.3.1.3.">4.3.1.3. Setting the sampling rate</A></h4>
<p>Setting large sampling rate will reduce the load of the router and will decrease the numbers of records on the database.
It is required in common cases.</p>
<h5><A name="4.3.1.3.1.">4.3.1.3.1. For Cisco 12000, watching one of 1000 packets.</A></h5>
You may need IOS 12.0(11)S or higher. See <a href="http://www.cisco.com/univercd/cc/td/doc/product/software/ios120/120newft/120limit/120s/120s11/12s_sanf.htm">http://www.cisco.com/univercd/cc/td/doc/product/software/ios120/120newft/120limit/120s/120s11/12s_sanf.htm</a>
<pre>Router(config)#ip flow-sampling-mode packet-interval 1000
Router(config)#int GigabitEthernet 1/0
Router(config-if)#ip route-cache flow sampled</pre>
<h5><A name="4.3.1.3.2.">4.3.1.3.2. For other Cisco routers, watching one of 1000 packets.</A></h5>
You may need IOS 12.3(2)T or higher. See <a href="http://www.cisco.com/en/US/products/sw/iosswrel/ps5207/products_feature_guide09186a00801a7618.html">http://www.cisco.com/en/US/products/sw/iosswrel/ps5207/products_feature_guide09186a00801a7618.html</a>
<pre>Router(config)#flow-sampler-map test
Router(config-sampler)# mode random one-out-of 1000
Router(config)#int GigabitEthernet 1/0
Router(config-if)#flow-sampler test</pre>
<h3><A name="4.3.2.">4.3.2. Juniper Examples</A></h3>
<p>Here is an example configuration of Juniper routers.</p>
<pre>
interfaces {
ge-0/0/0 {
unit 0 {
family inet {
filter {
input netflow; // apply NetFlow filter to this interface
}
address 1.2.3.4/24;
}
}
firewall {
filter netflow {
term netflow {
then {
sample;
accept;
}
}
}
}
forwarding-options {
sampling {
input {
family inet {
rate 1000; // set sampling rate 1000
}
}
output {
cflowd 10.10.10.10 { //export NetFlow version 5 packets to 10.10.10.10:5000
port 5000;
version 5;
no-local-dump;
autonomous-system-type origin;
}
}
}
}
</pre>
<h3><A name="4.3.3.">4.3.3. <a href="http://cluster19.aist-nara.ac.jp/public/#pcNetFlow">pcNetFlow</a> example</A></h3>
<h2><A name="4.4.">4.4. Storing Exporter Addresses(Optional)</A></H2>
<p>If you want to store IPv4 addresses of you NetFlow exporters, you need following steps. First, you need to change your header table structure. You can done it with <strong>exporter_address.sql</strong>. The file is shipped with NetFlow2MySQL version 0.21 or above. You can apply it like</p>
<pre>$ mysql -u [user_name] -p [database_name] < exporter_address.sql</pre>
<p>The user must have ALTER privilege.</p>
<p><strong>WARNING:</strong> This patch add a column and a index to your header table. The header table is locked while patching. If your header table is huge, It take long time and you may lost NetFlow packets received while patching. You can igonore this warning if you do this just after creating the tables.</p>
<p>After applying <strong>exporter_address.sql</strong>, run NetFlow2MySQL with '--store-exporter-address' option. You can get the information as exporter_address on your header table. Exporter IP addresses are stored as 32bit unsigned integer. You can use MySQL's inet_ntoa() function to show them as typical IP addresses.</p>
<h1><A name="5.">5. Usage</A></h1>
<pre>$ ./NetFlow2MySQL -i <listen interface> -p <listen port>
[--receive-buffer-size >receive buffer size of UDP socket>]
[--mysql-host <mysql host name>]
[--mysql-user <mysql-user>]
[--mysql-password <mysql password>]
[--mysql-database <mysql database>]
[--mysql-port <mysql port>]
[--store-exporter-address]
[-d]
[--packet-sampling-rate <sampling rate>]</pre>
<pre>$ ./NetFlow2MySQL -v</pre>
<p>Required arguments are -i and -p.</p>
<h3>-i <i><listen interface></i></h3>
<p>Interface to listen to NetFlow packets.</p>
<h3>-p <i><listen port></i></h3>
<p>UDP port number to listen to NetFlow packets.</p>
<h3>--receive-buffer-size <i><receive buffer size of UDP socket></i></h3>
<p>To set UDP receive buffer size manually. See FAQ.html for more detail.</p>
<h3>--mysql-host <i><mysql host name></i></h3>
<p>Host name where the target MySQL server is working on.</p>
<h3>--mysql-user <i><mysql-user></i></h3>
<p>User name to connect to MySQL server.
If you do not specify the argument, the program will ask you.</p>
<h3>--mysql-password <i><mysql password></i></h3>
<p>Password to connect to MySQL server.
<br><b>**For security reason, we do not recommend you to use this argument.**</b>
<br>If you do not specify the argument, the program will ask you.</p>
<h3>--mysql-database <i><mysql database></i></h3>
<p>Database name to connect to.
Specify the database name you created.
If you do not specify the argument, <b>'flow'</b> is assumed.</p>
<h3>--store-exporter-address</h3>
<p>Store NetFlow exporter's IP addresses.
You have to use this option with a patched header table. See <a href="#4.4.">section 4.4.</a></p>
<h3>-d</h3>
<p>Output debug information.</p>
<h3>-v</h3>
<p>Output NetFlow2MySQL's version and exits.</p>
<h3>--packet-sampling-rate <sampling rate></h3>
<p><b>*VERY EXPERIMENTAL*</b></p>
<p>Emulating packet sampling at the router.
If you set this value higher than 1, the program will emulate 1/N sampling at
the NetFlow source routers.
Enabling this will decrease number of records be inserted.</P>
<p>If your target routers can create NetFlow data based on sampled packets, you should use the router's function rather than this option. Sampling at the routers is more effective to decrease the router load and save the bandwidth between the router and the host where NetFlow2MySQL run on.</p>
<h1><A name="6.">6. License</A></h1>
<p>This software is released under the BSD license.</p>
<p>Some libraries linked by the compiling may be under the GNU Lesser General Public License or other licences.</p>
<h1><A name="7.">7. Contact</A></h1>
<p>We have a discussion forum at <a href="http://cluster19.aist-nara.ac.jp/public/forum/">http://cluster19.aist-nara.ac.jp/public/forum/</a>. You can
post any articles related to NetFlow2MySQL and pcNetFlow, including Questions, HOW-TOs, TO-DOs or NetFlow protocol itself.</p>
<address>
<p>Author: FUJII Satoshi <<a href="mailto:sato-fu@wide.ad.jp">sato-fu@wide.ad.jp</a>><br>
<a href="http://www.aist-nara.ac.jp/~sato-fu/">http://www.aist-nara.ac.jp/~sato-fu/</a><br>
Our related softwares and examples can be found at <a href="http://cluster19.aist-nara.ac.jp/public/">http://cluster19.aist-nara.ac.jp/public/</a></p>
</address>
<h1><A name="8.">8. Changes</A></h1>
<H2>04/09/2005 Version 0.24 Malformed packet error may be fixed.</H2>
<H2>09/05/2004 Version 0.23 Add function to try to set larger SO_RCVBUF to avoid NetFlow packet loss.</H2>
<H2>09/01/2004 Version 0.22-1 Only add FAQ topics, no changes in the sources.</H2>
<H2>08/20/2004 Version 0.22 Fixed a bug in hanbling ENDIANs in BSD systems. Fixed internal overflow possibilities.</H2>
<H2>07/30/2004 Version 0.21 Add a feature to store NetFlow exporter's IP addresses.</H2>
<H2>06/18/2004 Version 0.11 Fixed problems in handling NetFlow v9 packets from multiple routers.</H2>
<H2>05/23/2004 Version 0.1b Fixed a critical syntax error in tables.sql</H2>
<H2>05/15/2004 Version 0.1a Fixed minor problems in tables.sql</H2>
<p>I noticed that tables made by old tables.sql cannot grow larger than 4GB in some plathomes. If you handle 4GB or more records, you should re-create tables by new tables.sql or see <A href="http://dev.mysql.com/doc/mysql/en/Full_table.html">http://dev.mysql.com/doc/mysql/en/Full_table.html</A> and set huge MAX_ROWS on your header and record tables. ALTER TABLE may take 1 or more hours on large database and it will lock your tables. Note that you will miss your traffic data while altering.</p>
<H2>03/25/2004 Version 0.1 initial release</H2>
</BODY>
</HTML>
⌨️ 快捷键说明
复制代码
Ctrl + C
搜索代码
Ctrl + F
全屏模式
F11
切换主题
Ctrl + Shift + D
显示快捷键
?
增大字号
Ctrl + =
减小字号
Ctrl + -