📄 faq.html
字号:
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.0 Transitional//EN">
<html>
<head>
<title>NetFlow2MySQL FAQ</title>
</head>
<body>
<h1>NetFlow2MySQL FAQ</h1>
<ol>
<li>
<p>Q1. NetFlow2Mysql says 'The Table "record" is full', I cannot store traffic data anymore!</p>
<p>A1. The problem is fixed in new tables.sql with version 0.1a. Tables made by old tables.sql cannot grow larger than 4GB in some plathomes. If you need to store more than roughly 20 million records, you should re-create tables by new tables.sql or see <A href="http://dev.mysql.com/doc/mysql/en/Full_table.html">http://dev.mysql.com/doc/mysql/en/Full_table.html</A> and set huge MAX_ROWS on your header and record tables. ALTER TABLE may take 1 or more hours on large database and it will lock your tables. Note that you will miss your traffic data while altering.</p></li>
<li>
<p>Q2. I collect traffic data from multiple routers. How to identify which records are from which routers? </p>
<p>A2. <STRIKE>You should not handle NetFlow packets from multiple routers in one NetFlow2MySQL process. Current version of NetFlow2MySQL does not assume such a usage. Please launch 1 NetFlow2MySQL per 1 NetFlow exporter in different listening ports. You should also use 1 database per 1 exporter. It is a simple way to identify record's origins.</STRIKE> See section 4.4. of <a href="./NetFlow2MySQL.html#4.4">NetFlow2MySQL.html</A>.</p>
</li>
<li>
<p>Q3. My SQL query requires too much time to return the result.</p>
<p>A3. There are some solutions to reduce the query responce time.</p>
<ol>
<li><p>Use WHERE syntax effectively in your queries.<br>
Try to use WHERE syntax on your queries to specify a time span, or any other parameters like port number, address, etc.... It will reduce required time. Observed time of each flows can be specified by unix_seconds, but they have maximum 30 min. margins from actual observed time of each packets.
</p></li>
<li>
<p>Reduce the number of records on DB.<br>
Setting higher sampling-interval on the routers will reduce the number of records on the DB. It helps reducing query response time. It also helps reducing the required disk space. Although it has bad effect to the preciseness of the traffic measurement.
</p>
</li>
</ol>
<li>
<p>Q4. I want to check that all NetFlow packets are successfully inserted into DB.</p>
<p>A4. In case of NetFlow version 9, if there are no packet loss, values of package_sequence on your header table should be sequentially increased per router. In NetFlow version 5 case, package_sequence stands for sequence of NetFlow records, so you can check as following.</p>
<pre>
NetFlow Version 9 Example
select * from header;
+-----+---------+---------------+--------------+------------------+-----------+
| hid | version | system_uptime | unix_seconds | package_sequence | source_id |
+-----+---------+---------------+--------------+------------------+-----------+
| 1 | 9 | 2913126284 | 1076000695 | 794498 | 257 |
| 2 | 9 | 2913129284 | 1076000698 | 794499 | 257 |
| 3 | 9 | 2913131284 | 1076000700 | 794500 | 257 |
+-----+---------+---------------+--------------+------------------+-----------+
^-increase sequencially if no loss
</pre>
<pre>
NetFlow Version 5 Example
mysql> select * from header;
+-----+---------+---------------+--------------+------------------+-----------+
| hid | version | system_uptime | unix_seconds | package_sequence | source_id |
+-----+---------+---------------+--------------+------------------+-----------+
| 1 | 5 | 2386680739 | 1078135266 | 12551985 | 0 |
| 2 | 5 | 2386680739 | 1078135266 | 12552014 | 0 |
| 3 | 5 | 2386680739 | 1078135266 | 12552043 | 0 |
+-----+---------+---------------+--------------+------------------+-----------+
^
if difference between 2 records is (12552014 - 12551985) =
mysql> select count(*) from record where hid = 1;
+----------+
| count(*) |
+----------+
| 29 |
+----------+
then, we can say there are no loss between these 2 packets.
</pre>
</li>
<li>
<p>Q5. My NetFlow packets seem to lose. How can I do?</p>
<p>A5. There are some resons that cause packet loss.</p>
<ol>
<li>Shortage of network bandwidth.</li>
<li>Overload of PC that NetFlow2MySQL is working on.</li>
<li>Something locks your header and record table.</li>
</ol>
<p>No. 3 often occur. If you execute SQL statements(incl. SELECT, ALTER or something) to the 'header' and 'record' tables which NetFlow2MySQL is trying to INSERT new records into, your SQL statements may lock the tables. It forces NetFlow2MySQL to wait to INSERT new records. If your SQL takes long, usually more than a few minutes, newly arrived NetFlow packets will be lost because of UDP receive queue overflow.</p>
<ol>
<li>Use InnoDB instead of MyISAM.
<p>InnoDB provides row locks, NetFlow2MySQL may be able to INSERT table while SELECTing. I've not tested, but you can try it by modifying TYPE in tables.sql. Note that InnoDB is generally slower than MyISAM.
See <a href="http://dev.mysql.com/doc/mysql/en/InnoDB_locking_reads.html">http://dev.mysql.com/doc/mysql/en/InnoDB_locking_reads.html</a> for more details of the lock mechanism.</p></li>
<li>Increase UDP receive queue size.
<p>By increasing UDP receive queue size, more NetFlow packets can wait for being INSERTed. From version 0.23, NetFlow2MySQL try to set the receive queue size to 10MB. If failed, it try to set it to 5MB, 2.5MB, 1.25MB... Generally, actual queue size may be set much smaller than 10MB because of OS or kernel limitation.
On linux, setting large net.core.rmem_max value looses the limitation. Similar settings may be available in other platfomes. Usually it is very enough, but you can also set custom queue size by <B>--receive-buffer-size</B> option.</p>
</li>
</ol>
</li>
</li>
</ol>
<h1>Contact</h1>
<p>We have a discussion forum at <a href="http://cluster19.aist-nara.ac.jp/public/forum/">http://cluster19.aist-nara.ac.jp/public/forum/</a>. You can
post any articles related to NetFlow2MySQL and pcNetFlow, including Questions, HOW-TOs, TO-DOs or NetFlow protocol itself.</p>
<address>
<p>Author: FUJII Satoshi <<a href="mailto:sato-fu@wide.ad.jp">sato-fu@wide.ad.jp</a>><br>
<a href="http://www.aist-nara.ac.jp/~sato-fu/">http://www.aist-nara.ac.jp/~sato-fu/</a><br>
Our related softwares and examples can be found at <a href="http://cluster19.aist-nara.ac.jp/public/">http://cluster19.aist-nara.ac.jp/public/</a></p>
</address>
</BODY>
</HTML>
⌨️ 快捷键说明
复制代码
Ctrl + C
搜索代码
Ctrl + F
全屏模式
F11
切换主题
Ctrl + Shift + D
显示快捷键
?
增大字号
Ctrl + =
减小字号
Ctrl + -