⭐ 欢迎来到虫虫下载站！ | 📦 资源下载 📁 资源专辑 ℹ️ 关于我们
⭐ 虫虫下载站
📤 上传资源

📄 crawler.html

📁 爬虫设计
💻 HTML
📖 第 1 页 / 共 5 页
字号:
🔍
12345 
	{margin-bottom:0cm;}
-->
</style>
</head>

<body lang=ZH-CN link=blue vlink=purple style='tab-interval:21.0pt;text-justify-trim:
punctuation'>

<div class=Section1 style='layout-grid:15.6pt'>

<p class=MsoNormal align=center style='text-align:center'><span lang=EN-US
style='font-size:16.0pt'>Intranet</span><span style='font-size:16.0pt;
font-family:宋体;mso-ascii-font-family:"Times New Roman"'>环境下守法爬虫程序的设计与应用</span><span
lang=EN-US style='font-size:16.0pt;color:red'><o:p></o:p></span></p>

<p class=MsoNormal align=center style='text-align:center'><span
style='mso-bidi-font-size:10.5pt;font-family:宋体;mso-ascii-font-family:"Times New Roman"'>付东来</span><span
style='mso-bidi-font-size:10.5pt'> </span><span style='mso-bidi-font-size:10.5pt;
font-family:宋体;mso-ascii-font-family:"Times New Roman"'>李</span><span
style='mso-bidi-font-size:10.5pt;mso-hansi-font-family:宋体'> </span><span
style='mso-bidi-font-size:10.5pt;font-family:宋体;mso-ascii-font-family:"Times New Roman"'>祥</span><span
lang=EN-US style='mso-bidi-font-size:10.5pt'><o:p></o:p></span></p>

<p class=MsoNormal align=center style='text-align:center'><span lang=EN-US
style='mso-bidi-font-size:10.5pt'>(</span><span style='mso-bidi-font-size:10.5pt;
font-family:宋体;mso-ascii-font-family:"Times New Roman"'>贵州大学计算机软件与理论研究所</span><span
style='mso-bidi-font-size:10.5pt'> </span><span style='mso-bidi-font-size:10.5pt;
font-family:宋体;mso-ascii-font-family:"Times New Roman"'>贵阳</span><span
lang=EN-US style='mso-bidi-font-size:10.5pt'> 550025)<o:p></o:p></span></p>

<p class=MsoNormal align=center style='text-align:center;mso-line-height-alt:
0pt'><span lang=EN-US>(</span><span style='font-family:宋体;mso-ascii-font-family:
"Times New Roman";mso-hansi-font-family:"Times New Roman"'>电子邮件：</span><span
lang=EN-US style='mso-bidi-font-size:10.5pt;color:black'>250776267@qq.com</span><span
lang=EN-US>)<span style='color:red'><o:p></o:p></span></span></p>

<p class=MsoNormal align=center style='text-align:center'><span
style='mso-bidi-font-size:10.5pt;font-family:宋体;mso-ascii-font-family:"Times New Roman";
mso-hansi-font-family:"Times New Roman";color:red'>（本文已发表）</span><span
lang=EN-US style='mso-bidi-font-size:10.5pt;color:red'><o:p></o:p></span></p>

<p class=MsoNormal style='mso-line-height-alt:0pt'><span style='font-size:9.0pt;
font-family:宋体;mso-ascii-font-family:"Times New Roman"'>摘要：爬虫程序是搜索引擎的关键部件，它负责发现和搜集信息。一个不遵守机器人禁止标准的爬虫程序不仅不会受到</span><span
lang=EN-US style='font-size:9.0pt'>Web</span><span style='font-size:9.0pt;
font-family:宋体;mso-ascii-font-family:"Times New Roman"'>站点管理员的欢迎，而且可能会给程序设计者带来一定的麻烦。本文在</span><span
lang=EN-US style='font-size:9.0pt'>NetBeans4.0</span><span style='font-size:
9.0pt;font-family:宋体;mso-ascii-font-family:"Times New Roman"'>下利用</span><span
lang=EN-US style='font-size:9.0pt'>BOT</span><span style='font-size:9.0pt;
font-family:宋体;mso-ascii-font-family:"Times New Roman"'>开发工具包实现了一个</span><span
lang=EN-US style='font-size:9.0pt'>Intranet</span><span style='font-size:9.0pt;
font-family:宋体;mso-ascii-font-family:"Times New Roman"'>环境下的守法爬虫程序。</span><span
lang=EN-US style='font-size:9.0pt'><o:p></o:p></span></p>

<p class=MsoNormal style='mso-line-height-alt:0pt'><span style='font-size:9.0pt;
font-family:宋体;mso-ascii-font-family:"Times New Roman"'>关键字：</span><span
lang=EN-US style='font-size:9.0pt'>Intranet,</span><span style='font-size:9.0pt;
font-family:宋体;mso-ascii-font-family:"Times New Roman"'>爬虫</span><span
lang=EN-US style='font-size:9.0pt'>,</span><span style='font-size:9.0pt;
font-family:宋体;mso-ascii-font-family:"Times New Roman"'>机器人禁止标准</span><span
lang=EN-US style='font-size:9.0pt;mso-hansi-font-family:宋体'>,</span><span
style='font-size:9.0pt;font-family:宋体;mso-ascii-font-family:"Times New Roman"'>搜索引擎</span><span
style='font-size:9.0pt'> <span lang=EN-US><o:p></o:p></span></span></p>

<p class=MsoNormal><span lang=EN-US style='mso-bidi-font-size:10.5pt'><![if !supportEmptyParas]>&nbsp;<![endif]><o:p></o:p></span></p>

<p class=MsoNormal align=center style='text-align:center'><span lang=EN-US
style='font-size:16.0pt'>Design and Application of A Crawler’s Program Abiding
by the Rules in Intranet Environment<o:p></o:p></span></p>

<p class=MsoNormal align=center style='text-align:center'><span lang=EN-US
style='mso-bidi-font-size:10.5pt'>Fu Donglai<span style="mso-spacerun:
yes">&nbsp; </span>Li Xiang<o:p></o:p></span></p>

<p class=MsoNormal style='text-indent:42.0pt;mso-char-indent-count:4.0;
mso-char-indent-size:10.5pt'><span style='mso-bidi-font-size:10.5pt;font-family:
宋体;mso-ascii-font-family:"Times New Roman"'>（</span><span lang=EN-US
style='mso-bidi-font-size:10.5pt'>Institute of Computer Science ,Guizhou
University,Guiyang ,550025</span><span style='mso-bidi-font-size:10.5pt;
font-family:宋体;mso-ascii-font-family:"Times New Roman"'>）</span><span
lang=EN-US style='mso-bidi-font-size:10.5pt;mso-hansi-font-family:宋体'><o:p></o:p></span></p>

<p class=MsoNormal align=center style='text-align:center'><span lang=EN-US>(Email</span><span
style='font-family:宋体;mso-ascii-font-family:"Times New Roman";mso-hansi-font-family:
"Times New Roman"'>：</span><span lang=EN-US style='mso-bidi-font-size:10.5pt;
color:black'>250776267@qq.com</span><span lang=EN-US>)</span><span lang=EN-US
style='mso-bidi-font-size:10.5pt'><o:p></o:p></span></p>

<p class=MsoNormal style='mso-line-height-alt:0pt'><span lang=EN-US
style='font-size:9.0pt'><![if !supportEmptyParas]>&nbsp;<![endif]><o:p></o:p></span></p>

<p class=MsoNormal style='mso-line-height-alt:0pt'><span lang=EN-US
style='font-size:9.0pt'>Abstract: A crawler’s program is one of search engine’s
key parts answering for finding and collecting infomati-o n.Administrators
dislike any crawler’s program violating the standard for robot exclusion. As a
result of using the program, its designer may stick in the mud. This paper
realizes a crawler’s program abiding by the rules by NetBeans 4.0 and a BOT
kit.<o:p></o:p></span></p>

<p class=MsoNormal align=left style='text-align:left;mso-line-height-alt:0pt;
mso-layout-grid-align:none;text-autospace:none'><span lang=EN-US
style='font-size:9.0pt'>Keywords: Intranet, </span><span lang=EN-US
style='font-size:9.0pt;mso-font-kerning:0pt'>Crawler, standard for robot
exclusion, search engine <o:p></o:p></span></p>

<p class=MsoNormal align=left style='text-align:left;mso-line-height-alt:0pt;
mso-layout-grid-align:none;text-autospace:none'><span lang=EN-US
style='font-size:9.0pt;mso-font-kerning:0pt'><![if !supportEmptyParas]>&nbsp;<![endif]><o:p></o:p></span></p>

<p class=MsoNormal align=center style='text-align:center'><span
style='font-size:14.0pt;font-family:宋体;mso-ascii-font-family:"Times New Roman"'>引</span><span
style='font-size:14.0pt;mso-hansi-font-family:宋体'> </span><span
style='font-size:14.0pt;font-family:宋体;mso-ascii-font-family:"Times New Roman"'>言</span><span
lang=EN-US style='font-size:14.0pt'><o:p></o:p></span></p>

<p class=MsoNormal><span lang=EN-US style='mso-bidi-font-size:10.5pt'><span
style="mso-spacerun: yes">&nbsp;&nbsp;&nbsp; </span></span><span
style='mso-bidi-font-size:10.5pt;font-family:宋体;mso-ascii-font-family:"Times New Roman"'>爬虫程序作为搜索引擎的关键部件，主要负责发现和搜集信息。虽然</span><span
lang=EN-US style='mso-bidi-font-size:10.5pt'>Intranet</span><span
style='mso-bidi-font-size:10.5pt;font-family:宋体;mso-ascii-font-family:"Times New Roman"'>采用了</span><span
lang=EN-US style='mso-bidi-font-size:10.5pt'>Internet</span><span
style='mso-bidi-font-size:10.5pt;font-family:宋体;mso-ascii-font-family:"Times New Roman"'>技术构建，但</span><span
lang=EN-US style='mso-bidi-font-size:10.5pt'>Intranet</span><span
style='mso-bidi-font-size:10.5pt;font-family:宋体;mso-ascii-font-family:"Times New Roman"'>本身有着不同于</span><span
lang=EN-US style='mso-bidi-font-size:10.5pt'>Internet</span><span
style='mso-bidi-font-size:10.5pt;font-family:宋体;mso-ascii-font-family:"Times New Roman"'>的特殊性。因此，研究面向</span><span
lang=EN-US style='mso-bidi-font-size:10.5pt'>Intranet</span><span
style='mso-bidi-font-size:10.5pt;font-family:宋体;mso-ascii-font-family:"Times New Roman"'>的爬虫程序，能够为许多企、事业门户网站实现自主定制搜索引擎提供强有力的帮助，对于推动企、事业单位信息化进程有着非常重要的意义。本文在分析爬虫的工作原理、</span><span
lang=EN-US style='mso-bidi-font-size:10.5pt'>Intranet</span><span
style='mso-bidi-font-size:10.5pt;font-family:宋体;mso-ascii-font-family:"Times New Roman"'>特点的基础上，采用</span><span
lang=EN-US style='mso-bidi-font-size:10.5pt'>BOT</span><span style='mso-bidi-font-size:
10.5pt;font-family:宋体;mso-ascii-font-family:"Times New Roman"'>开发工具包实现了一个</span><span
lang=EN-US style='mso-bidi-font-size:10.5pt'>Intranet</span><span
style='mso-bidi-font-size:10.5pt;font-family:宋体;mso-ascii-font-family:"Times New Roman"'>环境下的守法爬虫程序。</span><span
lang=EN-US style='mso-bidi-font-size:10.5pt'><o:p></o:p></span></p>

<p class=MsoNormal align=center style='text-align:center'><span
style='font-size:14.0pt;mso-bidi-font-size:12.0pt;font-family:宋体;mso-ascii-font-family:
"Times New Roman";mso-hansi-font-family:"Times New Roman"'>一、</span><span
style='font-size:14.0pt;font-family:宋体;mso-ascii-font-family:"Times New Roman"'>爬虫程序与</span><span
lang=EN-US style='font-size:14.0pt'>BOT</span><span style='font-size:14.0pt;
font-family:宋体;mso-ascii-font-family:"Times New Roman"'>包介绍</span><span
lang=EN-US style='font-size:14.0pt'><o:p></o:p></span></p>

<p class=MsoNormal><span lang=EN-US style='mso-bidi-font-size:10.5pt'>1.1</span><span
style='mso-bidi-font-size:10.5pt;font-family:宋体;mso-ascii-font-family:"Times New Roman"'>爬虫程序的工作原理</span><span
lang=EN-US style='mso-bidi-font-size:10.5pt'><o:p></o:p></span></p>

<p class=MsoNormal style='margin-left:1.71gd;text-indent:17.95pt;mso-char-indent-count:
1.71;mso-char-indent-size:10.45pt'><span style='mso-bidi-font-size:10.5pt;
font-family:宋体;mso-ascii-font-family:"Times New Roman"'>爬虫程序一般是根据用户提供的</span><span
lang=EN-US style='mso-bidi-font-size:10.5pt'>URL</span><span style='mso-bidi-font-size:
10.5pt;font-family:宋体;mso-ascii-font-family:"Times New Roman"'>地址获得一个</span><span
lang=EN-US style='mso-bidi-font-size:10.5pt'>WEB</span><span style='mso-bidi-font-size:
10.5pt;font-family:宋体;mso-ascii-font-family:"Times New Roman"'>文档，分析网页代码，查找页内所有链接到其他网页的标签。然后，按照广度或深度爬行算法随机选择一个链接再分析并寻找新的链接，如此反复，理论上就可遍历所有的</span><span
lang=EN-US style='mso-bidi-font-size:10.5pt'>WEB</span><span style='mso-bidi-font-size:
10.5pt;font-family:宋体;mso-ascii-font-family:"Times New Roman"'>文档。</span><span
lang=EN-US style='mso-bidi-font-size:10.5pt'><o:p></o:p></span></p>

<p class=MsoNormal><span lang=EN-US style='mso-bidi-font-size:10.5pt'>1.2</span><span
style='mso-bidi-font-size:10.5pt;font-family:宋体;mso-ascii-font-family:"Times New Roman"'>机器人禁止标准</span><span
lang=EN-US style='mso-bidi-font-size:10.5pt'><o:p></o:p></span></p>

<p class=MsoNormal align=left style='margin-left:1.71gd;text-align:left;
text-indent:2.95pt;mso-char-indent-count:.28;mso-char-indent-size:10.5pt;
mso-layout-grid-align:none;text-autospace:none'><span lang=EN-US
style='mso-bidi-font-size:10.5pt'><span style="mso-spacerun: yes">&nbsp;&nbsp;
</span></span><span style='mso-bidi-font-size:10.5pt;font-family:宋体;mso-ascii-font-family:
"Times New Roman";mso-font-kerning:0pt'>在</span><span lang=EN-US
style='mso-bidi-font-size:10.5pt;mso-font-kerning:0pt'>1994</span><span
style='mso-bidi-font-size:10.5pt;font-family:宋体;mso-ascii-font-family:"Times New Roman";
mso-font-kerning:0pt'>年当互连网还处于起步状态时，就有一批</span><span lang=EN-US
style='mso-bidi-font-size:10.5pt;mso-font-kerning:0pt'>Internet</span><span
style='mso-bidi-font-size:10.5pt;font-family:宋体;mso-ascii-font-family:"Times New Roman";
mso-font-kerning:0pt'>用户认识到不守秩序的爬虫程序可能导致一些问题的发生，并为此开发了一个非官方标准，即</span><span
lang=EN-US style='mso-bidi-font-size:10.5pt;mso-font-kerning:0pt'>“</span><span
style='mso-bidi-font-size:10.5pt;font-family:宋体;mso-ascii-font-family:"Times New Roman";
mso-font-kerning:0pt'>机器人禁止标准</span><span lang=EN-US style='mso-bidi-font-size:
10.5pt;mso-font-kerning:0pt'>”</span><span style='mso-bidi-font-size:10.5pt;
font-family:宋体;mso-ascii-font-family:"Times New Roman";mso-font-kerning:0pt'>。标准定义了一个允许网络管理员禁止爬虫程序访问其</span><span
12345

⌨️ 快捷键说明

复制代码 Ctrl + C
搜索代码 Ctrl + F
全屏模式 F11
切换主题 Ctrl + Shift + D
显示快捷键 ?
增大字号 Ctrl + =
减小字号 Ctrl + -