138.txt

来自「This complete matlab for neural network」· 文本 代码 · 共 27 行

发信人: iamgufeng (古风), 信区: DataMining
标  题: Re: How to get the website topology
发信站: 南京大学小百合站 (Fri Dec 28 21:38:23 2001), 站内信件

: 1.links hiden in scripts
----if simple scripts just like "window.open()",it is easy to parse. for links
----generated by complex encoding functions,hope someone tell me the answer,hehe

: 2.advertizement links and irrelated links
----pre or post filter url by type or other attributes

: 3.the site keeps changing.
----when download a page we can store some http header fields such as
----"Last-Modified"

: 4.dynamic web pages.
----by my experience,the dynamic urls referring similar content have analogical
----structure generally. e.g. /cjzh/cjzh_detail.asp?nid=xxxxxxxx
----                /bbs/show.asp?BoardID=x&RootID=xxxxxx&ID=xxxxxx


This is a good topic to discuss,we can write a site spider ourselves.
come on,these experts help us please ;)

--
※ 来源:．南京大学小百合站 bbs.nju.edu.cn．[FROM: 210.83.133.26]