135.txt

来自「This complete matlab for neural network」· 文本 代码 · 共 40 行

发信人: easehawking (以鹰行猎), 信区: DataMining
标  题: Re: How to get the website topology
发信站: 南京大学小百合站 (Fri Dec 28 20:20:00 2001)

Yours is a HTML parser plus directed-graph trasversing.
HTML parser can be coded by LEX+YACC.

There are more to consider to make it pratical:
1.links hiden in scripts
2.advertizement links and irrelated links
3.the site keeps changing.
4.dynamic web pages.

【 在 helloboy 的大作中提到: 】
: My algorithm is to use a queue to store unanalysed pages.
: Then we get one out from queue and create a node ,at the same
: time insect its linkage pages into the queue.
: Go on this cycle until the queue is empty
: But I am still thinking...
: because some automated dl images,such as gif,jpg, will make
: us annoyed for they make the topology so complicated.
: Perhaps I can use some prune algorithm to get rid of
: these image files.

you can know what kinda files they are by the http header.

: 【 在 iamgufeng (古风) 的大作中提到: 】
: : Can you share your algorithm with us?
: : I think it is not very difficult to traverse the website along links
: : in pages. but how to identify real/best topology because of the complex
: : links between pages. For instance,we all like create links on homepage
: : to refer to the important content of other page level. Perhaps we should

: : adjust the extracted topology manually.

--
清除前进路上的一切障碍！

※ 来源:．南京大学小百合站 http://bbs.nju.edu.cn [FROM: 202.38.215.11]