网络爬虫论文VIP专享

下载后可任意编辑摘 要网络爬虫（Web Crawler），通常被称为爬虫，是搜索引擎的重要组成部分。随着信息技术的飞速进步，作为搜索引擎的一个组成部分——网络爬虫，一直是讨论的热点，它的好坏会直接决定搜索引擎的未来。目前，网络爬虫的讨论包括 Web 搜索策略讨论的讨论和网络分析的算法，两个方向，其中在 Web 爬虫网络搜索主题是一个讨论方向，根据一些网站的分析算法，过滤不相关的链接，连接到合格的网页，并放置在一个队列被抓取。把互联网比方成一个蜘蛛网，那么 Spider 就是在网上爬来爬去的蜘蛛。网络蜘蛛是通过网页的链接地址来寻找网页，从网站某一个页面（通常是首页）开始，读取网页的内容，找到在网页中的其它链接地址，然后通过这些链接地址寻找下一个网页，这样一直循环下去，直到把这个网站所有的网页都抓取完为止。假如把整个互联网当成一个网站，那么网络爬虫就可以用这个原理把互联网上所有的网页都抓取下来。关键词：网络爬虫；Linux Socket；C/C++;多线程；互斥锁下载后可任意编辑AbstractWeb Crawler, usually called Crawler for short, is an important part of search engine. With the high-speed development of information, Web Crawler-- the search engine can not lack of-- which is a hot research topic those years. The quality of a search engine is mostly depended on the quality of a Web Crawler. Nowadays, the direction of researching Web Crawler mainly divides into two parts: one is the searching strategy to web pages; the other is the algorithm of analysis URLs. Among them, the research of Topic-Focused Web Crawler is the trend. It uses some webpage analysis strategy to filter topic-less URLs and add fit URLs into URL-WAIT queue.The metaphor of a spider web internet, then Spider spider is crawling around on the Internet. Web spider through web link address to find pages, starting from a one page website (usually home), read the contents of the page, find the address of the other links on the page, and then look for the next Web page addresses thr...