一种基于Heritrix可配置主题的聚焦爬虫方法

楚雄师范学院学报 ›› 2020, Vol. 35 ›› Issue (6): 124-131.

一种基于Heritrix可配置主题的聚焦爬虫方法

王松^1,*, 刘洪基¹, 叶晓波²

1.楚雄师范学院经济与管理学院,云南楚雄 675000;
2.楚雄师范学院国有资产与信息化管理处,云南楚雄 675000

收稿日期:2020-05-06 出版日期:2020-11-20 发布日期:2021-03-29
通讯作者: * 王松（1976–）,男,硕士,楚雄师范学院经济与管理学院高级实验师,研究方向为大数据挖掘管理与分析、机器学习、数据库。E-mail：ws@cxtc.edu.cn,Tel. 13508789000

A Focused Crawler Method of Configurable Theme Based Heritrix

WANG Song, LIU Hongji, YE Xiaobo

School of Economics & Management, Chuxiong Normal University, Chuxiong, Yunnan Province 675000;
Dept. of Management of State-owned Assets and Informationalization, Chuxiong Normal University, Chuxiong, Yunnan Province 675000

Received:2020-05-06 Online:2020-11-20 Published:2021-03-29

摘要/Abstract

摘要： 通用搜索引擎存在不能有针对性地满足用户查询需求和搜索关键词难以准确描述的问题。从数据挖掘和机器学习的角度出发,提出一种基于网络爬虫开源框架Heritrix的可配置主题的聚焦爬虫方法,从指定的站源,根据不同的爬取策略,启动多线程爬取,按照预先设置的关键字和栏目信息进行分类搜索,把最符合条件和要求的信息爬取下来供选择、判断、分析和利用。这种方法可在一定程度上解决搜索引擎查询信息的需求问题,提升用户体验,提高检索效率。

关键词: 聚焦爬虫, 可配置主题, Heritrix

Abstract: During the time of development of the Internet, massive information was generated in the cyber-world and has become an important asset. Meanwhile users’ requirement on information search has become higher and higher. How to search key information quickly and effectively is one of the most difficult problems to solve. Basically, the search engine satisfies needs in data searching. However, needs of users only focusing on special themes or fields cannot be satisfied. Through searching key words only is hard to describe their needs or their problems. Thus, this study focuses on data mining and machine learning and proposes a crawler method of configurable theme focused on crawler system that is based on open-source framework of web crawler Heritrix. To a certain extent this method can solve the above mentioned problems and improve users’ perception and searching efficiency.

Key words: focused crawler, configurable theme, Heritrix

中图分类号:

G252.7

王松, 刘洪基, 叶晓波. 一种基于Heritrix可配置主题的聚焦爬虫方法[J]. 楚雄师范学院学报, 2020, 35(6): 124-131.

WANG Song, LIU Hongji, YE Xiaobo. A Focused Crawler Method of Configurable Theme Based Heritrix[J]. Journal of Chuxiong Normal University, 2020, 35(6): 124-131.

参考文献

[1] 左薇,张熹,董红娟,等.主题网络爬虫研究综述[J].软件导刊,2020,19(2):278–281.
Zuo W,Zhang X,Dong H J,et al.Overview of research on topic-focused web crawler[J].Software Guide,2020,19(2):278–281.
[2] 刘强.复合规则驱动聚焦爬虫系统的设计与实现[D].哈尔滨:哈尔滨工业大学,2016.
Liu Q.The design and implementation of the complex rules-driven focused crawler system [D].Harbin:Harbin Institute of Technology,2016.
[3] 刘晨晖.中文文章与主题关键短语提取方法研究[D].西安:西安理工大学,2019.
Liu C H.Research on extraction methods of Chinese articles and topic key phrases [D].Xian:Xian University of Technology,2019.
[4] Vani K,Gupta D.Detection of idea plagiarism using syn-tax-semantic concept extractions with genetic algorithm[J].Expert Systems with Applications,2017,73:11–26.
[5] 王玮.基于Bi-LSTM-6Tags的智能中文分词方法[J].计算机应用,2018,38(S2):107–110.
Wang W.Smart Chinese word segmentation method based on Bi-LSTM- 6Tags[J].Journal of Computer Applications,2018,38(S2):107–110.
[6] Varatharajan R,Manogaran G,Priyan M K.A big data classification approach using LDA with an enhanced SVM method for ECG signals in cloud computing[J].Multimedia Tools and Appli-cations,2017,77(8):10195–10215.
[7] 邵云飞.融合主题模型与词向量的短文本分类方法研究[D].西安:西安电子科技大学,2019.
Shao Y F.Combining topic model and word embedding for short-text classification [D].Xian:Xidian University,2019.
[8] 龚静,黄欣阳.基于k最近邻和改进TF-IDF的文本分类框架[J].计算机工程与设计,2018,39(5):1340–1344.
Gong J,Huang X Y.Text categorization framework based on improved TF-IDF and k-nearest neighbor[J].Computer Engineering and Design,2018,39(5):1340–1344.
[9] Wang P,Zheng H C,Chen D Y,et al.Exploring the critical factors influencing online lending intentions[J].Finacial Lnnovation,2015,(1):8.
[10] 许甜华,吴明礼.一种基于TF-IDF的朴素贝叶斯算法改进[J].计算机技术与发展,2020,30(2):75–79.
Xu T H,Wu M L.An improved naive Bayes algorithm based on TF–IDF[J].Computer Technology and Development,2020,30(2):75–79.
[11] 夏修臣,王秀英.基于余弦相似度的改进C4.5决策树算法[J].计算机工程与设计,2018,39(1):120–125.
Xia X C,Wang X Y.Improved C4.5 decision tree algorithm based on cosine similarity[J].Computer Engineering and Design,2018,39(1):120–125.
[12] Khan M N A,Mahmood A.A distinctive approach to obtain higher page rank through search engine optimization[J].Sādhanā,2018,43( 3) :43.
[13] 林椹尠,袁柱,李小平.结合文本密度的语义聚焦爬虫方法[J].计算机应用与软件,2019,36(9):270–275.
Lin Z X,Yuan Z,Li X P.Semantic focused crawler method combining text density[J].Computer Applications and Software,2019,36(9):270–275.
[14] 林椹尠,袁柱,李小平.一种主题自适应聚焦爬虫方法[J].计算机应用与软件,2019,36(5):316–321.
Lin Z X,Yuan Z,Li X P.A topic adaptive focusing crawler method[J].Computer Applications and Software,2019,36(5):316–321.
[15] 赵康.面向主题的网络爬虫系统的设计与实现[D].北京:北京邮电大学,2019.
Zhao K.The design and implementation of the topic-focused web crawler system [D].Beijing:Beijing University of Posts and Telecommunications,2019.
[16] Tarik B,Mahahmoud D D,Zakaria E.Classifying Web pages by aimed nation using machine learning[J].International Journal of Organizational and Collective Intelligence,2017,7(1) :20–35.