Employing a novel content-based similarity measure for a machine learning-driven focused crawler

The volume of the World Wide Web is growing rapidly, reaching a point where governing data is challenging. Search engines are used to collect data across the web for users. Web crawlers as the major part of search engines are then used to retrieve relevant data on the web according to the user requests. Accordingly, a focused crawler considers a predefined subject and retrieves corresponding relevant pages. In this paper, we propose an efficient focused web crawling approach, which uses a combination of a content-based similarity measure and a Naive Bayes learning classifier in order to find relevant pages to a particular subject. Our first experimental studies show satisfactory improvements where accuracy and recall are increased by 4% and 1% respectively.

Keywords:

Focused crawler , Web crawler , Naive Bayes classification , Relevant page , TF-IDF criteria