A Focused Crawler Based on Topic Mining Approach

A focused crawler is a crawler that returns relevant pages related to a specific topic. This kind of crawler hasadvantages over a general crawler because of low cost and efficiency. In this paper, we present and implement asimple, focused crawler that works based on topic mining approach. This focused crawler, starts from specific seeds,calculates the similarity of each page with a specific page that we have given as our initial topic. With thesecalculations that are based on vectors dot products, we sort all of the pages based on this score, and therefore at theend we have a set of relevant pages to the specific page.

Keywords:

focused crawler , topic mining , information retrieval , cosine similarity