Web Content Extraction Using Contextual Rules

Extracting the main content from web pages has many applications, such as mobile phone browsing, enhancing the page readability and speech rendering for the visually impaired. In applications that provide a service to end users, identifying the content of interest is better served with user assistance through a visual tool rather than an unsupervised method. In this paper, we propose a wrapping language supported by a visual tool to create wrappers for extracting the main content from web pages. The language is designed to be easy to use, and expressive enough to cover most common scenarios. In this language, various types of features (syntactical, semantic, visual, densitometric) can be employed in the extraction rules to identify the content of interest. Moreover, contextual information can be utilized as context variables to restrict the application of each rule to certain parts of the page and refining their content. Furthermore, the rules can be organized hierarchically to share common rules among wrappers for similar websites. The system is particularly suitable for extracting the main content from blogs, news and encyclopedia websites.

Keywords:

Web mining , Main Content Extraction , Web wrappers