Albert Weichselbraun

Web Content Extraction

1 minute read

Introduction

The amount of information available in the Internet increases continuously, providing interesting research options that in many fields, including informatics and social sciences.

Many of these methods are text corpus based and therefore require an accurate textual representation of the investigated Web pages. Advertisements, navigational elements and more recently Web 2.0 techniques, considerably complicate the task of extracting the Web page’s content.

Web content extraction is concerned with extracting the relevant text from Web pages by removing unrelated textual noise like advertisements, navigational elements, contact and copyright notes.

The goals of this thesis are to

  1. define requirements for content extraction techniques applicable to a search and web analysis tool,
  2. investigate state of the art content extraction techniques, and
  3. implement and test one of those techniques.

Structure

  1. introduction and problem definition
  2. requirement analysis
  3. content extraction techniques
  4. implementation
    1. justification of the chosen technique
    2. system architecture and implementation details
  5. testing
  6. outlook and conclusions

Literature

  1. Juffinger, A., Neidhart, T., Granitzer, M., R., Kern, Weichselbraun, A., Wohlgenannt, G. and Scharl, A. (2007). Distributed Web2.0 Crawling for Ontology Evolution, Proceedings of the Second International Conference on Digital Information Management (ICDIM’07)
  2. Gottron, Thomas (2008). Content Code Blurring: A New Approach to Content Extraction, Nineteenth International Workshop on Database and Expert Systems Application (DEXA 2008); Fifth International Workshop on Text-based Information Retrieval (TIR’08), ISBN: 978-0-7695-3299-8, IEEE Computer Society Press, pages 29–33
  3. Weninger, Tim and William, Hsu H. (2008). Text Extraction from the Web via Text-to-Tag Ratio, Nineteenth International Workshop on Database and Expert Systems Application (DEXA 2008); Fifth International Workshop on Text-based Information Retrieval (TIR’08), ISBN: 978-0-7695-3299-8, IEEE Computer Society Press, pages 23–28

Updated:

You may also enjoy