LE9 - Web Analytics

Crawling is essentially following links, both internal and external.

Parsing means to make something understandable (by analyzing its parts). Convert information represented in one form into another form that is easier to work with

• tags and elements

• attributes

• platform and language nautral interface

• allow programs and scripts to dynamically access and update the content, structure and style of documents

• documents can be furhter processed and the results if that processing can be incorporated back into the presented page

is an Object Model for HTML -> it defines:

• HTML elements as objects

• Properties for all HTML elements

• Methods for all HTML elements

• Events for all HTML elements

• document is represented as a tree nodes (DOM tree)

• A node have child nodes with a direct parent

• Sibling node -> same level

• Meta language for the definition of markup languages

• IDEA: Metadata about structure, format, partly semantics become part of the message itself. Thus adaptable to sender and receiver

• HTLM: display data on web page

• XML: describe data and information

• DTD: Document Type Definition

• XML got stricter rules than HTML

• check if document conforms rules a validation step can be included after DOM has ben created by setting the validate argument = TRUE

• relevant extensions of XML => RSS

• R is not a typical language for web scraping

• alternatives

  • Python

  • Java

  • C++

Problems by web scrape:

• IP Blockade, Captcha triggers, request throttling

=> choose the path of least resistence

=> follow rules of the webmaster

=> be ethical

• created after a badly behaved crawler caused a denial of service attak on Kosters server

• REP (robots.txt) instruct crawlers how to crawl and index pages on their website

• webmasters usually defince rules for crawlers, -> www. xxx / robots.txt

• Compliance with REP protects against

  • Tripwires, IP adres blocking / blacklisting, misguidance, legal aftermath

• script is created that will work for long time

• request data at reasonable rate

• only request and save data you absolutely need from the page

• scrape during off peak hours

• respect the copyright lawy -> check website terms of service