Website Scraping

May 14, 2008

In recent years, a number of comparative websites have appeared online in which consumers can compare prices of a number of businesses without the need to search individual websites.

Websites such as Expedia use special web crawling software which accesses information from various websites in order to obtain information – key to the consumer being price information. Whilst the technique is of benefit to consumers in many ways, as the user can compare a number of prices or products with one search, there are potentially legal implications for those websites that employ web crawling software to search other websites.

The websites that are subjected to web scrapping by web crawler software are faced with a dilemma. One the one hand, there is potential for extra business generated through referrals from a comparison website, whilst on the other hand it distracts consumers away from their own website and their own marketing and branding. There is also an increase in website traffic, pushing up maintenance costs and slowing the user experience down when their website is accessed directly.

What remedies there are to a website owner could, in many instances, be limited. Whilst website terms and conditions may prohibit use of web crawling software to extract information, an automated machine does not read, understand and agree to terms and merely extracts information from the website. The consequence is that there is an issue of whether such terms of use could ever be incorporated and enforceable.

As websites have their information organised in databases which are arranged in a systematic and methodical way, there could be some reprieve for European Union businesses through the Database Directive. However, this is untested and in a national decision of the Danish Maritime and Commercial Court (Copenhagen), the court ruled in 2006 (in Danish only) that using deep linking and extracting information using web crawling software was not in conflict with national law or the Database Directive. It remains to be seen if other member states take the same view, and what the European Court of Justice’s view would be.

If the access is unauthorised, there is an argument to support the idea that the Computer Misuse Act 1990 is breached – it is likely that the access provided by web crawling software is unauthorised access to computer material. Any aggrieved website owner would of course still have the task of persuading the authorities to initiate criminal proceedings against an alleged offender.

For the meantime, websites who are aggrieved by web scrapping seem to have little recourse in law, and may for the meantime be better served by technological means such as road blocking software if they want to prevent web scraping.