The British Library UK Web Domain Research Project
Information for webmasters
You have probably reached this page because you have noticed our crawler has visited your site and left an entry in your access logs.
The British Library is currently undertaking a research project to assess the feasibility of archiving the UK domain. In an attempt to determine the boundaries of the UK domain, your site has been identified as ‘in scope’ or has been linked from a site that is ‘in scope’ of the UK domain. The content harvested at this time will not be made available.
This work is undertaken in anticipation of forthcoming Legal Deposit regulations that will make it the Library’s statutory responsibility to collect, preserve and provide long-term access to the UK’s online intellectual and cultural heritage.
Robots.txt directives are being followed and the crawlers are set to be polite. A 5x multiple of the time it takes to retrieve a document is used to limit the time between sequential requests.
If you have an objection to further crawls of your site please send your concerns to:
Email LDCrawlQueries@bl.uk
Please identify yourself, your site and your objections and we will respond accordingly.
