Robots.txt (a piece of code designed to limit crawler activity within a website) was ignored with the permission of the content owner. Crawl was limited to domains and subdomains of http://www.autonomedia.org/ in order to remain within the collection scope and data constraints.
