Web Robots Bibliography

[Heydon & Najork 1999]
Allan Heydon, Marc Najork : Mercator: A Scalable, Extensible Web Crawler, 1999.