You can now keep an eye on the bandwidth used per crawl.

Pages = Downloaded in full to find page links.
Images / CSS / JavaScript = HTTP Header information only, reducing bandwidth used and keeping it speedy.

We are doing our best to be a polite #webcrawler.
#buildinpublic