I wonder this about the massive increase in crawlers too.

What happened to the computer misuse act? If I specifically state that my site is not to be crawled, via robots.txt and other mechanisms, why does continuously hammering it not count as illegitimate access? Do they need to breach some sign in / explicit t+c agreement for that to apply?