A new robots.txt parser

The one shipped with Python is quite complicated to use, my own parser is too limited (it parses only the original syntax), the rule of the longest match in more recent specifications make difficult to use regular expressions (the current implementation), to summarize: the current state of robots.txt handling is not good.