Skip to content

New robots.txt handling, with the standard Python library robotparser. Closes #19

Stéphane Bortzmeyer requested to merge robotparser into master

Merge request reports