Handling of capsules entirely forbidden by robots.txt
As of today, capsules with a complete disallowance by the robots.txt (like hannuhartikainen.fi
) are treated as if they were unreachable. They are not counted in the "working capsules" tally and are purged after a delay. It may be better to treat them as working capsules.