preventing search engines to index accounts
In the past months I have noticed search bots getting data from all kinds of search bots (fedisearch.com). Within Mastodon and Pixelfed you can set the config to make sure noindex is being on your account. Problem is that there is no way to tell if bots respect those settings. Besides that there is no support for people to add an entry to a robots.txt file in de root of the machine. Certain bots still use that as a valid file to see what they can index.
To prevent further indexing I started to simply make privacy groups with Hubzilla so I can be sure these postings will not be read by bots. This works well on hubzilla (I can still make a public post) but on other platforms this is not possible unless you work with dm's
I would like to see what others might find a better solution and I have a few suggestions:
- Give people an per post option for it to be indexed or block all bots
- Give people a config option to block all bots all of the time. (general config).
Bots ignoring these settings can be blocked by the admin.
The first suggestion would make it possible, if activity pub finds it useful too, to enable selective indexing on Mastodon too (it isn't possible there). It would also prevent search engines to index comments you make on posts other people wrote (yes, per comment would be the same option as per post)
People have lots of reasons to be indexed or not. Would be nice if those who do not like it would have options too`
If there is some kind of anonymized database available I would be able to help test this.