• 0 Posts
  • 12 Comments
Joined 1 year ago
cake
Cake day: July 22nd, 2023

help-circle








  • robots.txt does not work. I don’t think it ever has - it’s an honour system with no penalty for ignoring it.

    I have a few low traffic sites hosted at home, and when a crawler takes an interest they can totally flood my connection. I’m using cloudflare and being incredibly aggressive with my filtering but so many bots are ignoring robots.txt as well as lying about who they are with humanesque UAs that it’s having a real impact on my ability to provide the sites for humans.

    Over the past year it’s got around ten times worse. I woke up this morning to find my connection at a crawl and on checking the logs, AmazonBot has been hitting one site 12000 times an hour, and that’s one of the more well-behaved bots. But there’s thousands and thousands of them.


  • If cookie prompts annoy you (and why wouldn’t they? Complicated and time wasting prompts caused by terrible and compromised legislation that’s led to far more intrusion instead of enforcing use of browser settings) and you don’t care about cookies, then the browser extension “I don’t care about cookies” suppresses the vast majority.



  • I think this type of scheme is illegal under the GDPR, which is in effect in the UK just as it is in the EU.

    It’s been a while since I worked with the GDPR, but from memory the wording is such that:

    The data holder needs to allow people to opt out of data collection. The subject can request to be forgotten. The data holder explicitly cannot charge for this.

    But changes move slow, and The Mirror is probably banking on nobody caring enough to complain, and Trading Standards being too underfunded and swamped with other work to investigate otherwise (which they are). If they’re challenged, they’ll just change tack, go “oops” and are unlikely to hit big fines unless they dig in.

    Cookie laws are a horrible mess and always have done - the resulting consent banners are far more intrusive than anyone wanted.