●Stories
●Firehose
●All
●Popular
●Polls
●Software
●Thought Leadership
Submit
●
Login
●or
●
Sign up
●Topics:
●Devices
●Build
●Entertainment
●Technology
●Open Source
●Science
●YRO
●Follow us:
●RSS
●Facebook
●LinkedIn
●Twitter
●
Youtube
●
Mastodon
●Bluesky
Become a fan of Slashdot on Facebook
Forgot your password?
Close
This discussion has been archived.
No new comments can be posted.
Load All Comments
Full
Abbreviated
Hidden
/Sea
Score:
5
4
3
2
1
0
-1
More
Login
Forgot your password?
Close
Close
Log In/Create an Account
●
All
●
Insightful
●
Informative
●
Interesting
●
Funny
The Fine Print: The following comments are owned by whoever posted them. We are not responsible for them in any way.
bySlashbotAgent ( 6477336 ) writes:
Destroying website? No, that's bullshit.
Destroying website page views by giving the user the data without attribution or even visiting the site? Yea. that's totally happening.
It's not damaging any sites. It's damaging the revenue of a few sites and their pissed. Perhaps rightly so. But the horses have left the barn and the barn has burned down.
bychristerk ( 141744 ) writes:
As someone who's actively fighting this type of traffic, let me share my perspective.
I have been running a small-ish website with user peaks at around 50 requests per second. Over the last couple of months, my site is getting hit with loads of up to 300 requests per second by these kinds of bots. They're using distributed IPs, and random user agents making it hard to block.
My site has a lot of data and pages to scan, and despite an appropriate robots.txt, these things ignore that and just scan endlessly. My
byHalo1 ( 136547 ) writes:
Anubis [github.com] has worked well for us to get rid of most of the scrapers from our wiki, including the ones faking regular user agents.
byallo ( 1728082 ) writes:
Anubis has the side effect that it stops the internet archive crawler.
Parent
twitter
facebook
byHalo1 ( 136547 ) writes:
Anubis has the side effect that it stops the internet archive crawler.
Even though it whitelists [github.com] the IA crawlers by default?
byallo ( 1728082 ) writes:
I am not sure what they are whitelisting, but I've seen pages in the archive that only show the anubis girl. Maybe it was from an older version, but I saw the page less than 5 month ago.
●threshold.
There may be more comments in this discussion. Without JavaScript enabled, you might want to turn on Classic Discussion System in your preferences instead.
Slashdot
●
●
Submit Story
It is much harder to find a job than to keep one.
●FAQ
●Story Archive
●Hall of Fame
●Advertising
●Terms
●Privacy Statement
●About
●Feedback
●Mobile View
●Blog
Do Not Sell or Share My Personal Information
Copyright © 2026 Slashdot Media. All Rights Reserved.
×
Close
Working...
Working...