Accidentally ran a Python script in an infinite loop for 45 minutes yesterday

I was trying to scrape a list of 200 URLs but forgot to add a sleep timer and it hammered the site so hard my IP got temporarily banned. Lesson learned - now I always throw in a time.sleep(1) between requests to play nice with servers. Anybody else get blocked by a site while learning?

3 comments

3 Comments

milaw1422d ago

Hang on, a single 45 minute scrape got your IP banned? Most sites have rate limits way higher than that unless you were hitting them nonstop. Sounds more like you were flooding their server with hundreds of requests per second or something. A lot of people blame the site when really they just went way overboard. Throw a delay in there, sure, but dont act like this is a giant life lesson.

the_miles22d ago

Whoa hold up, I think everyone is missing something here. What if the site just has a terrible bot detection setup? I've seen some smaller sites that use the WORST rate limiting code that just flags ANY repeated requests in a short window, even like 2 or 3 per minute. So a 45 minute scrape might just hit that dumb threshold. It's not always about flooding or going too fast sometimes the site is just broken. Not defending bad scraping but acting like every ban is justified is just ignoring that some devs really don't know what they're doing.

claire_gibson22d ago

Honestly, I think we're all overlooking the fact that some sites actually want you to scrape them but just have really poorly documented APIs. I had a similar situation where I got blocked from a local events site, but turns out they had an official JSON feed for developers they just buried in their footer. If you're getting blocked, do a quick search for "API" or "developer" on the site before assuming you're in the wrong. Sometimes you can get everything you need legally with a simple URL change.