Hacker Newsnew | past | comments | ask | show | jobs | submitlogin

Amazon is now the #1 choice for web scrapers everywhere :)


I thought most AWS IP addresses were blocked by most admins. Does any one have experience scraping using Amazon.


Most admins? Most admins probably don't even know what AWS is. Most admins probably don't even block a single IP. I would be surprised if as many as 0.1% of admins block AWS.


I don't scrape, but I use an EC2 micro instance as a VPN endpoint when I am on an unsecured wifi network.

It's a little annoying that some sites block me solely because I'm coming in from AWS (StackOverflow, I'm looking at you).


I suggest using prgmr.com instead. $6/mo is worth a Xen instance with IPv6 and everything.


We have a pretty big e-commerce scraping project, and we don't really run into many problems regarding blacklisted ip's. There are a few sites who we get consistent bans from, but with elastic ip's it's pretty much a non-issue. I have yet to see a site that ban's all amazon ip's.


I run http://whirlpool.net.au and I religiously check the Amazon EC2 forum announcements[1] for new IP ranges to ban.

[1] https://forums.aws.amazon.com/ann.jspa?annID=1030


> I run http://whirlpool.net.au

An excellent and very useful forum! It seems like whatever topic I'm searching for, google(.com.au) returns a useful result on your site.


Would you mind telling why?


Why?

Name me one good reason. Name me one.


Shitloads of rogue bots doing "social media monitoring".

Shitloads of rogue bots stealing content for black-hat SEO.

Shitloads of rogue bots harvesting email addresses.

Shitloads of rogue bots submitting spammy replies.


So maintain a blacklist of elastic IPs. If it's too big for you, make it a community effort.

Those are bad reasons to close your site to all of AWS.

As nupark2 mentioned, there are legitimate users routing traffic through EC2, even some bots that you'd want to visit your site. Archive.org comes to mind (many of there scrapers are or were behind AWS). Closing your site or app to a large swath of the web is the wrong solution. It's like killing a spider with a bazooka.


Unlike the assumptions you're limited to making, I know how much of my AWS traffic is human, and it's really very very very small. The sad reality is I'm sick and tired of rogue bots, and the tiny sliver of collateral damage can fill out the CAPTCHA validation every so often.

(I also blacklist GWS, rackspace, linode, softlayer, reliablehosting, ovh.net, node4, netdirect, layer42, all TOR exits... it's actually a pretty huge list.)


I whitelist archive.org, and they've never hit through AWS.


That's unfortunate. I don't know what the answer is, but real people do route their traffic out of AWS endpoints.


Far fewer than you might think.


It is annoying when it happens and you're on something like Heroku.


In my experience no UK newspapers block AWS IP addresses.


I usually use Tor when I scrape anyway :)


Free (or cheap) inbound bandwidth is pretty common from colo providers.




Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: