Blog

Lots of confusion about the proxies

  • Written by Sum Random Guy on Sunday 28 September 2014 in the category Proxies.

First off, 95% of the folks will never even encounter the throttling.  Only those who are requesting a fricking ton of proxies will notice.

What happens when it throttles?

Well, the Director asks for a proxy and the website determines you've requested a whole lot within a very short time so it delays for 5-15 seconds before replying.

What's a "whole lot" and "what's a short time?"

A whole lot is about 200 proxies.  A short time is 5-10 minutes.

OK, who would possibly request that many proxies within that short of time?

Well, certainly nobody legitimate.  I have a very large list of very well-tested proxies.  Lots of people sell those lists.  They try to steal the proxies that I've collected to add to their own lists.  Competitors.

Competitors?

Yep.  There are people competing with a free service (facepalm).

Do you care about that?

Nope.  Not in the slightest.  What I do care about, though, is that when these leachers do come and steal my proxies, they put undue strain on the server which impacts the service I'm providing (for free, did I mention that?).

So why do you throttle again?

Well, if I didn't, the leachers would hammer my server into the ground and your service would suck.

How many people have been throttled?

I'm going to take that question at face value.  In the beginning as I was tweaking the settings, about 5% of the folks were throttled.  They're the massive players.  Some have literally hundreds of accounts.  Some folks have hundreds of accounts but they play on multiple servers.

I have tweaked the throttling so that no legitimate player gets throttled.  The only throttling is when the leachers get out of hand.  The folks stealing my proxies for their own use and not legitimately using the Director.

OK, so don't take the question literally and let's change it to "how many leachers are there?"

There is a surprising number of them.  Some are just proxy scrapers who are building lists.  Some are actually building competing proxy lists for use with Neatbot.  Take that info as you will.

Where do you get your lists?

Now we get there.  I have spent a lot of time writing some programs that help me out.  There are about 25 or so major sources of proxy lists.  15 of them are "free" lists.  Places like nntime.com for example offer free lists that are very well regarded and work quite well.  I have some programs that I wrote to gather those lists and import them into my database.  I've worked with all of these sources so they know I'm doing it.  Many are fine as long as it's a manual process but manual processes require a person to do them.  To do something automated requires you to contact them and arrange details.  I did that.

I also have subscribed to several paid lists.  Yes, I pay for proxies and give them out for free.  I have subscribed to about a dozen lists.  There is some crossover between the lists but for the most part, those dozen lists give me a huge amount of fresh proxies daily.

How often do you update them?

I import new proxies into the list 3-4 times per day.  All automated.  I go on vacation and it runs itself.

How do you test them?

Secret processes, mostly.  Every 15 minutes, I spin up 250-500 simultaneous test threads that test each proxy.  Each one is tested every 15 minutes.  Again, this is all code that I wrote specifically for testing proxies to see if they work with Evony.  I test running Flash.  I test to see if the IP address actually changed.  I test the level of anonymity.  I test the speed.  I test the number of simultaneous connections.

How many proxies do you have?

Well, with the number of sources I get them from, many, many, many.  I have lots.  Thousands of proxies are active and live and working.  I import 15,000 proxies per day.  Many don't work.  They have to be alive for several hours before I even add them to my list.  75% of my proxies have been alive for more than 3 days.  50% of my proxies have been alive for more than a week.  I have a couple hundred that have been alive for several weeks.  I have good proxies.  I just spent some time recently automating some additional sources.  

OK, so when you hand the out, what happens?

More secret processes but here's some of it.  The proxies are given scores.  The score is made up of the proxy's speed, how long it's been alive (longer-lived proxies are very valuable), how many simultaneous connections it can support, the level of anonymity, the geographical location, the number of proxies within the same IP address range, etc.

I have a "reputation" given to each user of the proxy service.  Your unique key defines you.  If you're trusted, your reputation goes up.  The higher your reputation, the better the chance that you'll get one of the higher scored proxies.

Oh!  I want to increase my reputation!  How do I do it?

Send in your Unique Key so I can identify you is the biggest part.  Anonymous people have near zero reputation (zero is reserved for special people).  If you send your unique key, I have a Skype name or a Forum name to reach out to if something happens.  I can trust someone I can identify.

The second method is to help weed out bad proxies.  There's an option on the Tools menu to send me info about bad proxies.  Yep.  I test them myself every 15 minutes, however, I can still use human feedback.  Some proxies may make it through my testing.  You may catch a bad proxy before I do.  I want to get the bad ones out.  If you send me info about a bad proxy, I'll test it.  If it's really bad, I'll increase your reputation.  If it's good, I'll assume it was a mistake and leave your reputation alone unless you submit mostly incorrect information.  If you send me 10 proxies and 7 are really good, your reputation will go down slightly.

You can do other things such as send me proxies.  Email them to me or send them to me via Skype.  I am always looking to feed the lists.  You might have a list that I do not.  

If you're a good donator to the forum or a frequent participant in the Skype rooms or Forums, I'll bump up your reputation.

More reputation means better chances of being hooked up with the higher scoring proxies.

But I want to stay anonymous.

That wasn't a question.

What if I want to stay anonymous?

That's fine.  You'll still be given proxies as quickly as you need them.  You'll still get the good proxies.  100% of my proxies were good at the last time I tested them which was within 15 minutes of handing them to you.

Gaining reputation increases the odds that you'll get the fastest, most anonymous, most long-lived proxies.  If you stay with the anonymous reputation, you'll still get those best proxies but not as often.

My server limits to 3, how do you keep track of that?

Well.  Obviously, sending more than 3 people through a proxy would trip that limit.  That'd be unfortunate.

I do make sure I don't send the same proxy too often.  With as many proxies as I have in the list (which is a LOT), the chances are small that I'll send the same proxy to 3 simultaneous users.  I do keep track of the servers, though, so I don't send the same proxy more than twice for the same server within 3 hours and not more than 3 times within a day.  Even if I did, the bot would notice that it's not connected, it'd exit and relaunch with a new proxy.  No harm done.

Free or paid?

Ouch.  If you watch your account every day and you have money to spare, go with a paid set of proxies.  If you have 30 accounts and are on a server with 3 connections max, you'll need a more expensive 10 IP account.

If you don't watch frequently, there is a host of things that can bite you in the butt.  Your home IP address changes, you forget to pay for the proxies, they change the login details, Evony blocks your proxy provider, Evony reduces the limit from 15 to 3 connections, etc.  All of those can stop your proxies from working.  If you only check those alts every couple days, they can be out of commission for a while.  

If you watch the accounts frequently, you are willing to spend some cash and you're willing to troubleshoot if the conditions change, the paid private proxies are your best bet.

If you check every couple days, the free ones are your best bet.  If a proxy dies, you'll restart with a new one.  Big whoop.

What are you running on?

Well, it actually takes a lot to run that amount of processing.  The servers are big.  Very big.  

New proxy stuff in the Director - Important stuff here

The Director has a new feature.  I got tired of troubleshooting people's proxy problems.  The proxy list was very, very easy to use but with the thousands of options, people got confused.  Instead of using the free proxies that I recommended, people were buying proxies and then trying to use them like the free ones.  We'd have to untangle that mess.

I had a TON of proxies and I test them every 15 minutes.  I put an option in the Director on the Tools menu to use a proxy from my server.  I feed this list new proxies every day and test the list every 15 minutes.  It's a very large, very clean list of proxies.  There are some that hate routing game traffic but that pass my tests.  It happens.  I'll sometimes hand those out but they don't come about often.  More about bad proxies later.

So, this is completely free for now.  It is completely and totally anonymous.  I don't know who is using it and don't know anything about the accounts they are using.  

Being free, I have to build some protections in.  If you request a bunch of proxies, I start slowing down the replies.  The first couple go blazingly fast.  Then after you get a bunch of them, it starts slowing down.  Each request takes slightly longer than the previous request.

Unfortunately, the Director can't do anything else while it's requesting a proxy.  It can't redraw the Director's window while it's talking to the web server.  That makes it look like the Director is locked up.  It's not.  It's working fine, it's just waiting for the proxy information.  If you're slowed down to 10 seconds to reply, the Director will look like it's overloaded and broken for the 10 seconds.  It's fine.  Don't panic.  Don't worry.  It'll be back.  If it bothers you, check the option down below which will make that not happen.  Or just ignore it and let it do its thing and it'll clear up on its own.  After a short time, it'll clear up and will not throttle you again so it's only when you request a bunch of proxies quickly.

Now, if you want to make things better for you, and for me, you can become less anonymous and identify yourself to me so I know that you're not an evil proxy leech.  In the Director, go to View then Edit Director Settings.  Copy the UniqueKey value and Skype it to me or (much better) send it to me via the Neatportal PM.  Once you tell me who you are, I'll happily hand out IP addresses without throttling like that.  I will NOT know anything about your account information or anything.  I just know your unique key and which server you're connecting to (to make sure I don't hand out the same proxy to 100 people on server ABC).  

Bad proxies will happen.  On the Tools menu is an option to tell me about it.  Copy the IP address and port provided and go to Tools | Tell SRG about a bad proxy.  Paste that IP address into the box and submit it.  I'll remove it from the pool so other users can benefit and we can keep the list clean.

I have hundreds of users using this feature already and I haven't really advertised it.  Performance has been great and I've been tweaking the settings such as the throttling to make sure it works well.  I value feedback.  Skype me or PM me on the Neatportal forums with feedback.

Here's what Inanna wrote up:

1. Go to the Director -> Help menu -> Check for Director Upgrade. Let it upgrade, if it's not already done so

2. Type -logintimeout 180 at the bottom in the Custom Params box  (http://prntscr.com/3v6dma)

3. For any profile you want on your REAL ip, select it, and click Account Details. Once in there check the "No proxy" box at the bottom (http://prntscr.com/3v6fu7)

4. Go to the Director -> Tools menu -> Use a Proxy from SumRandomGuy, click it.

5. Close all your bots, turn off your IP hider, let the Director restart them using a random proxy from SRG.

6. If a proxy dies while it's connected or fails to connect at all, the bot will try to reconnect for 180 seconds with that proxy. If it cannot reconnect it will close the bot (that's what your logintimeout was for). Once the bot is closed, the Director will restart the bot using a new random proxy. Make sure all bots are set to "Maintain ON"

7. Tell SRG he is your absolute hero.

 

And if you're looking for a VPS (virtual private server) to run your bots, check this out:

http://sumrandomguy.com/blog/detail/i-finally-found-a-vps-virtual-private-server-that-works-for-me

Security:  These are free proxies that were found on the Internet.  They may be malicious.  See my other articles on the risk but if I have a TON of proxies (and I do) and one or two of them are malicious, your chances of getting them are slim.  The chances of the malicious proxy being there to gather Evony logins are vanishingly small.  The chances that I'd even FIND an Evony-login-stealing proxy are infinitesimal.  The chances of you getting a proxy that is there to steal passwords is really small.  The chances that it's on a server they care about is tiny.  The chances that it's on a server they care about and it's an account they care about is miniscule.  I'm out of small adjectives so I'll let you think about that and read my other articles.

Easy-Hide-IP - Anonymous surfing, encrypted connections, over 700 ips in 13 countriesspace