DDoS Attacks - What do you do?

[loopyman]

We have noticed an increase in DDoS attacks targeting our GarrysMod clients. So far we have just been terminating the customers and null routing the IP's when it occurs. Most of the attacks total around 2Gbps which murders our gigabit ports.

 

What do you do when a customer brings in a DDoS attack?

[jbiloh]

What supplier are you using for your dedicated/colo?

 

In most cases when our GSP customers at ColoCrossing get attacked we implement an ACL at the router level to block the source traffic from hitting the customer port. This is a pretty unique service -- most datacenters don't offer this.

[adamnp]

Null route is usually first...Can't disrupt other clients due to one clients issues, especially if they specifically are being targetted and not your 'host' in general. If you are able to, place the affected ip's behind a cisco guard or other other nice piece of hardware that can filter the attack.

 

Advanced users can custom add thier own ACL rules like Jon has stated. Most importantly use common sense - Clients that 'bring' issues shouldn't be kept onboard obviously....Get rid of them.

 

What kind of attack is it? There is much more you can do given more knowledge of the type of attack...Sometimes they are simply exploiting a simple hole on a mass scale-Patch the hole and boom its gone.

[ViolentCrimes]

Actually has less to do with the customer and more with a kid that is pissed off at gmod.

 

http://www.facepunch.com/threads/1062747-DDos-attacks-on-GMod-servers-on-the-rise

 

They are targeting all gmod servers at random. Blame COD4 developers for not fixing the exploit. More info on the exploit http://www.nfoservers.com/forums/viewtopic.php?f=25&t=4960

 

 

Also Jon you never emailed me back when I emailed you a month ago.

[jbiloh]

I usually respond to every email that comes my way, if you like, please re-send your email and I'll take a look at it right away: jbiloh-AT-colocrossing.com

[trancemode]

yep garry's mod the problem. Especially any RP games like The specialist, Garry's Mod, etc.

 

I've had DDoS just for hosting the game due to "angry kids". When a client either kicks or ban them from the server that is when the DoS/DDOS comes.

[adamnp]

There is a fix to this already published in the article you posted. -- utilize reverse path filtering. Any decent sized provider should be utilizing this to begin with to deter a majority of other DoS attacks that stem from the same issue.

 

All you need is a Cisco router that supports CiscoExpressForwarding, or any other competing product that supports RPF. Juniper products also support it, and you can also do a low level kernel mod for *ix to also assist you if you don't have the budget for hardware interception. You can turn this feature on with the rp_filter kernel option found in /proc/sys/net/ipv4/conf/*/rp_filter. If you change this option to 1 it will enable reverse path filtering on the specified interface, while setting it to 0 disables it.

conf/all/rp_filter must be set to 1 for filtering to work on any interface.

 

Of course, this will mainly stop your servers from being able to be used to launch the attacks - and again, if you have customers that are helpinh to 'drive' the attacks your way, some interjection should be taking place to minimize the effects.

[loopyman]

Thanks for the input. We have been using Wholesale Internet out of KC, they just null route, but we just recently have been removed from Netelligent out of Montreal due to attacks, and Continuum in Chicago hit us with administration fees for the null routes.

 

Wholesale doesn't allow us to implement any ACL's on their network devices. I'm considering seeing if they would allow me to colo a firewall next to their edge with a 10Gbps connection.

 

Also the DRDoS is exactly what is happening to us. We started to contact the biggest sender's abuse emails and realized that every one of them was a game server provider and was a Quake 3 based game. We have been hit with attacks totaling over 3Gbps using this method. Almost all of them are targeting "Roleplay" Garrysmod servers.

 

Thanks for the assistance.

 

If you would share, Which inbound ports would you recommend we deny?

[jbiloh]

I hate to blatantly do this, but consider contacting our sales team (sales@colocrossing.com). If you're looking for a protected environment, we can provide it.

 

Or, you can ask Continuum to offer you ACL-level service on their core equipment. They might complain because it takes time from experts/engineers to administer, but it's the only true way to protect users like you.

 

PS: Cisco devices, like the ASA series (which is probably what you're referencing) provide zero DDOS protection.

[loopyman]

I'm not interested in DDoS protection, Just want to solve the damn DRDoS attacks without much additional cost.

[KingJ]

If the attack volume is low enough we generally block it at the server-level. More than that though and we either request a nullroute to preserve our bandwidth allowance or if the attack continues we have little choice but to stop hosting the targeted server. I wish I didn't have to stop hosting the target, but if it's disrupting other servers there is little I can do.

 

Recently, i've noticed a lot of 1 minute shot DDoS attacks, enough to timeout everyone on the server, but not long enough for us to analyse and block.

[Monk]

I'm not interested in DDoS protection, Just want to solve the damn DRDoS attacks without much additional cost.

 

You can't, unless you run unicast reverse path filter on your cisco stuff, but that doesn't work under certain conditions, and adds more overhead to the FIB/RIB of switches..

 

One thing you could try is discard traffic that doesn't have an ACK (tcp only) or an established connection. You could technically setup something to permit traffic like this

 

ipfw add permit udp data eq "\0FF\0FF\029\029\030" range 20000-30000 eq log

ipfw add deny

 

It's pseudo code, it permits /established/ connections that contain an INIT string, normally sent from the server during a handshake (q3 protocol), and then permits anything else afterward. This will stop fraggle attacks (which is what it is, with UDP pingpong, cuz the engine allows queries to originate anywhere without any kind of protection)

 

53 bytes per query, 10,000 servers is a 4mbit attack, of course, you can multiple the queries to give out ERROR data which is like 512 bytes, smaller than an MTU but can increase traffic by a factor of 10..

[Bubka3]

What supplier are you using for your dedicated/colo?

 

In most cases when our GSP customers at ColoCrossing get attacked we implement an ACL at the router level to block the source traffic from hitting the customer port. This is a pretty unique service -- most datacenters don't offer this.

 

I am sorry, and I don't mean to bash, but this is a lie.

 

When we had a server with you guys through a reseller, our entire box got nulled. All the IPs. Then "Alex" went on to ignore our reseller's tickets after the 24 hour null-route was over for a good 4-6 hours.

[jbiloh]

I am sorry, and I don't mean to bash, but this is a lie.

 

When we had a server with you guys through a reseller, our entire box got nulled. All the IPs. Then "Alex" went on to ignore our reseller's tickets after the 24 hour null-route was over for a good 4-6 hours.

 

Certainly not a lie.

 

I can't speak to your specific experience though, seeing as you were not a customer of ColoCrossing.

 

It's also worth noting that in some instances, depending on the size of the attack a null route is the only realistic option.

[Bubka3]

Certainly not a lie.

 

I can't speak to your specific experience though, seeing as you were not a customer of ColoCrossing.

 

It's also worth noting that in some instances, depending on the size of the attack a null route is the only realistic option.

 

1. It is a lie, comparing you did not filter any traffic.

2. So this means you don't treat all customers the same, as my reseller was a direct customer?

3. The attack was 1 IP. Alex confirmed it and even pulled logs out of the router, yet he null-routes the entire machine. I wouldn't mind a 1 IP null-route. I do mind when 16 IPs are just taken offline for a day.

 

Like I said, I am not here to bash, and would use your services again, if Alex didn't do this to me. At this point I am just looking for some answers.

[jbiloh]

1. It is a lie, comparing you did not filter any traffic.

2. So this means you don't treat all customers the same, as my reseller was a direct customer?

3. The attack was 1 IP. Alex confirmed it and even pulled logs out of the router, yet he null-routes the entire machine. I wouldn't mind a 1 IP null-route. I do mind when 16 IPs are just taken offline for a day.

 

Like I said, I am not here to bash, and would use your services again, if Alex didn't do this to me. At this point I am just looking for some answers.

 

Since you were not a customer of ColoCrossing I cannot speak to the specifics of your experience. All of the benefits offered by ColoCrossing are best enjoyed by direct customers.

 

In your case I suspect your reseller was actually a colocation customer of ColoCrossing. With that many layers of abstraction it is totally unreasonable to throw phrases like "lie" out there. Your supplier may have had different policies in place to deal with attacks, etc. The truth is, ColoCrossing takes a best effort approach in defending our customers against attacks -- and normally that means excellent results.

 

PS: If IP Engineering disabled an entire range, and not just the targeted IP I suspect something more was going on... like the range was attracting multiple, large attacks, or the attack was cycling through the allocation. Just food for thought...

[Bubka3]

Since you were not a customer of ColoCrossing I cannot speak to the specifics of your experience. All of the benefits offered by ColoCrossing are best enjoyed by direct customers.

 

In your case I suspect your reseller was actually a colocation customer of ColoCrossing. With that many layers of abstraction it is totally unreasonable to throw phrases like "lie" out there. Your supplier may have had different policies in place to deal with attacks, etc. The truth is, ColoCrossing takes a best effort approach in defending our customers against attacks -- and normally that means excellent results.

 

PS: If IP Engineering disabled an entire range, and not just the targeted IP I suspect something more was going on... like the range was attracting multiple, large attacks, or the attack was cycling through the allocation. Just food for thought...

 

I was told that Alex specifically said what IP it was, and null-routed the whole machine. After reseller tried to get it taken down to 1 IP, Alex didn't answer until 4-6 hours after the initial 24 hour null-route period.

 

My second question is why are you so attached to the direct customer theory? Yes, the reseller was a colocation customer, does that mean he will always be null-routed box wise instead of your dedicated server clients which get 1 IP null-route? I know said reseller pretty well, and when this happened he even offered me a credit, so I doubt any policy of his was the doing. Furthermore, I even remember sending a packet capture file to further prove that it still is one IP. Unfortunately, Alex didn't know (or want to accept) a packet capture file.

[bullfrog3459]

I was told that Alex specifically said what IP it was, and null-routed the whole machine. After reseller tried to get it taken down to 1 IP, Alex didn't answer until 4-6 hours after the initial 24 hour null-route period.

 

My second question is why are you so attached to the direct customer theory? Yes, the reseller was a colocation customer, does that mean he will always be null-routed box wise instead of your dedicated server clients which get 1 IP null-route? I know said reseller pretty well, and when this happened he even offered me a credit, so I doubt any policy of his was the doing. Furthermore, I even remember sending a packet capture file to further prove that it still is one IP. Unfortunately, Alex didn't know (or want to accept) a packet capture file.

 

Hi,

 

When we null our machines or even colocated customer machines, is because of a risk to other customers and the network. We have our policy that requires us to null a server and it MUST stay null-routed until a senior network engineer such as myself or Alex can approve the removal as such. We do have other responsibilities that do tend to keep us from having 5 minute ticket times.

 

Next, as such you were not a client directly of ours and through one of our customers. Policies may or may not differ depending on the agreements they have with us. If we as senior network engineers make a decision that the attack whether originating on/to/from the server(s) poses a threat to the network, in any manner, we have the right to execute any means we need to keep the network stable and reliable for our other customers; thus, Alex executed his right because he found it to be a significant threat. At that point, even if you provide us logs or wireshark packet captures or whatever else, it means little because our network comes first and foremost for all of our customers. This has and will always be the case.

 

So, what we need to get past is, what has happened has happened, and nothing we can do to change it. We implement ACL's when we can or act upon what we have to do to protect the network at all costs. Any senior network administrators/engineer will tell you this and stand by their network.

 

Thanks

[CloudCUBE]

Null Route is the best option.

 

Waiting for Network upgrade - splitting pipe to cut off international (Australian Host) most infected computers are overseas, Australian ISP's actually pick up on junk traffic from home connections and email them about it.

[adamnp]

I seem to find it unplausible that ColoCrossing, or any host for that matter would null for no reason -- Think if it from a different point of view; if it isn't affecting their network or their other users, it is simply going to run your bandwidth allotment up and possibly bring additional charges from your account that most likely wont cost them a penny extra, unless they run their commit's that tight. If you are causing issues to other users, you are obviously going to be nulled. Typically this happens in a do first ask questions later fashion to protect the interests of the network and other clients.

 

There had to be an issue that caused the null - though I can 100% agree that many hosts "forget" they placed the null and don't remove it after the time frame suggested, which would burn a hole right through my nice chair here as well. I can't stand stuff like that.

 

I also typically now notice some hosts suggesting '24' hours as a 'normal' wait period. In my opinion, There is no reason why a 4 hour window cannot be executed and an evaluation done at that point to cause as little disruption to everyone as possible, none the less - it seems some are swapping to this '24' hour method for some reason I have yet to explain.

 

Communication is key, and the fundamental element to any null route.

[Monk]

DDoS mitigation costs money. Null routing stops it from coming down your pipes. DDoS mitigation only works if you have piles of cash, multiple peering points and equipment to load-shed the incoming attack. Null routing is easy peasy:

 

access-list 101 permit ip target.machine 255.255.255.255 any

 

route-map nullroutez0r

match ip address 101

set community 123:666

!

[CookieCrusher]

Well we @GameServersOnline use a dedicated machine in our datacenter inorder to block DDOS attacks using APF firewall's DDOS protection .

So far it has blocked about 5000 DDOS attacks ...

[jbiloh]

Well we @GameServersOnline use a dedicated machine in our datacenter inorder to block DDOS attacks using APF firewall's DDOS protection .

So far it has blocked about 5000 DDOS attacks ...

 

Sounds like you deal with very small attacks.

[adamnp]

Sounds like you deal with very small attacks.

 

lol -- I think you're right

[Shepsie]

What do you do if there are multiple customers on the same IP, for example customer on port 27015 is under attack but customer 27016 is'nt?