Checking the Network DNS traffic for the last 24 hour I noticed that DNS amplification attacks are becoming very common. DNS is just one of the protocols that attackers use to send amplification attacks to target IPs. Additional to DNS, there are additional protocols (SSDP, NTP, SNMP,..) that are vulnerable to reflection/amplification attacks. An attacker can easily use the address spoofing method to remotely send an attack using a botnet network. Consequences of a DDOS attack against an enterprise can be dire, from financial costs to a negative impact on a brand’s reputation.
Facts about DDOS
- Most of the attacks are ~1Gbps of traffic, but as we can see in the traffic graph there is an attack that is ~17Gbps. Depending on you Internet connectivity, an attack of >1Gbps is enough to cause service disruption.
- DDOS attacks are no longer a matter of it companies will get hit, it’s now a question of how often and how long.
- Attacks are becoming bigger. Check what happened to Github back on Feb 28th
- An attacker can anonymously pay for for a DDOSER service for $24 a month or less. The more money, the more larger and destructive botnet network. Some options iddos.net, str3ssed.me, critical-boot.com.
- Some DDOS attacks (Not volumetric) can be identified and mitigated with on premise UTMs/WAFs. But if the attack is larger that your Internet bandwidth, any security device is useless.
Notes and protection options for enterprise customers
- On premise firewalls, IPS, IDS, WAF, UTM are useless for volumetric attacks.
- Firewalls and IPs are stateful devices, which often means they become target of DDoS attacks
- If possible avoid Email/DNS servers on premise. There are multiple cloud providers that you can use for this purpose.
- If you host web applications, consider using Cloud providers and CDN services. Multiple options available depending on the type of content, bandwidth, geography, etc:
- Make sure that your service provider can help with volumetric attacks and that you have complete control to trigger the protection. Some service providers provide an on demand service where protection is triggered as soon as an attack is detected or you can manually trigger it using a web portal or a BGP community.
- Use an on demand protection service of your service provider. An always on solution in your service provider cloud, usually degrades latency.
- Deploy internal traffic analysis tools using netflow/sflow/ipfix. The protocol will depend on the type of gear that you use in your network. Multiple open source and commercial options are available. Some options:
- Avoid GRE based DDOS solutions. It has several disadvantages like:
- Tunneling usually degrades performance.
- The DDOS solution provider advertise the subnet via BGP using their own ASN.
- If a single host in your network is under attack, you have to protect a whole /24.
- When your traffic is being protected, the latency of your apps will increase.
- Triggering the protection usually takes time > 20-30 minutes and you will depend completely on the service provider.
- Visibility and analytics tools are important. Understanding where traffic is going and coming can help you to proactively detect and mitigate a DDOS attack.
Factors to consider for Service providers
- Make sure that you have a blackhole BGP community deployed and that you test the community with every single transit and peering provider.
- Real time analytics of your traffic is important. Even if you have a large sampling in the flow export, you will be able to detect volumetric attacks.
- Deploy automatic protection methods. If an attack is detected have automatic ways to send the attacked IP to a scrubbing center or send it to blackhole
- A DDOS attack can cause service disruption in a complete metro market.
- Have a plan for scenarios where the attacked IP is an IP of one of your core routers.
- Deploy software based/SDN traffic injection and rerouting. Avoid GRE based solutions if possible.
- Consider using BGP Flowspec. You will have more mitigation options
Whether you face DDOS attacks of 1Gbps+, 10Gbps+ or 100Gbps+ work on a plan. It’s a matter of time.
Having CLI access to remote servers from external operators to run commands like ping, traceroute, mtr, etc., makes the troubleshooting proccess easier. A point of view outside your network is essential when trying identify latency, packet loss or connectivity issues between your IPs and the Internet. Thanks Job Snijders and the NLNOG community for making the ring possible.
What is NLNOG Ring?
NLNOG Ring is a community of network operators that share a virtual linux machine within their network. All members have access to all the servers that are part of the project and you can run commands from multiple servers at the same time. Some examples of what you can do in the ring:
- Run a traceroute from 10 remote servers to a specific IP and paste the output to pastebin in jpg format
- Ping an IP from N remote servers and see the latency per server
- Run DNS queries from remote servers
- Send http requests from remote servers to a specific IP
- Validate AS-PATH and BGP routing information from web based looking glass
NLNOG Ring in numbers
- 416 organizations
- 496 servers
- 426 ASNs
- Members from 56 countries
The requirements to participate are:
- Your organisation has its own ASN, IPv4 and IPv6 prefix(es).
- You are a network operator
- The organisation you work for has BGP routers connected to the ”Default Free Zone” and maybe even IXP’s.
- You have enable or configure rights on those routers.
- You are involved in the networkers community.
- You have permission from your organisation to become involved in the NLNOG RING.
1.- Ping Google DNS server from 10 random servers
Command: ring-ping -n 10 -d -i -t 184.108.40.206
2.- Run trace route to Google DNS from 15 remote random servers and display the trace information in JPG
Command: ring-trace 220.127.116.11 -n 15 -vv -b
Real time and historical latency analysis in a Telco environment is an important troubleshooting tool that can help the NOC and Ops teams to identity networking issues. Issues like fiber cuts, errors in backbone links, wrong configuration in routing protocols and link saturations can affect customers applications running over those links. Being able to proactively identify issues before customers applications are affected due to latency or packet loss is one of the most important activities of NOC teams.
Typical monitoring tools (Opensource based open Nagios or commercial ones like Solarwinds, Zabbix, etc) are centralized and usually only provide ICMP analysis from one or two geographic locations which is not enough when trying to identify latency between the different markets or POPs.
There are several commercial and open source solutions that allow you to deploy internal probes in such a way that you can send ICMP tests every 60 seconds and send alarms via email or web hooks to a chat application like slack or Hangouts chats. An interesting one is Worldping which is a cloud based solution that can be integrated as plugin with Grafana and have the following advantages:
- Cloud based
- You don’t have to deploy an internal VM or public cloud VM and install Linux and Grafana
- Grafana upgrades
- Hosted Grafana supports community plugins which means that can be used not only for Worldping
- Additional to ICMP tests the probe supports DNS, HTTP and HTTPS
- You can use external probes additional to the private ones
Steps to deploy a hosted Grafana with Worldping plugin:
- Create an account in grafana.net
- Install and enable Worldping plugin the the hosted Grafana
- Install raintank in the internal probes. You can use any Linux server, either a vm or baremetal server or use a Raspberry Pi with Raspbian. Make sure that Go is installed
- Create API Key
- Configure key in internal probes under probe.ini and start daemon
- Configure Internal probes in Worldping (Should appear automatically if the probe list is key was configured correctly)
- Configure Endpoints in Worldping
- Configure alarms
- Enjoy and share dashboards with your team
Other commercial and open source alternatives:
Vaping (Interesting alternative to Smokeping based on Python)