Limited Time Offer!

For Less Than the Cost of a Starbucks Coffee, Access All DevOpsSchool Videos on YouTube Unlimitedly.
Master DevOps, SRE, DevSecOps Skills!

Enroll Now
AWS

DDOS attack and HTTP flood attack on AWS from AWS EC2 instance using New Relic Synthetics & infalted aws bill to $3000

February 15, 2021 comments off

How I prevented DDOS attack and HTTP flood attack from New Relic Synthetics Monitoring of AWS EC2 Instance?

5 Dec 2020 – DDOS attack and HTTP flood attack Start Date from AWS Ec2 Instance

10 Jan 2021 – I found out this attack when the bill has come for more than $3000.

Phase 1 – Duration 10 Days – Contacting AWS Support team – No Help from them

I noticed that my AWS Data transfer bill has come to almost $3000 which is 100 times more than regular. I was shocked and for the moment, I assumed that it was an AWS issue and I reported it to them. After 5-6 days of struggling with the AWS support team who were not able to help me, they asked me to reach out to the AWS ec2 abuse team. Meanwhile everyday, the Data transfer cost was coming almost $100/day. I was helpless. I started regretting my decision to migrate from Godaddy VPC to an AWS reserved instance. So many hidden charges which you would come to know only once you start using it even if you make a payment in advance for an AWS ec2 reserved instance. Definitely, I felt helpless that AWS is not good for small business with no proper IT team in place.

Phase 2 – Duration 15 Days – Contacting EC2 Abuse Team – Not much help from them

However, the AWS ec2 abuse team was taking a huge amount of waiting time and no useful response from the last 10 days, thus i decided to investigate myself only.

I was not skilled enough but I stressed myself beyond my work time and got into an apache log and realized that its nothing but DDOS/HTTP flood attack. Meanwhile, I installed many tools to block the request but found none helpful. It’s very difficult to trace HTTP flood attacks. Finally i decided to block all the IP addresses which are making requests to my server more than 1000 in a day.

Some of the commands which helped me as below;

Later, when I could control the DDOS attack, I tracked the IP Address using https://www.opentracker.net/ and found that those IPs address belong to no one but Amazon itself. I was shocked. For some time, I started assuming that AWS itself is involved in this attack. Finally, I submitted my analysis report to the Ec2 abuse team. Again Abuse team took 3-4 days to locate the owner of IP addresses and it was owned by NewRelic and instead of blocking those IPaddress, they asked me to reach the NewRelic support team directly. For some time, I felt strange but I had to do it. For all these processes of the AWS Support Team, EC2 abuse team’s decision to reach out to the NewRelic support team was 15 days. Let me also, one interesting observation is – None of the team was able to help me anything rather just sending email responses and hurry to close a ticket. Here I was the myself who studied all the findings and shared with the abuse team and they asked me to reach out to the NewRelic Support team.

Phase 3 – Duration – 20 Days – Contacting NewRelic Support team – – Not much help from them

However, I reached out to the NewRelic Support and NewRelic Abuse team with all the details of the IP address and their response was more shocking. Instead of finding a culprit accounts and blocking them, which you can find below;

Thanks for reaching out - sorry for your stressful situation. Those IP addresses do look to be related to New Relic Synthetics monitors.

You can identify the account is which the monitors live by using the X-Abuse-Info.

Here are instructions for locating the account:

https://docs.newrelic.com/docs/synthetics/synthetic-monitoring/administration/identify-synthetic-monitoring-requests-your-app

Please let me know the account IDs that you find so that I can investigate this further.

I felt strange that NewRelic does not maintain the customer details of which website is monitored by which customer id. They were having domain names on which synthetic-monitoring is enabled but they were not able to trace the account. I was shocked. They sent me this email in response.

We are unable to assist you with finding the headers or the request logs on your network. Perhaps there is a network engineer within your organization who can assist with this.

The following header is sent along with every request that our Synthetics monitors make

X-Abuse-Info: Request sent by a New Relic Synthetics Monitor (https://docs.newrelic.com/docs/synthetics/new-relic-synthetics/administration/identify-synthetics-requests-your-app) - monitor id: ${MONITOR_ID_STRING} | account id: ${ACCOUNT_NUMBER}
Without the above information there is unfortunately no way we can assist any further other than recommending blacklisting all of our associated IP addresses which will block these requests.

If you have any questions about this please let me know.

This blog is still under in-progress

How to save HTTP requests headers, methods and body to a file?

Final updates from NewRelic

Phase 4 – Dealing with Arrogant AWS customer support team

Initially AWS customer support team had promised that, upon recieveing the closure from NewRelic Team and AWS Abuse Team, They would adjust amount with unnecessary charges of Data transfer which has occured coz of this DDOS attack from AWS resources. From last 10 Days, I am trying to reach out to AWS customer support team and AWS Support team declined to accept my call and simply they close the ticket without any response.

I am very much shocked that any ticket which i open questioning about updates, they simply close without even one line of response. Thus, I am stuck in same loop; I have sent an email again to following and waiting for response

ec2-abuse@amazon.com ,
abuse@amazonaws.com ,
aws-in-receivables-support
Twitter – @awscloud @AWSSupport

Log in AWS Public Forum is at – https://forums.aws.amazon.com/thread.jspa?messageID=974670&#974670

Rajesh Kumar
Follow me
Latest posts by Rajesh Kumar (see all)