AWS And Security - The Lost Keys

Submitted by Peter on Sun, 25/06/2017 - 23:55

Today I was working on my submission for an Amazon contest over at Hackster.io (for those interested here is my submission). 1 of the requirements of the contest is that all code is posted publicly either in zip form directly in the skill brief or linked externally from a repository provider such as GitHub.

All day I have been non-stop preparing the code comments (not the best but commented none the less) and the moment I finished I submitted the skill to my GitHub account.

Unfortunately, in my haste, I failed to notice that I had my Secret Key used for debugging the skill left in the code. What made this more embarrassing is that the code was commented out and not required (why didn't I just delete it). I happened to go to the repository to check something and low and behold I found those keys in the comments. While rushing to remove the keys I received an email from Amazon that read as follows:

Amazon Web Services has opened case REDACTED on your behalf.

The details of the case are as follows:

Case ID: REDACTED
Subject: Your AWS account REDACTED is compromised
Severity: Low
Correspondence: Dear AWS Customer,

Your AWS Account is compromised! Please review the following notice and take immediate action to secure your account.

Your security is important to us. We have become aware that the AWS Access Key REDACTED (belonging to IAM user "REDACTED") along with the corresponding Secret Key is publicly available online at REDACTED.

This poses a security risk to your account and other users, could lead to excessive charges from unauthorized activity or abuse, and violates the AWS Customer Agreement.

Please delete the exposed credentials from your AWS account by using the instructions below and take steps to prevent any new credentials from being published in this manner again. Unfortunately, deleting the keys from the public website and/or disabling them is NOT sufficient to secure your account.

To additionally protect your account from excessive charges, we have temporarily limited your ability to create some AWS resources. Please note that this does not make your account secure, it just partially limits the unauthorized usage for which you could be charged.

Detailed instructions are included below for your convenience.

CHECK FOR UNAUTHORIZED USAGE
We strongly encourage you to immediately review your AWS account for any unauthorized AWS usage, suspect running instances, or inappropriate IAM users and policies. To check the usage, please log into your AWS Management Console and go to each service page to see what resources are being used. Please pay special attention to the running EC2 instances and IAM users, roles, and groups. You can also check for any unexpected usage on the "Bills" page in the Billing console.

https://console.aws.amazon.com/billing/home#/bill

Please keep in mind that unauthorized usage can occur in any region and that in your console you only see one region at a time. To switch between regions, you can use the dropdown in the top-right corner of the console screen.

DELETE THE KEY (ROOT ACCOUNT)
If you are not using the access key, you can simply delete it. To delete the exposed key, visit the "Security Credentials" page here: https://console.aws.amazon.com/iam/home#security_credential. Your keys will be listed in the "Access Keys" section.

DELETE THE KEY (IAM USERS)
Navigate to your IAM Users list in the AWS Management Console, here: https://console.aws.amazon.com/iam/home#users. Please select the IAM user identified above. Click on the "User Actions" drop-down menu and then click "Manage Access Keys" to show that user's active Access Keys. Click "Delete" next to the access key identified above.

ROTATE THE KEY
If your application uses the access key, you need to replace the exposed key with a new one. To do this, first create a second key (at that point both keys will be active) and modify your application to use the new key.
Then disable (but do not delete) the first key. If there are any problems with your application, you can make the first key active again. When your application is fully functional with the first key inactive, please delete the first key.

Please follow the Best Practices of Managing your Access Keys at http://docs.aws.amazon.com/general/latest/gr/aws-access-keys-best-practices.html.

If you have any additional questions or concerns regarding this notification, please reach out to us via the AWS Support Center: https://aws.amazon.com/support.
=======================================

To contact us again about this issue, please use the following link to enter your correspondence or attach any files you think would be useful:

https://console.aws.amazon.com/support/home?#/case/?caseId=REDACTED&displayId=REDACTED&language=en

(If you will connect by federation, log in before following the link.)

Sincerely,
The Amazon Web Services Team

*Please note: this e-mail was sent from an address that cannot accept incoming e-mail. Please use the appropriate link above if you need to contact us again about this same issue.

Amazon Web Services, Inc. is an affiliate of Amazon.com, Inc. Amazon.com is a registered trademark of Amazon.com, Inc. or its affiliates.

Although the email came a few hours after the repository was created I am thoroughly impressed by the speed of it. Having such keys in the wild can allow others to make use of your account and can potentially cause nefarious usage that could be quite costly.

About 40 minutes after clearing the issue I then received the following email:

Dear AWS Customer,

Thank you for deleting your compromised AWS Access Key(s) of account ending with 8681. Please ensure that no unauthorized resources remain on your account, and we ask that you pay special attention to IAM users and EC2 instances. Please take steps to prevent any new credentials from being published in this manner again.

Thank you for trusting your business to AWS. We work hard to keep you safe!

Sincerely,
Amazon Wb Services

Amazon Web Services, Inc. is a subsidiary of Amazon.com, Inc. Amazon.com is a registered trademark of Amazon.com, Inc. This message was produced and distributed by Amazon Web Services Inc., 410 Terry Ave. North, Seattle, WA 98109-5210

I had heard that Amazon actively scan GitHub for sensitive data that has been posted but this is the first time I have seen it in action. The fact that Amazon takes this so seriously is a credit to them.

Regardless of how well Amazon did there, of course, is a lesson here, The change of keys has caused some work (although potentially saved a small fortune). In future, I will be ensuring that such sensitive data is removed prior to committing to a repository (regardless of public or private in all honesty).

Moving forward I will be considering potential workflow methods of ensuring that such data is removed automatically. Do you have such a process? If so what tools do you use?

----- UPDATE -----

Nearly had a heart attack this morning when I saw an email come in entitled "RE: [Case REDACTED] Your AWS account REDACTED is compromised" Thankfully just a further email advising that my account had been reviewed:

Hi There

Thank you for taking quick action to delete your exposed access key.

I reviewed your account and can confirm your exposed access key has been deleted.

It appears that all issues have been addressed I will therefore be resolving your case.

As a reminder, we strongly encourage you to immediately review your AWS account for any unauthorized AWS usage, suspect running instances and volumes, or inappropriate IAM users and policies.

We also encourage you to visit the AWS Security Center for additional information and resources related to AWS security best practices:
http://aws.amazon.com/security/?nc1=h_l3_cc

Please feel free to contact us should you have future questions or comments,as we always want to ensure our customers are happy with our services.

Until then have an awesome day!

Best regards,

Tommy M.
Amazon Web Services

Check out the AWS Support Knowledge Center, a knowledge base of articles and videos that answer customer questions about AWS services: https://aws.amazon.com/premiumsupport/knowledge-center/?icmpid=support_email_category

We value your feedback. Please rate my response using the link below.
===================================================

To contact us again about this case, please return to the AWS Support Center using the following URL:

https://console.aws.amazon.com/support/home#/case/?displayId=REDACTED&language=en

(If you are connecting by federation, log in before following the link.)

*Please note: this e-mail was sent from an address that cannot accept incoming e-mail. Please use the link above if you need to contact us again about this same issue.

====================================================================
Learn to work with the AWS Cloud. Get started with free online videos and self-paced labs at http://aws.amazon.com/training/ ====================================================================

Amazon Web Services, Inc. is an affiliate of Amazon.com, Inc. Amazon.com is a registered trademark of Amazon.com, Inc. or its affiliates.