LASCON 2016 - It's 10PM Do You Know Where Your Access Keys Are?

@cktricky
Download this presentation:
<insert link>

@cktricky
It's 10pm, Do You Know
Where Your Access Keys Are?
Ken Johnson

@cktricky@cktricky
Things to Mention
• DoubleTree by Hilton at 8901 Business
Park Drive in Austin, TX is great at:
– Selling your room for you without telling you
– Fire alarms
– Murdering puppies and kittens
– Created cancer?
– Created cancer in puppies and kittens?

@cktricky@cktricky
Things to Mention
• Ask questions throughout presentation
• There will be no dedicated Q&A – so stick
around after and find me if you want to
chat
• This presentation will cover a lot. Slides
will be available so don’t worry about
minutia.

@cktricky@cktricky
Background/About
• Ken Johnson, CTO and Partner at nVisium
• Veteran, US Navy
• I speak about:
– DevOps (In)Security
– Exploiting Web Applications
– Coding and Coding + Security
– Node, Elixir, Python, Ruby, Go
– AWS Security (clearly)

@cktricky@cktricky
Background/About
This talk came about because…
– I’m the CTO of a security company and we use
AWS… and it is a challenge
– For some, this is a new challenge, and this is my
opportunity to share

@cktricky@cktricky
Background/About
50k foot view of our security plan
– Prevent bad stuff
– Alert to bad stuff
– Recover from bad stuff

@cktricky@cktricky
Our Plan
Our “practical plan”
– Harden – Make it difficult to reach our AWS
environment
– Monitor – If our AWS environment is breached, we
need to know and alert ourselves
– Restore – Have the ability to reconstruct data/configs
after a “hack”

@cktricky@cktricky
AWS’s Plan
The AWS Security Fundamentals Course provides
the framework for your plan:
– You are responsible for leveraging the tools AWS
provides to secure your environment (financially)
– Your configuration… that is on you
– https://aws.amazon.com/training/course-
descriptions/security-fundamentals/

@cktricky@cktricky
Most Security Checklists
Most AWS security talks and documentation
discuss:
– S3 bucket policies
– Security Group configurations
– SSH Key Management
– Encrypting Data (Volumes, S3 buckets)

@cktricky@cktricky
Most Security Checklists
What we’ll mention on the subject:
1. Trusted Advisor – Use it, because it catches a lot of
“low hanging fruit” style issues
2. There are checklists, use them:
– https://media.amazonwebservices.com/AWS_Operational_Che
cklists.pdf
– http://d0.awsstatic.com/whitepapers/compliance/AWS_Auditing
_Security_Checklist.pdf
3. Again let’s reiterate that AWS provides a security
fundamentals course for free (CBT)

@cktricky@cktricky
About / Background Recap
Recap:
– We’re not going to cover basic security fundamentals
of unencrypted volumes, security groups, etc.
– We are going to focus on:
• Hardening
• Monitoring
• Recovery

@cktricky
AWS IAM Hardening Basics
Making it difficult (for attackers) to
reach our environment

@cktricky@cktricky
IAM Hardening Checklist
1. Don’t Use The Root Account!
2. Remove Access Keys for Root Account
3. Audit IAM user policies
4. Multi-Factor Authentication
5. API + MFA
6. Strong Password Policy

@cktricky@cktricky
Don’t Use The Root Account
Every AWS environment has a root account
– Root account is the king/god/all-powerful
– Use only when you absolutely must
– When those circumstances arise, notify your team
first
– This is because we will be configuring alerts to notify
our team when the root account is used

@cktricky@cktricky
Remove Access Keys for Root Account
Simple steps:
– Disable or delete access keys if they exist:
– Implement verbal/written policy that states “we don’t
create access keys for the root account”

@cktricky
Auditing IAM Permissions

@cktricky@cktricky
Audit IAM User Policies
IAM user policy management:
– A single IAM user can have…
• Multiple Managed Policies
• Multiple Inline Policies
• Belong to multiple IAM Groups which…
– Have multiple managed policies
– Have multiple inline policies

@cktricky@cktricky
Explanation
– Managed Policies: Policies that can be
attached to multiple users, groups, or roles
– Inline Policies: Directly attached to a single
user, group, or role

@cktricky@cktricky
Tool to inspect each user’s permissions:
– https://gist.github.com/cktricky/257990df2f36aa3a01a
8809777d49f5d
– Will create a CSV file
– Provides you with
• Usernames
• Inline Policies
• Managed Policies
• Groups

@cktricky@cktricky
Tool output

@cktricky@cktricky
Closer look:

@cktricky@cktricky

@cktricky@cktricky
Why this is important
– If you house sensitive data, you need to know who
has access
– Permissions should be a need-to-have/know situation
in order to limit damage should creds get stolen
– AWS is a flexible environment that changes – your
permission model might need to change with it
(inventory it)

@cktricky
Multi-Factor Authentication
(MFA)

@cktricky@cktricky
MFA
• MFA == 2-Factor Authentication
• If credentials are stolen or guessed, we want a second
layer of protection
• You can use apps or hardware to do this
– Google Authenticator (Apps)
– Gemalto (Hardware)
• Find the full list of MFA devices here:
https://aws.amazon.com/iam/details/mfa/

@cktricky@cktricky
MFA
Let’s demonstrate enabling MFA using a virtual
device (app) on an IAM account

@cktricky@cktricky
MFA
Navigate to Identity & Access Management

@cktricky@cktricky
MFA
Next, manage the MFA device

@cktricky@cktricky
MFA
Choose a virtual device

@cktricky@cktricky
MFA
Lastly, use Google Authenticator to take a snapshot of the
QR code

@cktricky@cktricky
MFA
• At this point, its worth mentioning that non-
administrators or those without IAM privileges
cannot enable MFA on their own account
• Why is this a problem? Well, they need to be
able to enable MFA on their own device… not
the administrator’s
• Fortunately, we have a solution!

@cktricky@cktricky
MFA
• Okay so that wasn’t the easiest to read, so
here is the link:
http://docs.aws.amazon.com/IAM/latest/Us
erGuide/id_credentials_delegate-
permissions_examples.html#creds-
policies-mfa-console
• Basically this IAM policy allows a user to
manage their *OWN* MFA device

@cktricky@cktricky
MFA (for Root Account)
• Need a shared MFA for root? TOTP!
• Recommend using something like
1password for teams, can share the TOTP
code:
https://support.1password.com/guides/mac/totp.html
https://www.youtube.com/watch?v=eZyb-ArMK9g

@cktricky@cktricky
API + MFA
API 101
– This is the alternative to interacting with the AWS
environment via the web console
– Typically used for automated tasks
– Automated tasks means “code”. Luckily, developers
never store keys in source, amiright?
– Hypothetically, what would happen if keys were
leaked?

@cktricky@cktricky
API + MFA
So that’s the “worst case scenario”, more likely:
– Costs unexpectedly and dramatically increase
– We’ll show examples later but remember, you are
financially responsible for your AWS environment’s
configuration
– Let’s talk about prevention

@cktricky@cktricky
API + MFA
• You have the ability to place a restriction where
resources can only be interacted with if the user
has authenticated with MFA
• This helps prevent (ab)use should someone
steal access keys or credentials

@cktricky@cktricky
API + MFA
1. At a minimum, apply to administrator & power user
group policies… really any group that can do anything
of importance

@cktricky@cktricky
API + MFA
This entry requires MFA for Web/API

@cktricky@cktricky
API + MFA
• Truth be told, doing this can be painful at
first
�� Things that used to work, might not (via
the API)
• Fortunately, we have some answers for
you
• Firstly, let’s discuss STS or SecurityToken
Service

@cktricky@cktricky
API + MFA
• Leverage STS in order to interact with the
AWS API should this MFA restriction be
placed on resources (and it should  )
• Example of using STS:
https://gist.github.com/cktricky/127be4e431563a986f0f

@cktricky@cktricky
API + MFA
Use this script to retrieve creds (from gist)

@cktricky@cktricky
API + MFA
Output of script

@cktricky@cktricky
API + MFA
Use the creds to leverage tools like ec2-api-
tools
(-O <access key id>–W <secret> and –T <session token>)

@cktricky@cktricky
API + MFA
And in case you don’t like Ruby…
https://github.com/jimbrowne/aws-sts-
helpers

@cktricky@cktricky
API + MFA
• ElasticBeanstalk does not work with STS. Le
Terrible.
• However, there is a workaround, use
CodePipeline.
• Very simple process to setup but only works
with:
– GitHub
– AWS CodeCommit
– Amazon S3

@cktricky@cktricky
API + MFA
Remember MFA only protects against the web and
NOT the API… unless you change your policies
and use STS

@cktricky@cktricky
Password Policy
• Password policies are important because
historically people do not choose complex
passwords
• MFA should help, but we’re talking about a
layered approach
• Again, making our AWS environment
harder to reach

@cktricky@cktricky
Example Password Policy

@cktricky@cktricky
Hardening Recap
• Make credentials hard to guess
• Make credentials hard to use if stolen with
MFA
• Audit your accounts and their access
• Root account is King, protect your King

@cktricky
AWS Monitoring
Detecting malicious activity

@cktricky@cktricky
AWS Monitoring
• Assuming hardening (prevention) has failed,
how would we know?
• Luckily, AWS provides several services which
alert to anomalies
• We will walk through examples of using these
services, but ultimately decide what is right for
you
• Fair warning, some of these services will provide
a lot of noise

@cktricky@cktricky
AWS Monitoring
4 important services:
1. CloudTrail – Logs
2. SNS – Notifications
3. Config – Alerts for modifications &
noncompliance
4. CloudWatch – Alerts for specific types of
behavior

@cktricky@cktricky
AWS Monitoring

@cktricky@cktricky
AWS Monitoring
CloudTrail
Config
CloudWatch
SNS

@cktricky@cktricky
AWS CloudTrail

@cktricky@cktricky
AWS Monitoring (CloudTrail)
• CloudTrail is primarily used for log collection
• Other services like CloudWatch, for example,
use those logs to filter relevant data

@cktricky@cktricky
Pretty easy, first turn it on..

@cktricky@cktricky
Configure the log group

@cktricky@cktricky
Allow the creation of an IAM role by CloudTrail

@cktricky@cktricky
• At this point you have cloudtrail enabled
• Next step, BEFORE moving to CloudWatch or
Config, is configuring SNS topics

@cktricky@cktricky
AWS Monitoring (SNS)
Fantastic offering, <3 it
– Examples of ways to be notified by SNS
• SMS
• Email
• JSON Post to your Application’s API endpoint
• Lambda

@cktricky@cktricky
• Receive SMS/Email/Slack notifications for
important events
• ^ This is so you get immediate notifications
• You can have multiple subscribers, I’d suggest
you use that functionality
• Basic gist? Receive immediate updates for
things you want to see… immediately ☺

@cktricky@cktricky
Create a topic

@cktricky@cktricky
Create Subscription

@cktricky@cktricky
Create SMS (or whatever, but in this case, SMS)

@cktricky@cktricky
Example of creating email subscription… bottomline you
can have multiple ways of notifying multiple people

@cktricky@cktricky
AWS Monitoring (Config)
Config:
– AWS resource inventory, configuration history, and
configuration change notifications
– Can either design custom Config rules or use
managed (pre-packaged) AWS Config rules
– Discovery
– Change Management
– Compliance
– Incident Response

@cktricky@cktricky
Pre-packaged “Managed” AWS Rules
– CLOUD_TRAIL_ENABLED
– EIP_ATTACHED
– ENCRYPTED_VOLUMES
– INCOMING_SSH_DISABLED
– INSTANCES_IN_VPC
– REQUIRED_TAGS
– RESTRICTED_INCOMING_TRAFFIC
– ROOT_MFA_ACCOUNT_ENABLED
– RDS_STORAGE_ENCRYPTED

@cktricky@cktricky
…And there are more and this list grows
https://docs.aws.amazon.com/config/latest/d
eveloperguide/evaluate-config_use-
managed-rules.html

@cktricky@cktricky
Examples of things you can have alerts set for:
– Change in Firewall (Security Group) ports
– Changes in VPC
– Any change… at all

@cktricky@cktricky
Go to the Config service and choose resources to track

@cktricky@cktricky
Or choose to track everything

@cktricky@cktricky
Create a bucket, create an SNS topic (…we’ll discuss next)

@cktricky@cktricky
Allow the role to be created and you’re all set!

@cktricky@cktricky
AWS CloudWatch

@cktricky@cktricky
AWS Monitoring (CloudWatch)
• We can be very particular here about what it is we want
to see
• Some very interesting things you can monitor
• Some examples:
– Billing Alerts (Important for detection of abuse or
mistakes)
– Track Root Account Usage
– Failed login attempts
– Unauthorized Activity

@cktricky@cktricky
AWS Monitoring (CloudWatch - Billing)
• Used to prevent abuse or mistakes from costing your
organization money
• Analyze and approximate your monthly spend
• Configure via CloudWatch
• Use SNS for instantaneous alerting

@cktricky@cktricky
Navigate to billing & cost management; enable
billing alerts

@cktricky@cktricky
Create an SNS topic

@cktricky@cktricky
Subscribe to Topic

@cktricky@cktricky
Navigate to CloudWatch -> Metrics -> Billing

@cktricky@cktricky
Choose USD/EstimateCharges -> Create Alarm

@cktricky@cktricky
Set price point, SNS topic, and create alarm

@cktricky@cktricky
Exact steps to enable can be found here:
http://docs.aws.amazon.com/awsaccountbilli
ng/latest/aboutv2/free-tier-alarms.html

@cktricky@cktricky
AWS Monitoring (CloudWatch – Root Login)
• Remember how I said don’t use the Root
account routinely?
• BUT… if this account is used, you should
know about it
• This is the reason you’ll want to notify
others (who receive SNS alerts) of the fact
you are about to use the account

@cktricky@cktricky
Choose log group, create metric

@cktricky@cktricky
Define Logs Metric Filter

@cktricky@cktricky
Assign/Create Filter

@cktricky@cktricky
Click “Create Alarm”

@cktricky@cktricky
Define Alarm and you’re good…

@cktricky@cktricky
Exact steps (with pics) exist here:
https://blogs.aws.amazon.com/security/post/Tx3PSPQSN8
374D/How-to-Receive-Notifications-When-Your-AWS-
Account-s-Root-Access-Keys-Are-Used

@cktricky@cktricky
AWS Monitoring (CloudWatch – Failed Logins)
• In the event someone is trying to break in,
let’s alert ourselves to this!
• Failed logins typically suggest either
someone forgot their password or…
someone is trying to guess yours

@cktricky@cktricky
The steps are pretty much the same as the
root login alarm
However, the Filter pattern is different

@cktricky@cktricky
Enter the relevant filter pattern, click create

@cktricky@cktricky
• Exact steps exist here:
http://docs.aws.amazon.com/awscloudtrail/la
test/userguide/cloudwatch-alarms-for-
cloudtrail.html#cloudwatch-alarms-for-
cloudtrail-signin

@cktricky
IAM Unauthorized Activity

@cktricky@cktricky
• Aws-interrogate tool
• This alarm is the antidote
• Alerts us when someone is trying to
access something in AWS, and does not
have permissions

@cktricky@cktricky
• Steps are same as root login, failed logins,
etc.
• Filter pattern is different

@cktricky@cktricky
Enter relevant filter pattern

@cktricky@cktricky
AWS Monitoring (Unauthorized Activity)
What happens when we run interrogate

@cktricky@cktricky
AWS Monitoring (Unauthorized Activity)
The result of doing that is a nice nifty email to the
engineering & security team

@cktricky@cktricky
AWS Monitoring (CloudWatch) – Filter Patterns
Create your own custom filter patterns, here is a
resource for that:
http://docs.aws.amazon.com/AmazonCloudWatch/latest/De
veloperGuide/FilterAndPatternSyntax.html

@cktricky@cktricky
Lambda+ Slack

@cktricky@cktricky
Lambda & Slack
Select slack-cloudwatch blueprint when creating
function

@cktricky@cktricky
Lambda & Slack
Configure trigger

@cktricky@cktricky
Lambda & Slack
The function is pre-populated (coded)

@cktricky@cktricky
Lambda & Slack
Start configuring incoming webhook

@cktricky@cktricky
Lambda & Slack
Add configuration inside of slack

@cktricky@cktricky
Lambda & Slack
Choose the channel (can also choose pic, name,
etc.)

@cktricky@cktricky
Lambda & Slack
Grab the webhook URL

@cktricky@cktricky
Lambda & Slack
Create KMS key, later used to decrypt

@cktricky@cktricky
Lambda & Slack
Name the key, follow steps 1 - 4

@cktricky@cktricky
Lambda & Slack
Use the AWS KMS encrypt function to
encrypt the webhook URL

@cktricky@cktricky
Lambda & Slack
Put the Base 64 encoded + KMS encrypted URL
into the code:

@cktricky@cktricky
Lambda & Slack
The result:

@cktricky@cktricky
AWS + Splunk

@cktricky@cktricky
AWS + Splunk
Splunk is a pretty great resource for monitoring
activity
• Two separate plugins:
– Splunk App for AWS
• https://splunkbase.splunk.com/app/1274/
– Splunk Add-On
• https://splunkbase.splunk.com/app/1876/

@cktricky@cktricky
AWS + Splunk
• Examples of things you can view:
– Billing
– Topology
– Usage
– IAM Activity
– SSH Key Pair Activity
– User Activity
– Network ACL(s)
– VPC Activity
and a lot more…

@cktricky@cktricky
AWS + Splunk
• Pretty Screenshot 1

@cktricky@cktricky
AWS + Splunk

@cktricky@cktricky
AWS + Splunk
• Splunk will need an AWS account in order
to retrieve data
• Create account(s) for Splunk, grab the
necessary permission policy from here:
http://docs.splunk.com/Documentation/AddOns/r
eleased/AWS/ConfigureAWSpermissions

@cktricky@cktricky
AWS + Splunk
• Configure AWS App for Splunk, add account(s),
configure each input accordingly:

@cktricky@cktricky
AWS + Splunk
• To view things like IAM Activity…
– Subscribe to a cloudtrail log via SNS
– Utilize SQS and subscribe SQS to an SNS
Topic

@cktricky@cktricky
AWS Monitoring Recap
• Alert yourself when things change
• This will get noisy, find a way to filter that which is
important
– If it’s a high risk event, send an SMS/Slack/Email
blast
• At a minimum, alert yourself when odd things occur…
like:
– Billing increases past your normal spend
– When somebody authenticates as Root
– When someone has a login failure
– Unauthorized IAM Activity

@cktricky@cktricky
AWS Monitoring Recap
• Interesting Quora thread:
– https://www.quora.com/My-AWS-account-was-hacked-and-I-
have-a-50-000-bill-how-can-I-reduce-the-amount-I-need-to-pay
• Highlights from the article:
– AWS has “a review board of sorts” to determine if you should be
refunded
– Bots are scouring GitHub searching for exposed access keys
– One of the more AWS-seasoned responders mentioned doing
part of what we discussed here today to avoid it
– A decent number of the people posting on this thread said “Yes,
happened to me too”

@cktricky
AWS Restoration & Recovery
Plan to fail, just don’t fail to plan

@cktricky@cktricky
AWS Restoration & Recovery – Basic Incident
Response (IR)
• Understand who to contact if things go bad
• Understand how to communicate (ex:
“speak only over the phone”)
• Understand what information to parse
• Understand where your backups are
located and how they are secured

@cktricky@cktricky
AWS Restoration & Recovery – Basic IR
• Do not USE AWS TO BACKUP YOUR
AWS
• Offsite backups (meaning, off AWS site)
• Common things to back-up:
– Databases/ Snapshots
– S3 Buckets
– EBS Volumes
– CloudFormation Templates

@cktricky@cktricky
AWS Restoration & Recovery – Basic IR
• Resources:
– http://stackoverflow.com/questions/17087542/
backup-solutions-for-aws-ec2-instances
– https://github.com/Scalr/installer-ng
– http://www.n2ws.com/blog/3-ways-ec2-
windows-backup-and-recovery.html

@cktricky
AWS Incident Response
Resources

@cktricky@cktricky
AWS Incident Response
• Scout 2 -- https://github.com/nccgroup/Scout2
• Andrew Krug & Alex McCormack – Hardening AWS
Environments and Automating Incident Response
– https://www.youtube.com/watch?v=cmEUxxYFjK8

@cktricky
Presentation Recap
Summary

@cktricky@cktricky
Recap
• DoubleTree by Hilton at 8901 Business
Park Drive in Austin, TX
– Sells your room
– Loves fire alarms at 5am
– Behind 9/11?
– Can go f**k itself

@cktricky@cktricky
Recap
• Makes your environment harder to reach… for
the bad guys
– Limit what stolen or “otherwise obtained”
access keys or credentials could be used to
do
– Prevent them being stolen in the first place
• Alert yourself to anomalies
• Have a plan for if things go bad
• Stay safe out there!

@cktricky
Contact Info
• My Info
• Twitter: @cktricky
• Email: ken@nvisium.com

LASCON 2016 - It's 10PM Do You Know Where Your Access Keys Are?

Related slideshows

More Related Content

LASCON 2016 - It's 10PM Do You Know Where Your Access Keys Are?

Editor's Notes