SlideShare a Scribd company logo

AWS re:Invent 2016: The Effective AWS CLI User (DEV402)

•
•
Amazon Web Services
Amazon Web Services

Understanding the internals of the AWS CLI will make you a more effective user. This talk provides a deep dive into the architecture, debugging techniques, advanced usage patterns, and some of its more advanced features. We recommend this talk for people who are already familiar with the AWS CLI because the topics will be advanced. By the end of the talk, audience members will have a deeper understanding of the AWS CLI, allowing them to tackle current or future applications from a different perspective.

1 of 155
Download to read offline
© 2016, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Kyle Knapp, Amazon Web Services December 1, 2016 DEV402 The Effective AWS CLI User
Important: This talk will be advanced
Background Material • AWS CLI User Guide • AWS CLI Command Reference • 2015 AWS CLI re:Invent talk • 2014 AWS CLI re:Invent talk • 2013 AWS CLI re:Invent talk
Background Material • AWS CLI User Guide • AWS CLI Command Reference • 2015 AWS CLI re:Invent talk • 2014 AWS CLI re:Invent talk • 2013 AWS CLI re:Invent talk Slides will be online!
Unified tool to manage AWS services AWS Command Line Interface
The Effective AWS CLI User Tenets The effective AWS CLI user:
The Effective AWS CLI User Tenets The effective AWS CLI user:  Uses an iterative workflow
The Effective AWS CLI User Tenets The effective AWS CLI user:  Uses an iterative workflow  Troubleshoots well
The Effective AWS CLI User Tenets The effective AWS CLI user:  Uses an iterative workflow  Troubleshoots well  Is resourceful with tooling
The Effective AWS CLI User Tenets The effective AWS CLI user:  Uses an iterative workflow  Troubleshoots well  Is resourceful with tooling  Understands performance implications
A Sample Application
Amazon VPC
Amazon VPC Amazon EC2
Amazon VPC Amazon EC2 IAM
Amazon VPC Amazon EC2 IAM Amazon S3
AWS Cloud
VPC 10.0.0.0/16 AWS Cloud router Destination Target 10.0.0.0/16 local
VPC 10.0.0.0/16 Subnet 10.0.0.0/24 AWS Cloud router Destination Target 10.0.0.0/16 local
VPC 10.0.0.0/16 Subnet 10.0.0.0/24 AWS Cloud Internet gateway router Destination Target 10.0.0.0/16 local 0.0.0.0/0 igw-id
VPC 10.0.0.0/16 Subnet 10.0.0.0/24 security group AWS Cloud Internet gateway router Destination Target 10.0.0.0/16 local 0.0.0.0/0 igw-id
VPC 10.0.0.0/16 Subnet 10.0.0.0/24 security group AWS Cloud instance Internet gateway router Destination Target 10.0.0.0/16 local 0.0.0.0/0 igw-id role
VPC 10.0.0.0/16 Subnet 10.0.0.0/24 security group AWS Cloud instance Internet gateway router Amazon S3 Destination Target 10.0.0.0/16 local 0.0.0.0/0 igw-id role
The Effective AWS CLI User Tenets The effective AWS CLI user:  Uses an iterative workflow
$ aws ec2 create-vpc --cidr-block 10.0.0.0/16
$ aws ec2 create-vpc --cidr-block 10.0.0.0/16 { "Vpc": { "VpcId": "vpc-24de2d4d", "InstanceTenancy": "default", "State": "pending", "DhcpOptionsId": "dopt-f8ca2291", "CidrBlock": "10.0.0.0/16", "IsDefault": false } }
$ aws ec2 create-vpc --cidr-block 10.0.0.0/16 { "Vpc": { "VpcId": "vpc-24de2d4d", "InstanceTenancy": "default", "State": "pending", "DhcpOptionsId": "dopt-f8ca2291", "CidrBlock": "10.0.0.0/16", "IsDefault": false } }
$ aws ec2 create-vpc --cidr-block 10.0.0.0/16 { "Vpc": { "VpcId": "vpc-24de2d4d", "InstanceTenancy": "default", "State": "pending", "DhcpOptionsId": "dopt-f8ca2291", "CidrBlock": "10.0.0.0/16", "IsDefault": false } } $ vpc_id=vpc-24de2d4d
$ aws ec2 create-vpc --cidr-block 10.0.0.0/16 { "Vpc": { "VpcId": "vpc-24de2d4d", "InstanceTenancy": "default", "State": "pending", "DhcpOptionsId": "dopt-f8ca2291", "CidrBlock": "10.0.0.0/16", "IsDefault": false } } $ vpc_id=vpc-24de2d4d We can do better…
$ vpc_id=$(aws ec2 create-vpc --cidr-block 10.0.0.0/16 --query Vpc.VpcId --output text)
$ vpc_id=$(aws ec2 create-vpc --cidr-block 10.0.0.0/16 --query Vpc.VpcId --output text) { "Vpc": { "VpcId": "vpc-24de2d4d", "InstanceTenancy": "default", "State": "pending", "DhcpOptionsId": "dopt-f8ca2291", "CidrBlock": "10.0.0.0/16", "IsDefault": false } }
$ vpc_id=$(aws ec2 create-vpc --cidr-block 10.0.0.0/16 --query Vpc.VpcId --output text) { "Vpc": { "VpcId": "vpc-24de2d4d", "InstanceTenancy": "default", "State": "pending", "DhcpOptionsId": "dopt-f8ca2291", "CidrBlock": "10.0.0.0/16", "IsDefault": false } }
$ vpc_id=$(aws ec2 create-vpc --cidr-block 10.0.0.0/16 --query Vpc.VpcId --output text) { "Vpc": { "VpcId": "vpc-24de2d4d", "InstanceTenancy": "default", "State": "pending", "DhcpOptionsId": "dopt-f8ca2291", "CidrBlock": "10.0.0.0/16", "IsDefault": false } }
$ vpc_id=$(aws ec2 create-vpc --cidr-block 10.0.0.0/16 --query Vpc.VpcId --output text) { "Vpc": { "VpcId": "vpc-24de2d4d", "InstanceTenancy": "default", "State": "pending", "DhcpOptionsId": "dopt-f8ca2291", "CidrBlock": "10.0.0.0/16", "IsDefault": false } }
$ vpc_id=$(aws ec2 create-vpc --cidr-block 10.0.0.0/16 --query Vpc.VpcId --output text) { "Vpc": { "VpcId": "vpc-24de2d4d", "InstanceTenancy": "default", "State": "pending", "DhcpOptionsId": "dopt-f8ca2291", "CidrBlock": "10.0.0.0/16", "IsDefault": false } }
$ vpc_id=$(aws ec2 create-vpc --cidr-block 10.0.0.0/16 --query Vpc.VpcId --output text) $ echo "$vpc_id” vpc-24de2d4d
$ vpc_id=$(aws ec2 create-vpc --cidr-block 10.0.0.0/16 --query Vpc.VpcId --output text) $ echo "$vpc_id" vpc-24de2d4d $ create_vpc_output=$(aws ec2 create-vpc --cidr-block 10.0.0.0/16)
$ vpc_id=$(aws ec2 create-vpc --cidr-block 10.0.0.0/16 --query Vpc.VpcId --output text) $ echo "$vpc_id" vpc-24de2d4d $ create_vpc_output=$(aws ec2 create-vpc --cidr-block 10.0.0.0/16) $ vpc_id=$(jp -u Vpc.VpcId <<< "$create_vpc_output“)
$ vpc_id=$(aws ec2 create-vpc --cidr-block 10.0.0.0/16 --query Vpc.VpcId --output text) $ echo "$vpc_id" vpc-24de2d4d $ create_vpc_output=$(aws ec2 create-vpc --cidr-block 10.0.0.0/16) $ vpc_id=$(jp -u Vpc.VpcId <<< "$create_vpc_output") $ echo "$vpc_id" vpc-24de2d4d
$ vpc_id=$(aws ec2 create-vpc --cidr-block 10.0.0.0/16 --query Vpc.VpcId --output text) $ echo "$vpc_id" vpc-24de2d4d $ create_vpc_output=$(aws ec2 create-vpc --cidr-block 10.0.0.0/16) $ vpc_id=$(jp -u Vpc.VpcId <<< "$create_vpc_output") $ echo "$vpc_id" vpc-24de2d4d And we can make this even better!
Problem Statement
Demo $
Let’s talk about how
$ aws ec2 create-vpc --cidr-block 10.0.0.0/16 --generate-cli-skeleton output { "Vpc": { "VpcId": "VpcId", "State": "State", "CidrBlock": "CidrBlock", "DhcpOptionsId": "DhcpOptionsId", "Tags": [ { "Key": "Key", "Value": "Value" } ], "InstanceTenancy": "InstanceTenancy", "IsDefault": true } }
$ aws ec2 create-vpc --cidr-block 10.0.0.0/16 --generate-cli-skeleton output { "Vpc": { "VpcId": "VpcId", "State": "State", "CidrBlock": "CidrBlock", "DhcpOptionsId": "DhcpOptionsId", "Tags": [ { "Key": "Key", "Value": "Value" } ], "InstanceTenancy": "InstanceTenancy", "IsDefault": true } } "Vpc":{ "type":"structure", "members":{ "VpcId":{ "shape":"String" }, "State":{ "shape":"VpcState" }, "CidrBlock":{ "shape":"String" }, "DhcpOptionsId":{ "shape":"String" }, "Tags":{ "shape":"TagList" }, "InstanceTenancy":{ "shape":"Tenancy" }, "IsDefault":{ "shape":"Boolean" } } } botocore/botocore/data/ec2/2016-09-15/service-2.json
$ aws ec2 create-vpc --cidr-block 10.0.0.0/16 --generate-cli-skeleton output { "Vpc": { "VpcId": "VpcId", "State": "State", "CidrBlock": "CidrBlock", "DhcpOptionsId": "DhcpOptionsId", "Tags": [ { "Key": "Key", "Value": "Value" } ], "InstanceTenancy": "InstanceTenancy", "IsDefault": true } } "Vpc":{ "type":"structure", "members":{ "VpcId":{ "shape":"String" }, "State":{ "shape":"VpcState" }, "CidrBlock":{ "shape":"String" }, "DhcpOptionsId":{ "shape":"String" }, "Tags":{ "shape":"TagList" }, "InstanceTenancy":{ "shape":"Tenancy" }, "IsDefault":{ "shape":"Boolean" } } } botocore/botocore/data/ec2/2016-09-15/service-2.json
$ aws ec2 create-vpc --cidr-block 10.0.0.0/16 --generate-cli-skeleton output { "Vpc": { "VpcId": "VpcId", "State": "State", "CidrBlock": "CidrBlock", "DhcpOptionsId": "DhcpOptionsId", "Tags": [ { "Key": "Key", "Value": "Value" } ], "InstanceTenancy": "InstanceTenancy", "IsDefault": true } } "Vpc":{ "type":"structure", "members":{ "VpcId":{ "shape":"String" }, "State":{ "shape":"VpcState" }, "CidrBlock":{ "shape":"String" }, "DhcpOptionsId":{ "shape":"String" }, "Tags":{ "shape":"TagList" }, "InstanceTenancy":{ "shape":"Tenancy" }, "IsDefault":{ "shape":"Boolean" } } } botocore/botocore/data/ec2/2016-09-15/service-2.json
$ aws ec2 create-vpc --cidr-block 10.0.0.0/16 --generate-cli-skeleton output { "Vpc": { "VpcId": "VpcId", "State": "State", "CidrBlock": "CidrBlock", "DhcpOptionsId": "DhcpOptionsId", "Tags": [ { "Key": "Key", "Value": "Value" } ], "InstanceTenancy": "InstanceTenancy", "IsDefault": true } } "Vpc":{ "type":"structure", "members":{ "VpcId":{ "shape":"String" }, "State":{ "shape":"VpcState" }, "CidrBlock":{ "shape":"String" }, "DhcpOptionsId":{ "shape":"String" }, "Tags":{ "shape":"TagList" }, "InstanceTenancy":{ "shape":"Tenancy" }, "IsDefault":{ "shape":"Boolean" } } } botocore/botocore/data/ec2/2016-09-15/service-2.json
AWS Cloud VPC 10.0.0.0/16 Destination Target 10.0.0.0/16 local router
VPC 10.0.0.0/16 Subnet 10.0.0.0/24 AWS Cloud Internet gateway router Destination Target 10.0.0.0/16 local 0.0.0.0/0 igw-id
Demo $
Important Takeaways • --generate-cli-skeleton output • Leverage UNIX commands (e.g. echo, history)
The Effective AWS CLI User Tenets The effective AWS CLI user:  Troubleshoots well
$ aws ec2 create-security-group --group-name re:Invent --vpc-id VpcId
$ aws ec2 create-security-group --group-name re:Invent --vpc-id VpcId aws: error: argument --description is required
$ aws ec2 create-security-group --group-name re:Invent --vpc-id VpcId aws: error: argument --description is required $ aws ec2 create-security-group --group-name re:Invent --description 're:Invent demo' --vpc-id VpcId
$ aws ec2 create-security-group --group-name re:Invent --vpc-id VpcId aws: error: argument --description is required $ aws ec2 create-security-group --group-name re:Invent --description 're:Invent demo' --vpc-id VpcId An error occurred (InvalidVpcID.NotFound) when calling the CreateSecurityGroup operation: The vpc ID 'VpcId' does not exist
--debug
$ aws ec2 create-security-group --group-name re:Invent --description 're:Invent demo' --vpc-id "$vpc_id” --debug
$ aws ec2 create-security-group --group-name re:Invent --description 're:Invent demo' --vpc-id "$vpc_id” --debug
$ aws ec2 create-security-group --group-name re:Invent --description 're:Invent demo' --vpc-id "$vpc_id" -–debug 2016-11-04 10:30:33,532 - MainThread - awscli.clidriver - DEBUG - CLI version: aws- cli/1.11.10 Python/2.7.10 Darwin/15.6.0 botocore/1.4.65 2016-11-04 10:30:33,533 - MainThread - awscli.clidriver - DEBUG - Arguments entered to CLI: ['ec2', 'create-security-group', '--group-name', 're:Invent', '--description', 're:Invent demo', '--vpc-id', 'vpc-43cb382a', '--debug'] 2016-11-04 10:30:33,533 - MainThread - botocore.hooks - DEBUG - Event session- initialized: calling handler <function add_scalar_parsers at 0x10eefc488> 2016-11-04 10:30:33,533 - MainThread - botocore.hooks - DEBUG - Event session- initialized: calling handler <function inject_assume_role_provider_cache at 0x10ecde578> 2016-11-04 10:30:33,533 - MainThread - botocore.credentials - DEBUG - Skipping environment variable credential check because profile name was explicitly set. 2016-11-04 10:30:33,537 - MainThread - botocore.loaders - DEBUG - Loading JSON file: /Users/kyleknap/GitHub/botocore/botocore/data/ec2/2016-09-15/service-2.json 2016-11-04 10:30:33,622 - MainThread - botocore.hooks - DEBUG - Event service-data- loaded.ec2: calling handler <function register_retries_for_service at 0x10e904500>
2016-11-04 10:30:33,623 - MainThread - botocore.handlers - DEBUG - Registering retry handlers for service: ec2 2016-11-04 10:30:33,628 - MainThread - botocore.hooks - DEBUG - Event building-command- table.ec2: calling handler <functools.partial object at 0x10ef02940> 2016-11-04 10:30:33,628 - MainThread - awscli.customizations.removals - DEBUG - Removing operation: import-instance 2016-11-04 10:30:33,628 - MainThread - awscli.customizations.removals - DEBUG - Removing operation: import-volume 2016-11-04 10:30:33,628 - MainThread - botocore.hooks - DEBUG - Event building-command- table.ec2: calling handler <function add_waiters at 0x10ef03578> 2016-11-04 10:30:33,631 - MainThread - botocore.loaders - DEBUG - Loading JSON file: /Users/kyleknap/GitHub/botocore/botocore/data/ec2/2016-09-15/waiters-2.json 2016-11-04 10:30:33,635 - MainThread - awscli.clidriver - DEBUG - OrderedDict([(u'dry- run', <awscli.arguments.BooleanArgument object at 0x10febe950>), (u'no-dry-run', <awscli.arguments.BooleanArgument object at 0x10febe990>), (u'group-name', <awscli.arguments.CLIArgument object at 0x10febe9d0>), (u'description', <awscli.arguments.CLIArgument object at 0x10febea10>), (u'vpc-id', <awscli.arguments.CLIArgument object at 0x10febea50>)])
2016-11-04 10:30:33,623 - MainThread - botocore.handlers - DEBUG - Registering retry handlers for service: ec2 2016-11-04 10:30:33,628 - MainThread - botocore.hooks - DEBUG - Event building-command- table.ec2: calling handler <functools.partial object at 0x10ef02940> 2016-11-04 10:30:33,628 - MainThread - awscli.customizations.removals - DEBUG - Removing operation: import-instance 2016-11-04 10:30:33,628 - MainThread - awscli.customizations.removals - DEBUG - Removing operation: import-volume 2016-11-04 10:30:33,628 - MainThread - botocore.hooks - DEBUG - Event building-command- table.ec2: calling handler <function add_waiters at 0x10ef03578> 2016-11-04 10:30:33,631 - MainThread - botocore.loaders - DEBUG - Loading JSON file: /Users/kyleknap/GitHub/botocore/botocore/data/ec2/2016-09-15/waiters-2.json 2016-11-04 10:30:33,635 - MainThread - awscli.clidriver - DEBUG - OrderedDict([(u'dry- run', <awscli.arguments.BooleanArgument object at 0x10febe950>), (u'no-dry-run', <awscli.arguments.BooleanArgument object at 0x10febe990>), (u'group-name', <awscli.arguments.CLIArgument object at 0x10febe9d0>), (u'description', <awscli.arguments.CLIArgument object at 0x10febea10>), (u'vpc-id', <awscli.arguments.CLIArgument object at 0x10febea50>)]) And this continues on for a while…
Parse command $ aws ec2 create-security-group --group-name re:Invent --description 're:Invent demo' --vpc-id vpc-24de2d4d --query GroupId --output text
Parse command botocore client call # Python code response = ec2_client.create_security_group( GroupName=‘re:Invent’, Description=‘re:Invent demo’, VpcId=‘vpc-24de2d4d’ ) $ aws ec2 create-security-group --group-name re:Invent --description 're:Invent demo' --vpc-id vpc-24de2d4d --query GroupId --output text
Parse command botocore client call $ aws ec2 create-security-group --group-name re:Invent --description 're:Invent demo' --vpc-id vpc-24de2d4d --query GroupId --output text # Python code response = ec2_client.create_security_group( GroupName=‘re:Invent’, Description=‘re:Invent demo’, VpcId=‘vpc-24de2d4d’ )
Parse command botocore client call $ aws ec2 create-security-group --group-name re:Invent --description 're:Invent demo' --vpc-id vpc-24de2d4d --query GroupId --output text # Python code response = ec2_client.create_security_group( GroupName=‘re:Invent’, Description=‘re:Invent demo’, VpcId=‘vpc-24de2d4d’ )
Parse command Format response botocore client call sg-1d032e64 $ aws ec2 create-security-group --group-name re:Invent --description 're:Invent demo' --vpc-id vpc-24de2d4d --query GroupId --output text # Python code response = ec2_client.create_security_group( GroupName=‘re:Invent’, Description=‘re:Invent demo’, VpcId=‘vpc-24de2d4d’ )
Parse command Format response botocore client call $ aws ec2 create-security-group --group-name re:Invent --description 're:Invent demo' --vpc-id vpc-24de2d4d --query GroupId --output text # Python code response = ec2_client.create_security_group( GroupName=‘re:Invent’, Description=‘re:Invent demo’, VpcId=‘vpc-24de2d4d’ ) sg-1d032e64
Parse command Format response botocore client call $ aws ec2 create-security-group --group-name re:Invent --description 're:Invent demo' --vpc-id vpc-24de2d4d --query GroupId --output text # Python code response = ec2_client.create_security_group( GroupName=‘re:Invent’, Description=‘re:Invent demo’, VpcId=‘vpc-24de2d4d’ ) sg-1d032e64
Parse command Format response botocore client call # Python code response = ec2_client.create_security_group( GroupName=‘re:Invent’, Description=‘re:Invent demo’, VpcId=‘vpc-24de2d4d’ )
Parse command Format response botocore client call Validate/Serialize parameters "CreateSecurityGroupRequest”{ "type":"structure”, "required":["GroupName", "Description”], "members":{ "DryRun":{ "shape":"Boolean” }, "GroupName":{ "shape":"String” }, "Description":{ "shape":"String” }, "VpcId":{ "shape":"String” } } } botocore/botocore/data/ec2/2016-09-15/service-2.json
Parse command Format response Make botocore client call Validate/Serialize parameters HTTP request/response
Parse command Format response Make botocore client call Validate/Serialize parameters HTTP request/response Parse response "CreateSecurityGroupResult”:{ "type":"structure”, "members":{ "GroupId":{ "shape":"String” } } } botocore/botocore/data/ec2/2016-09-15/service-2.json
Parse command Format response Validate/Serialize parameters HTTP request/response Parse response --debug
Parse command Format response Validate/Serialize parameters HTTP request/response Parse response $ aws ec2 create-security-group --group-name re:Invent --description 're:Invent demo’ --vpc-id "$vpc_id" -–debug
Parse command Format response Validate/Serialize parameters HTTP request/response Parse response awscli.clidriver - DEBUG - Arguments entered to CLI: ['ec2', 'create-security- group', '--group-name', 're:Invent’, '--description', 're:Invent demo’, '--vpc-id', 'vpc-43cb382a', '--debug'] $ aws ec2 create-security-group --group-name re:Invent --description 're:Invent demo' --vpc-id "$vpc_id" -–debug
Parse command Format response Validate/Serialize parameters HTTP request/response Parse response awscli.clidriver - DEBUG - Arguments entered to CLI: ['ec2', 'create-security- group', '--group-name', 're:Invent’, '--description', 're:Invent demo’, '--vpc-id', 'vpc-43cb382a', '--debug'] $ aws ec2 create-security-group --group-name re:Invent --description 're:Invent demo' --vpc-id "$vpc_id" -–debug
Parse command Format response Validate/Serialize parameters HTTP request/response Parse response awscli.clidriver - DEBUG - Arguments entered to CLI: ['ec2', 'create-security- group', '--group-name', 're:Invent’, '--description', 're:Invent demo’, '--vpc-id', 'vpc-43cb382a', '--debug'] $ aws ec2 create-security-group --group-name re:Invent --description 're:Invent demo' --vpc-id "$vpc_id" -–debug
Parse command Format response Validate/Serialize parameters HTTP request/response Parse response $ aws ec2 create-security-group --group-name re:Invent --description file://description.txt --vpc-id "$vpc_id" -–debug
Parse command Format response Validate/Serialize parameters HTTP request/response Parse response $ aws ec2 create-security-group --group-name re:Invent --description file://description.txt --vpc-id "$vpc_id" -–debug
Parse command Format response Validate/Serialize parameters HTTP request/response Parse response $ aws ec2 create-security-group --group-name re:Invent --description file://description.txt --vpc-id "$vpc_id" -–debug awscli.clidriver - DEBUG - Arguments entered to CLI: ['ec2', 'create-security- group', '--group-name', 're:Invent’, '--description', 'file://description.txt’, '--vpc-id', 'vpc-43cb382a', '--debug']
Parse command Format response Validate/Serialize parameters HTTP request/response Parse response $ aws ec2 create-security-group --group-name re:Invent --description file://description.txt --vpc-id "$vpc_id" -–debug awscli.clidriver - DEBUG - Arguments entered to CLI: ['ec2', 'create-security- group', '--group-name', 're:Invent’, '--description', 'file://description.txt’, '--vpc-id', 'vpc-43cb382a', '--debug']
Parse command Format response Validate/Serialize parameters HTTP request/response Parse response awscli.arguments - DEBUG - Unpacked value of u're:Invent demo' for parameter "description": u're:Invent demo' $ aws ec2 create-security-group --group-name re:Invent --description file://description.txt --vpc-id "$vpc_id" -–debug awscli.clidriver - DEBUG - Arguments entered to CLI: ['ec2', 'create-security- group', '--group-name', 're:Invent’, '--description', 'file://description.txt’, '--vpc-id', 'vpc-43cb382a', '--debug']
Parse command Format response Validate/Serialize parameters HTTP request/response Parse response awscli.arguments - DEBUG - Unpacked value of u're:Invent demo' for parameter "description": u're:Invent demo' $ aws ec2 create-security-group --group-name re:Invent --description file://description.txt --vpc-id "$vpc_id" -–debug awscli.clidriver - DEBUG - Arguments entered to CLI: ['ec2', 'create-security- group', '--group-name', 're:Invent’, '--description', 'file://description.txt’, '--vpc-id', 'vpc-43cb382a', '--debug']
Parse command Format response Validate/Serialize parameters HTTP request/response Parse response $ aws ec2 create-security-group --group-name re:Invent --description 're:Invent demo' --vpc-id "$vpc_id" -–debug
Parse command Format response Validate/Serialize parameters HTTP request/response Parse response $ aws ec2 create-security-group --group-name re:Invent --description 're:Invent demo' --vpc-id "$vpc_id" -–debug botocore.endpoint - DEBUG - Making request for OperationModel(name=CreateSecurityGroup) (verify_ssl=True) with params: { 'body': { 'Action': 'CreateSecurityGroup', 'GroupName': 're:Invent’, 'Version': '2016-09-15’, 'VpcId': 'vpc-43cb382a', ‘GroupDescription': 're:Invent demo’ }, ‘url':'https://ec2.us-east-2.amazonaws.com/’, ... }
Parse command Format response Validate/Serialize parameters HTTP request/response Parse response $ aws ec2 create-security-group --group-name re:Invent --description 're:Invent demo' --vpc-id "$vpc_id" -–debug botocore.endpoint - DEBUG - Making request for OperationModel(name=CreateSecurityGroup) (verify_ssl=True) with params: { 'body': { 'Action': 'CreateSecurityGroup', 'GroupName': 're:Invent’, 'Version': '2016-09-15’, 'VpcId': 'vpc-43cb382a', ‘GroupDescription': 're:Invent demo’ }, ‘url':'https://ec2.us-east-2.amazonaws.com/’, ... }
Parse command Format response Validate/Serialize parameters HTTP request/response Parse response $ aws ec2 create-security-group --group-name re:Invent --description 're:Invent demo' --vpc-id "$vpc_id" -–debug botocore.endpoint - DEBUG - Making request for OperationModel(name=CreateSecurityGroup) (verify_ssl=True) with params: { 'body': { 'Action': 'CreateSecurityGroup', 'GroupName': 're:Invent’, 'Version': '2016-09-15’, 'VpcId': 'vpc-43cb382a', ‘GroupDescription': 're:Invent demo’ }, ‘url':'https://ec2.us-east-2.amazonaws.com/’, ... }
Parse command Format response Validate/Serialize parameters HTTP request/response Parse response $ aws ec2 create-security-group --group-name re:Invent --description 're:Invent demo' --vpc-id "$vpc_id" -–debug botocore.endpoint - DEBUG - Making request for OperationModel(name=CreateSecurityGroup) (verify_ssl=True) with params: { 'body': { 'Action': 'CreateSecurityGroup', 'GroupName': 're:Invent’, 'Version': '2016-09-15’, 'VpcId': 'vpc-43cb382a', ‘GroupDescription': 're:Invent demo’ }, ‘url':'https://ec2.us-east-2.amazonaws.com/’, ... }
Parse command Format response Validate/Serialize parameters HTTP request/response Parse response $ aws ec2 create-security-group --group-name re:Invent --description 're:Invent demo' --vpc-id "$vpc_id" -–debug botocore.vendored.requests.packages.urllib3 .connectionpool - DEBUG - "POST / HTTP/1.1" 200 None
Parse command Format response Validate/Serialize parameters HTTP request/response Parse response $ aws ec2 create-security-group --group-name re:Invent --description 're:Invent demo' --vpc-id "$vpc_id" -–debug botocore.vendored.requests.packages.urllib3 .connectionpool - DEBUG - "POST / HTTP/1.1" 200 None
Parse command Format response Validate/Serialize parameters HTTP request/response Parse response $ aws ec2 create-security-group --group-name re:Invent --description 're:Invent demo' --vpc-id "$vpc_id" -–debug botocore.parsers - DEBUG - Response body: <?xml version="1.0" encoding="UTF-8"?> <CreateSecurityGroupResponse> <requestId>request-id</requestId> <return>true</return> <groupId>sg-5ec80f37</groupId> </CreateSecurityGroupResponse>
Parse command Format response Validate/Serialize parameters HTTP request/response Parse response $ aws ec2 create-security-group --group-name re:Invent --description 're:Invent demo' --vpc-id "$vpc_id" -–debug botocore.parsers - DEBUG - Response body: <?xml version="1.0" encoding="UTF-8"?> <CreateSecurityGroupResponse> <requestId>request-id</requestId> <return>true</return> <groupId>sg-5ec80f37</groupId> </CreateSecurityGroupResponse>
Parse command Format response Validate/Serialize parameters HTTP request/response Parse response $ aws ec2 create-security-group --group-name re:Invent --description 're:Invent demo' --vpc-id "$vpc_id" -–debug botocore.parsers - DEBUG - Response body: <?xml version="1.0" encoding="UTF-8"?> <CreateSecurityGroupResponse> <requestId>request-id</requestId> <return>true</return> <groupId>sg-5ec80f37</groupId> </CreateSecurityGroupResponse>
Parse command Format response Validate/Serialize parameters HTTP request/response Parse response $ aws ec2 create-security-group --group-name re:Invent --description 're:Invent demo' --vpc-id "$vpc_id" -–debug botocore.retryhandler - DEBUG - No retry needed. botocore.parsers - DEBUG - Response body: <?xml version="1.0" encoding="UTF-8"?> <CreateSecurityGroupResponse> <requestId>request-id</requestId> <return>true</return> <groupId>sg-5ec80f37</groupId> </CreateSecurityGroupResponse>
Parse command Format response Validate/Serialize parameters HTTP request/response Parse response $ aws ec2 create-security-group --group-name re:Invent --description 're:Invent demo' --vpc-id "$vpc_id" -–debug botocore.retryhandler - DEBUG - No retry needed. botocore.parsers - DEBUG - Response body: <?xml version="1.0" encoding="UTF-8"?> <CreateSecurityGroupResponse> <requestId>request-id</requestId> <return>true</return> <groupId>sg-5ec80f37</groupId> </CreateSecurityGroupResponse>
VPC 10.0.0.0/16 Subnet 10.0.0.0/24 AWS Cloud Internet gateway router Destination Target 10.0.0.0/16 local 0.0.0.0/0 igw-id
VPC 10.0.0.0/16 Subnet 10.0.0.0/24 security group AWS Cloud Internet gateway router Destination Target 10.0.0.0/16 local 0.0.0.0/0 igw-id
Demo $
Important Takeaways • --debug • Search for Traceback’s • Leverage knowledge of AWS CLI architecture
The Effective AWS CLI User Tenets The effective AWS CLI user:  Is resourceful with tooling
• BASH: variables, pipes, functions
• BASH: variables, pipes, functions • UNIX commands: xargs, sort, tail
• BASH: variables, pipes, functions • UNIX commands: xargs, sort, tail • External tools: jp, jpterm, aws-shell
• BASH: variables, pipes, functions • UNIX commands: xargs, sort, tail • External tools: jp, jpterm, aws-shell • DIY (Do it yourself) tooling
alias
[alias] st = status ci = commit br = branch co = checkout ~/.gitconfig
[alias] st = status ci = commit br = branch co = checkout ~/.gitconfig $ git co some-branch
[alias] st = status ci = commit br = branch co = checkout ~/.gitconfig $ git co some-branch
[alias] st = status ci = commit br = branch co = checkout ~/.gitconfig $ git co some-branch $ git checkout some-branch
VPC 10.0.0.0/16 Subnet 10.0.0.0/24 security group AWS Cloud Internet gateway router Destination Target 10.0.0.0/16 local 0.0.0.0/0 igw-id
VPC 10.0.0.0/16 Subnet 10.0.0.0/24 security group AWS Cloud instance Internet gateway router Destination Target 10.0.0.0/16 local 0.0.0.0/0 igw-id role
Demo $
Important Takeaways • alias • Leverage BASH and external tooling
The Effective AWS CLI User Tenets The effective AWS CLI user:  Understands performance implications
Performance Implications  Client-side vs. server-side filtering  Pagination  aws s3 cp
Performance Implications  Client-side vs. server-side filtering
$ aws ec2 describe-images --query 'Images[?starts_with(to_string(Name),`amzn-ami-hvm-`)]'
$ aws ec2 describe-images --query 'Images[?starts_with(to_string(Name),`amzn-ami-hvm-`)]'
$ aws ec2 describe-images --query 'Images[?starts_with(to_string(Name),`amzn-ami-hvm-`)]'
$ aws ec2 describe-images --query 'Images[?starts_with(to_string(Name),`amzn-ami-hvm-`)]' $ aws ec2 describe-images --filters Name=name,Values='amzn-ami-hvm-*' --query 'Images'
$ aws ec2 describe-images --query 'Images[?starts_with(to_string(Name),`amzn-ami-hvm-`)]' $ aws ec2 describe-images --filters Name=name,Values='amzn-ami-hvm-*' --query 'Images'
$ aws ec2 describe-images --query 'Images[?starts_with(to_string(Name),`amzn-ami-hvm-`)]' $ aws ec2 describe-images --filters Name=name,Values='amzn-ami-hvm-*' --query 'Images'
Performance Implications  Pagination
$ aws ec2 describe-snapshots
$ aws ec2 describe-snapshots { "Snapshots”: [ { "SnapshotId": "snap-0", ... }, ... ] } }
$ aws ec2 describe-snapshots { "Snapshots”: [ { "SnapshotId": "snap-0", ... }, ... ] } } { "Snapshots”: [ { "SnapshotId": "snap-1000", ... }, ... ] } }
$ aws ec2 describe-snapshots { "Snapshots”: [ { "SnapshotId": "snap-0", ... }, ... ] } } { "Snapshots”: [ { "SnapshotId": "snap-1000", ... }, ... ] } } { "Snapshots”: [ { "SnapshotId": "snap-2000", ... }, ... ] } }
$ aws ec2 describe-snapshots { "Snapshots”: [ { "SnapshotId": "snap-0", ... }, ... ] } } { "Snapshots”: [ { "SnapshotId": "snap-1000", ... }, ... ] } } { "Snapshots”: [ { "SnapshotId": "snap-2000", ... }, ... ] } } { "Snapshots”: [ { "SnapshotId": "snap-0", ... }, ... { "SnapshotId": "snap-1000", ... }, ... { "SnapshotId": "snap-2000", ... }, ... ] } }
$ aws ec2 describe-snapshots --output text --query 'Snapshots[].SnapshotId' { "Snapshots”: [ { "SnapshotId": "snap-0", ... }, ... ] } } { "Snapshots”: [ { "SnapshotId": "snap-1000", ... }, ... ] } } { "Snapshots”: [ { "SnapshotId": "snap-2000", ... }, ... ] } }
$ aws ec2 describe-snapshots --output text --query 'Snapshots[].SnapshotId' { "Snapshots”: [ { "SnapshotId": "snap-0", ... }, ... ] } } { "Snapshots”: [ { "SnapshotId": "snap-1000", ... }, ... ] } } { "Snapshots”: [ { "SnapshotId": "snap-2000", ... }, ... ] } }
{ "Snapshots”: [ { "SnapshotId": "snap-0", ... }, ... ] } } { "Snapshots”: [ { "SnapshotId": "snap-1000", ... }, ... ] } } { "Snapshots”: [ { "SnapshotId": "snap-2000", ... }, ... ] } } $ aws ec2 describe-snapshots --output text --query 'Snapshots[].SnapshotId' snap-0 ... snap-1000 ... snap-2000 ... }
Performance Implications  aws s3 cp
upload Thread pool Thread 1 Thread 2 Thread 3
upload Thread pool 6 5 4 3 2 1 Thread 1 Thread 2 Thread 3 Parts
upload Thread pool 6 5 4 3 2 1 Thread 1 Thread 2 Thread 3 Parts
upload Thread pool 6 5 3 4 1 Thread 1 Thread 2 Thread 3 Parts
upload Thread pool 6 5 4 1 Thread 1 Thread 2 Thread 3 Parts
upload Thread pool 5 4 6 Thread 1 Thread 2 Thread 3
upload Thread pool Thread 1 Thread 2 Thread 3
[default] region = us-east-1 ~/.aws/config
[default] region = us-east-1 s3 = max_concurrent_requests = 20 multipart_chunksize = 16MB multipart_threshold = 64MB max_queue_size = 10000 ~/.aws/config
[default] region = us-east-1 s3 = max_concurrent_requests = 20 multipart_chunksize = 16MB multipart_threshold = 64MB max_queue_size = 10000 ~/.aws/config
max_concurrent_requests Thread pool 6 5 4 3 2 1 Thread 1 Thread 2 Thread 3 Parts
VPC 10.0.0.0/16 Subnet 10.0.0.0/24 security group AWS Cloud instance Internet gateway router Destination Target 10.0.0.0/16 local 0.0.0.0/0 igw-id role
VPC 10.0.0.0/16 Subnet 10.0.0.0/24 security group AWS cloud instance Internet gateway router Amazon S3 Destination Target 10.0.0.0/16 local 0.0.0.0/0 igw-id role
Demo $
Important Takeaways • Use server-side filtering when possible • Pagination with --output text • s3 configuration options
The Effective AWS CLI User Tenets The effective AWS CLI user:
The Effective AWS CLI User Tenets The effective AWS CLI user:  Uses an iterative workflow  Example: --generate-cli-skeleton output
The Effective AWS CLI User Tenets The effective AWS CLI user:  Troubleshoots well  Example: --debug
The Effective AWS CLI User Tenets The effective AWS CLI user:  Is resourceful with tooling  Example: alias
The Effective AWS CLI User Tenets The effective AWS CLI user:  Understands performance implications  Example: s3 configurations
Remember to complete your evaluations!
Thank you!

More Related Content

AWS re:Invent 2016: The Effective AWS CLI User (DEV402)

  • 1. © 2016, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Kyle Knapp, Amazon Web Services December 1, 2016 DEV402 The Effective AWS CLI User
  • 3. Background Material • AWS CLI User Guide • AWS CLI Command Reference • 2015 AWS CLI re:Invent talk • 2014 AWS CLI re:Invent talk • 2013 AWS CLI re:Invent talk
  • 4. Background Material • AWS CLI User Guide • AWS CLI Command Reference • 2015 AWS CLI re:Invent talk • 2014 AWS CLI re:Invent talk • 2013 AWS CLI re:Invent talk Slides will be online!
  • 5. Unified tool to manage AWS services AWS Command Line Interface
  • 6. The Effective AWS CLI User Tenets The effective AWS CLI user:
  • 7. The Effective AWS CLI User Tenets The effective AWS CLI user:  Uses an iterative workflow
  • 8. The Effective AWS CLI User Tenets The effective AWS CLI user:  Uses an iterative workflow  Troubleshoots well
  • 9. The Effective AWS CLI User Tenets The effective AWS CLI user:  Uses an iterative workflow  Troubleshoots well  Is resourceful with tooling
  • 10. The Effective AWS CLI User Tenets The effective AWS CLI user:  Uses an iterative workflow  Troubleshoots well  Is resourceful with tooling  Understands performance implications
  • 14. Amazon VPC Amazon EC2 IAM
  • 15. Amazon VPC Amazon EC2 IAM Amazon S3
  • 23. The Effective AWS CLI User Tenets The effective AWS CLI user:  Uses an iterative workflow
  • 24. $ aws ec2 create-vpc --cidr-block 10.0.0.0/16
  • 25. $ aws ec2 create-vpc --cidr-block 10.0.0.0/16 { "Vpc": { "VpcId": "vpc-24de2d4d", "InstanceTenancy": "default", "State": "pending", "DhcpOptionsId": "dopt-f8ca2291", "CidrBlock": "10.0.0.0/16", "IsDefault": false } }
  • 26. $ aws ec2 create-vpc --cidr-block 10.0.0.0/16 { "Vpc": { "VpcId": "vpc-24de2d4d", "InstanceTenancy": "default", "State": "pending", "DhcpOptionsId": "dopt-f8ca2291", "CidrBlock": "10.0.0.0/16", "IsDefault": false } }
  • 27. $ aws ec2 create-vpc --cidr-block 10.0.0.0/16 { "Vpc": { "VpcId": "vpc-24de2d4d", "InstanceTenancy": "default", "State": "pending", "DhcpOptionsId": "dopt-f8ca2291", "CidrBlock": "10.0.0.0/16", "IsDefault": false } } $ vpc_id=vpc-24de2d4d
  • 28. $ aws ec2 create-vpc --cidr-block 10.0.0.0/16 { "Vpc": { "VpcId": "vpc-24de2d4d", "InstanceTenancy": "default", "State": "pending", "DhcpOptionsId": "dopt-f8ca2291", "CidrBlock": "10.0.0.0/16", "IsDefault": false } } $ vpc_id=vpc-24de2d4d We can do better…
  • 29. $ vpc_id=$(aws ec2 create-vpc --cidr-block 10.0.0.0/16 --query Vpc.VpcId --output text)
  • 30. $ vpc_id=$(aws ec2 create-vpc --cidr-block 10.0.0.0/16 --query Vpc.VpcId --output text) { "Vpc": { "VpcId": "vpc-24de2d4d", "InstanceTenancy": "default", "State": "pending", "DhcpOptionsId": "dopt-f8ca2291", "CidrBlock": "10.0.0.0/16", "IsDefault": false } }
  • 31. $ vpc_id=$(aws ec2 create-vpc --cidr-block 10.0.0.0/16 --query Vpc.VpcId --output text) { "Vpc": { "VpcId": "vpc-24de2d4d", "InstanceTenancy": "default", "State": "pending", "DhcpOptionsId": "dopt-f8ca2291", "CidrBlock": "10.0.0.0/16", "IsDefault": false } }
  • 32. $ vpc_id=$(aws ec2 create-vpc --cidr-block 10.0.0.0/16 --query Vpc.VpcId --output text) { "Vpc": { "VpcId": "vpc-24de2d4d", "InstanceTenancy": "default", "State": "pending", "DhcpOptionsId": "dopt-f8ca2291", "CidrBlock": "10.0.0.0/16", "IsDefault": false } }
  • 33. $ vpc_id=$(aws ec2 create-vpc --cidr-block 10.0.0.0/16 --query Vpc.VpcId --output text) { "Vpc": { "VpcId": "vpc-24de2d4d", "InstanceTenancy": "default", "State": "pending", "DhcpOptionsId": "dopt-f8ca2291", "CidrBlock": "10.0.0.0/16", "IsDefault": false } }
  • 34. $ vpc_id=$(aws ec2 create-vpc --cidr-block 10.0.0.0/16 --query Vpc.VpcId --output text) { "Vpc": { "VpcId": "vpc-24de2d4d", "InstanceTenancy": "default", "State": "pending", "DhcpOptionsId": "dopt-f8ca2291", "CidrBlock": "10.0.0.0/16", "IsDefault": false } }
  • 35. $ vpc_id=$(aws ec2 create-vpc --cidr-block 10.0.0.0/16 --query Vpc.VpcId --output text) $ echo "$vpc_id” vpc-24de2d4d
  • 36. $ vpc_id=$(aws ec2 create-vpc --cidr-block 10.0.0.0/16 --query Vpc.VpcId --output text) $ echo "$vpc_id" vpc-24de2d4d $ create_vpc_output=$(aws ec2 create-vpc --cidr-block 10.0.0.0/16)
  • 37. $ vpc_id=$(aws ec2 create-vpc --cidr-block 10.0.0.0/16 --query Vpc.VpcId --output text) $ echo "$vpc_id" vpc-24de2d4d $ create_vpc_output=$(aws ec2 create-vpc --cidr-block 10.0.0.0/16) $ vpc_id=$(jp -u Vpc.VpcId <<< "$create_vpc_output“)
  • 38. $ vpc_id=$(aws ec2 create-vpc --cidr-block 10.0.0.0/16 --query Vpc.VpcId --output text) $ echo "$vpc_id" vpc-24de2d4d $ create_vpc_output=$(aws ec2 create-vpc --cidr-block 10.0.0.0/16) $ vpc_id=$(jp -u Vpc.VpcId <<< "$create_vpc_output") $ echo "$vpc_id" vpc-24de2d4d
  • 39. $ vpc_id=$(aws ec2 create-vpc --cidr-block 10.0.0.0/16 --query Vpc.VpcId --output text) $ echo "$vpc_id" vpc-24de2d4d $ create_vpc_output=$(aws ec2 create-vpc --cidr-block 10.0.0.0/16) $ vpc_id=$(jp -u Vpc.VpcId <<< "$create_vpc_output") $ echo "$vpc_id" vpc-24de2d4d And we can make this even better!
  • 43. $ aws ec2 create-vpc --cidr-block 10.0.0.0/16 --generate-cli-skeleton output { "Vpc": { "VpcId": "VpcId", "State": "State", "CidrBlock": "CidrBlock", "DhcpOptionsId": "DhcpOptionsId", "Tags": [ { "Key": "Key", "Value": "Value" } ], "InstanceTenancy": "InstanceTenancy", "IsDefault": true } }
  • 44. $ aws ec2 create-vpc --cidr-block 10.0.0.0/16 --generate-cli-skeleton output { "Vpc": { "VpcId": "VpcId", "State": "State", "CidrBlock": "CidrBlock", "DhcpOptionsId": "DhcpOptionsId", "Tags": [ { "Key": "Key", "Value": "Value" } ], "InstanceTenancy": "InstanceTenancy", "IsDefault": true } } "Vpc":{ "type":"structure", "members":{ "VpcId":{ "shape":"String" }, "State":{ "shape":"VpcState" }, "CidrBlock":{ "shape":"String" }, "DhcpOptionsId":{ "shape":"String" }, "Tags":{ "shape":"TagList" }, "InstanceTenancy":{ "shape":"Tenancy" }, "IsDefault":{ "shape":"Boolean" } } } botocore/botocore/data/ec2/2016-09-15/service-2.json
  • 45. $ aws ec2 create-vpc --cidr-block 10.0.0.0/16 --generate-cli-skeleton output { "Vpc": { "VpcId": "VpcId", "State": "State", "CidrBlock": "CidrBlock", "DhcpOptionsId": "DhcpOptionsId", "Tags": [ { "Key": "Key", "Value": "Value" } ], "InstanceTenancy": "InstanceTenancy", "IsDefault": true } } "Vpc":{ "type":"structure", "members":{ "VpcId":{ "shape":"String" }, "State":{ "shape":"VpcState" }, "CidrBlock":{ "shape":"String" }, "DhcpOptionsId":{ "shape":"String" }, "Tags":{ "shape":"TagList" }, "InstanceTenancy":{ "shape":"Tenancy" }, "IsDefault":{ "shape":"Boolean" } } } botocore/botocore/data/ec2/2016-09-15/service-2.json
  • 46. $ aws ec2 create-vpc --cidr-block 10.0.0.0/16 --generate-cli-skeleton output { "Vpc": { "VpcId": "VpcId", "State": "State", "CidrBlock": "CidrBlock", "DhcpOptionsId": "DhcpOptionsId", "Tags": [ { "Key": "Key", "Value": "Value" } ], "InstanceTenancy": "InstanceTenancy", "IsDefault": true } } "Vpc":{ "type":"structure", "members":{ "VpcId":{ "shape":"String" }, "State":{ "shape":"VpcState" }, "CidrBlock":{ "shape":"String" }, "DhcpOptionsId":{ "shape":"String" }, "Tags":{ "shape":"TagList" }, "InstanceTenancy":{ "shape":"Tenancy" }, "IsDefault":{ "shape":"Boolean" } } } botocore/botocore/data/ec2/2016-09-15/service-2.json
  • 47. $ aws ec2 create-vpc --cidr-block 10.0.0.0/16 --generate-cli-skeleton output { "Vpc": { "VpcId": "VpcId", "State": "State", "CidrBlock": "CidrBlock", "DhcpOptionsId": "DhcpOptionsId", "Tags": [ { "Key": "Key", "Value": "Value" } ], "InstanceTenancy": "InstanceTenancy", "IsDefault": true } } "Vpc":{ "type":"structure", "members":{ "VpcId":{ "shape":"String" }, "State":{ "shape":"VpcState" }, "CidrBlock":{ "shape":"String" }, "DhcpOptionsId":{ "shape":"String" }, "Tags":{ "shape":"TagList" }, "InstanceTenancy":{ "shape":"Tenancy" }, "IsDefault":{ "shape":"Boolean" } } } botocore/botocore/data/ec2/2016-09-15/service-2.json
  • 51. Important Takeaways • --generate-cli-skeleton output • Leverage UNIX commands (e.g. echo, history)
  • 52. The Effective AWS CLI User Tenets The effective AWS CLI user:  Troubleshoots well
  • 53. $ aws ec2 create-security-group --group-name re:Invent --vpc-id VpcId
  • 54. $ aws ec2 create-security-group --group-name re:Invent --vpc-id VpcId aws: error: argument --description is required
  • 55. $ aws ec2 create-security-group --group-name re:Invent --vpc-id VpcId aws: error: argument --description is required $ aws ec2 create-security-group --group-name re:Invent --description 're:Invent demo' --vpc-id VpcId
  • 56. $ aws ec2 create-security-group --group-name re:Invent --vpc-id VpcId aws: error: argument --description is required $ aws ec2 create-security-group --group-name re:Invent --description 're:Invent demo' --vpc-id VpcId An error occurred (InvalidVpcID.NotFound) when calling the CreateSecurityGroup operation: The vpc ID 'VpcId' does not exist
  • 58. $ aws ec2 create-security-group --group-name re:Invent --description 're:Invent demo' --vpc-id "$vpc_id” --debug
  • 59. $ aws ec2 create-security-group --group-name re:Invent --description 're:Invent demo' --vpc-id "$vpc_id” --debug
  • 60. $ aws ec2 create-security-group --group-name re:Invent --description 're:Invent demo' --vpc-id "$vpc_id" -–debug 2016-11-04 10:30:33,532 - MainThread - awscli.clidriver - DEBUG - CLI version: aws- cli/1.11.10 Python/2.7.10 Darwin/15.6.0 botocore/1.4.65 2016-11-04 10:30:33,533 - MainThread - awscli.clidriver - DEBUG - Arguments entered to CLI: ['ec2', 'create-security-group', '--group-name', 're:Invent', '--description', 're:Invent demo', '--vpc-id', 'vpc-43cb382a', '--debug'] 2016-11-04 10:30:33,533 - MainThread - botocore.hooks - DEBUG - Event session- initialized: calling handler <function add_scalar_parsers at 0x10eefc488> 2016-11-04 10:30:33,533 - MainThread - botocore.hooks - DEBUG - Event session- initialized: calling handler <function inject_assume_role_provider_cache at 0x10ecde578> 2016-11-04 10:30:33,533 - MainThread - botocore.credentials - DEBUG - Skipping environment variable credential check because profile name was explicitly set. 2016-11-04 10:30:33,537 - MainThread - botocore.loaders - DEBUG - Loading JSON file: /Users/kyleknap/GitHub/botocore/botocore/data/ec2/2016-09-15/service-2.json 2016-11-04 10:30:33,622 - MainThread - botocore.hooks - DEBUG - Event service-data- loaded.ec2: calling handler <function register_retries_for_service at 0x10e904500>
  • 61. 2016-11-04 10:30:33,623 - MainThread - botocore.handlers - DEBUG - Registering retry handlers for service: ec2 2016-11-04 10:30:33,628 - MainThread - botocore.hooks - DEBUG - Event building-command- table.ec2: calling handler <functools.partial object at 0x10ef02940> 2016-11-04 10:30:33,628 - MainThread - awscli.customizations.removals - DEBUG - Removing operation: import-instance 2016-11-04 10:30:33,628 - MainThread - awscli.customizations.removals - DEBUG - Removing operation: import-volume 2016-11-04 10:30:33,628 - MainThread - botocore.hooks - DEBUG - Event building-command- table.ec2: calling handler <function add_waiters at 0x10ef03578> 2016-11-04 10:30:33,631 - MainThread - botocore.loaders - DEBUG - Loading JSON file: /Users/kyleknap/GitHub/botocore/botocore/data/ec2/2016-09-15/waiters-2.json 2016-11-04 10:30:33,635 - MainThread - awscli.clidriver - DEBUG - OrderedDict([(u'dry- run', <awscli.arguments.BooleanArgument object at 0x10febe950>), (u'no-dry-run', <awscli.arguments.BooleanArgument object at 0x10febe990>), (u'group-name', <awscli.arguments.CLIArgument object at 0x10febe9d0>), (u'description', <awscli.arguments.CLIArgument object at 0x10febea10>), (u'vpc-id', <awscli.arguments.CLIArgument object at 0x10febea50>)])
  • 62. 2016-11-04 10:30:33,623 - MainThread - botocore.handlers - DEBUG - Registering retry handlers for service: ec2 2016-11-04 10:30:33,628 - MainThread - botocore.hooks - DEBUG - Event building-command- table.ec2: calling handler <functools.partial object at 0x10ef02940> 2016-11-04 10:30:33,628 - MainThread - awscli.customizations.removals - DEBUG - Removing operation: import-instance 2016-11-04 10:30:33,628 - MainThread - awscli.customizations.removals - DEBUG - Removing operation: import-volume 2016-11-04 10:30:33,628 - MainThread - botocore.hooks - DEBUG - Event building-command- table.ec2: calling handler <function add_waiters at 0x10ef03578> 2016-11-04 10:30:33,631 - MainThread - botocore.loaders - DEBUG - Loading JSON file: /Users/kyleknap/GitHub/botocore/botocore/data/ec2/2016-09-15/waiters-2.json 2016-11-04 10:30:33,635 - MainThread - awscli.clidriver - DEBUG - OrderedDict([(u'dry- run', <awscli.arguments.BooleanArgument object at 0x10febe950>), (u'no-dry-run', <awscli.arguments.BooleanArgument object at 0x10febe990>), (u'group-name', <awscli.arguments.CLIArgument object at 0x10febe9d0>), (u'description', <awscli.arguments.CLIArgument object at 0x10febea10>), (u'vpc-id', <awscli.arguments.CLIArgument object at 0x10febea50>)]) And this continues on for a while…
  • 63. Parse command $ aws ec2 create-security-group --group-name re:Invent --description 're:Invent demo' --vpc-id vpc-24de2d4d --query GroupId --output text
  • 64. Parse command botocore client call # Python code response = ec2_client.create_security_group( GroupName=‘re:Invent’, Description=‘re:Invent demo’, VpcId=‘vpc-24de2d4d’ ) $ aws ec2 create-security-group --group-name re:Invent --description 're:Invent demo' --vpc-id vpc-24de2d4d --query GroupId --output text
  • 65. Parse command botocore client call $ aws ec2 create-security-group --group-name re:Invent --description 're:Invent demo' --vpc-id vpc-24de2d4d --query GroupId --output text # Python code response = ec2_client.create_security_group( GroupName=‘re:Invent’, Description=‘re:Invent demo’, VpcId=‘vpc-24de2d4d’ )
  • 66. Parse command botocore client call $ aws ec2 create-security-group --group-name re:Invent --description 're:Invent demo' --vpc-id vpc-24de2d4d --query GroupId --output text # Python code response = ec2_client.create_security_group( GroupName=‘re:Invent’, Description=‘re:Invent demo’, VpcId=‘vpc-24de2d4d’ )
  • 67. Parse command Format response botocore client call sg-1d032e64 $ aws ec2 create-security-group --group-name re:Invent --description 're:Invent demo' --vpc-id vpc-24de2d4d --query GroupId --output text # Python code response = ec2_client.create_security_group( GroupName=‘re:Invent’, Description=‘re:Invent demo’, VpcId=‘vpc-24de2d4d’ )
  • 68. Parse command Format response botocore client call $ aws ec2 create-security-group --group-name re:Invent --description 're:Invent demo' --vpc-id vpc-24de2d4d --query GroupId --output text # Python code response = ec2_client.create_security_group( GroupName=‘re:Invent’, Description=‘re:Invent demo’, VpcId=‘vpc-24de2d4d’ ) sg-1d032e64
  • 69. Parse command Format response botocore client call $ aws ec2 create-security-group --group-name re:Invent --description 're:Invent demo' --vpc-id vpc-24de2d4d --query GroupId --output text # Python code response = ec2_client.create_security_group( GroupName=‘re:Invent’, Description=‘re:Invent demo’, VpcId=‘vpc-24de2d4d’ ) sg-1d032e64
  • 70. Parse command Format response botocore client call # Python code response = ec2_client.create_security_group( GroupName=‘re:Invent’, Description=‘re:Invent demo’, VpcId=‘vpc-24de2d4d’ )
  • 71. Parse command Format response botocore client call Validate/Serialize parameters "CreateSecurityGroupRequest”{ "type":"structure”, "required":["GroupName", "Description”], "members":{ "DryRun":{ "shape":"Boolean” }, "GroupName":{ "shape":"String” }, "Description":{ "shape":"String” }, "VpcId":{ "shape":"String” } } } botocore/botocore/data/ec2/2016-09-15/service-2.json
  • 72. Parse command Format response Make botocore client call Validate/Serialize parameters HTTP request/response
  • 73. Parse command Format response Make botocore client call Validate/Serialize parameters HTTP request/response Parse response "CreateSecurityGroupResult”:{ "type":"structure”, "members":{ "GroupId":{ "shape":"String” } } } botocore/botocore/data/ec2/2016-09-15/service-2.json
  • 74. Parse command Format response Validate/Serialize parameters HTTP request/response Parse response --debug
  • 75. Parse command Format response Validate/Serialize parameters HTTP request/response Parse response $ aws ec2 create-security-group --group-name re:Invent --description 're:Invent demo’ --vpc-id "$vpc_id" -–debug
  • 76. Parse command Format response Validate/Serialize parameters HTTP request/response Parse response awscli.clidriver - DEBUG - Arguments entered to CLI: ['ec2', 'create-security- group', '--group-name', 're:Invent’, '--description', 're:Invent demo’, '--vpc-id', 'vpc-43cb382a', '--debug'] $ aws ec2 create-security-group --group-name re:Invent --description 're:Invent demo' --vpc-id "$vpc_id" -–debug
  • 77. Parse command Format response Validate/Serialize parameters HTTP request/response Parse response awscli.clidriver - DEBUG - Arguments entered to CLI: ['ec2', 'create-security- group', '--group-name', 're:Invent’, '--description', 're:Invent demo’, '--vpc-id', 'vpc-43cb382a', '--debug'] $ aws ec2 create-security-group --group-name re:Invent --description 're:Invent demo' --vpc-id "$vpc_id" -–debug
  • 78. Parse command Format response Validate/Serialize parameters HTTP request/response Parse response awscli.clidriver - DEBUG - Arguments entered to CLI: ['ec2', 'create-security- group', '--group-name', 're:Invent’, '--description', 're:Invent demo’, '--vpc-id', 'vpc-43cb382a', '--debug'] $ aws ec2 create-security-group --group-name re:Invent --description 're:Invent demo' --vpc-id "$vpc_id" -–debug
  • 79. Parse command Format response Validate/Serialize parameters HTTP request/response Parse response $ aws ec2 create-security-group --group-name re:Invent --description file://description.txt --vpc-id "$vpc_id" -–debug
  • 80. Parse command Format response Validate/Serialize parameters HTTP request/response Parse response $ aws ec2 create-security-group --group-name re:Invent --description file://description.txt --vpc-id "$vpc_id" -–debug
  • 81. Parse command Format response Validate/Serialize parameters HTTP request/response Parse response $ aws ec2 create-security-group --group-name re:Invent --description file://description.txt --vpc-id "$vpc_id" -–debug awscli.clidriver - DEBUG - Arguments entered to CLI: ['ec2', 'create-security- group', '--group-name', 're:Invent’, '--description', 'file://description.txt’, '--vpc-id', 'vpc-43cb382a', '--debug']
  • 82. Parse command Format response Validate/Serialize parameters HTTP request/response Parse response $ aws ec2 create-security-group --group-name re:Invent --description file://description.txt --vpc-id "$vpc_id" -–debug awscli.clidriver - DEBUG - Arguments entered to CLI: ['ec2', 'create-security- group', '--group-name', 're:Invent’, '--description', 'file://description.txt’, '--vpc-id', 'vpc-43cb382a', '--debug']
  • 83. Parse command Format response Validate/Serialize parameters HTTP request/response Parse response awscli.arguments - DEBUG - Unpacked value of u're:Invent demo' for parameter "description": u're:Invent demo' $ aws ec2 create-security-group --group-name re:Invent --description file://description.txt --vpc-id "$vpc_id" -–debug awscli.clidriver - DEBUG - Arguments entered to CLI: ['ec2', 'create-security- group', '--group-name', 're:Invent’, '--description', 'file://description.txt’, '--vpc-id', 'vpc-43cb382a', '--debug']
  • 84. Parse command Format response Validate/Serialize parameters HTTP request/response Parse response awscli.arguments - DEBUG - Unpacked value of u're:Invent demo' for parameter "description": u're:Invent demo' $ aws ec2 create-security-group --group-name re:Invent --description file://description.txt --vpc-id "$vpc_id" -–debug awscli.clidriver - DEBUG - Arguments entered to CLI: ['ec2', 'create-security- group', '--group-name', 're:Invent’, '--description', 'file://description.txt’, '--vpc-id', 'vpc-43cb382a', '--debug']
  • 85. Parse command Format response Validate/Serialize parameters HTTP request/response Parse response $ aws ec2 create-security-group --group-name re:Invent --description 're:Invent demo' --vpc-id "$vpc_id" -–debug
  • 86. Parse command Format response Validate/Serialize parameters HTTP request/response Parse response $ aws ec2 create-security-group --group-name re:Invent --description 're:Invent demo' --vpc-id "$vpc_id" -–debug botocore.endpoint - DEBUG - Making request for OperationModel(name=CreateSecurityGroup) (verify_ssl=True) with params: { 'body': { 'Action': 'CreateSecurityGroup', 'GroupName': 're:Invent’, 'Version': '2016-09-15’, 'VpcId': 'vpc-43cb382a', ‘GroupDescription': 're:Invent demo’ }, ‘url':'https://ec2.us-east-2.amazonaws.com/’, ... }
  • 87. Parse command Format response Validate/Serialize parameters HTTP request/response Parse response $ aws ec2 create-security-group --group-name re:Invent --description 're:Invent demo' --vpc-id "$vpc_id" -–debug botocore.endpoint - DEBUG - Making request for OperationModel(name=CreateSecurityGroup) (verify_ssl=True) with params: { 'body': { 'Action': 'CreateSecurityGroup', 'GroupName': 're:Invent’, 'Version': '2016-09-15’, 'VpcId': 'vpc-43cb382a', ‘GroupDescription': 're:Invent demo’ }, ‘url':'https://ec2.us-east-2.amazonaws.com/’, ... }
  • 88. Parse command Format response Validate/Serialize parameters HTTP request/response Parse response $ aws ec2 create-security-group --group-name re:Invent --description 're:Invent demo' --vpc-id "$vpc_id" -–debug botocore.endpoint - DEBUG - Making request for OperationModel(name=CreateSecurityGroup) (verify_ssl=True) with params: { 'body': { 'Action': 'CreateSecurityGroup', 'GroupName': 're:Invent’, 'Version': '2016-09-15’, 'VpcId': 'vpc-43cb382a', ‘GroupDescription': 're:Invent demo’ }, ‘url':'https://ec2.us-east-2.amazonaws.com/’, ... }
  • 89. Parse command Format response Validate/Serialize parameters HTTP request/response Parse response $ aws ec2 create-security-group --group-name re:Invent --description 're:Invent demo' --vpc-id "$vpc_id" -–debug botocore.endpoint - DEBUG - Making request for OperationModel(name=CreateSecurityGroup) (verify_ssl=True) with params: { 'body': { 'Action': 'CreateSecurityGroup', 'GroupName': 're:Invent’, 'Version': '2016-09-15’, 'VpcId': 'vpc-43cb382a', ‘GroupDescription': 're:Invent demo’ }, ‘url':'https://ec2.us-east-2.amazonaws.com/’, ... }
  • 90. Parse command Format response Validate/Serialize parameters HTTP request/response Parse response $ aws ec2 create-security-group --group-name re:Invent --description 're:Invent demo' --vpc-id "$vpc_id" -–debug botocore.vendored.requests.packages.urllib3 .connectionpool - DEBUG - "POST / HTTP/1.1" 200 None
  • 91. Parse command Format response Validate/Serialize parameters HTTP request/response Parse response $ aws ec2 create-security-group --group-name re:Invent --description 're:Invent demo' --vpc-id "$vpc_id" -–debug botocore.vendored.requests.packages.urllib3 .connectionpool - DEBUG - "POST / HTTP/1.1" 200 None
  • 92. Parse command Format response Validate/Serialize parameters HTTP request/response Parse response $ aws ec2 create-security-group --group-name re:Invent --description 're:Invent demo' --vpc-id "$vpc_id" -–debug botocore.parsers - DEBUG - Response body: <?xml version="1.0" encoding="UTF-8"?> <CreateSecurityGroupResponse> <requestId>request-id</requestId> <return>true</return> <groupId>sg-5ec80f37</groupId> </CreateSecurityGroupResponse>
  • 93. Parse command Format response Validate/Serialize parameters HTTP request/response Parse response $ aws ec2 create-security-group --group-name re:Invent --description 're:Invent demo' --vpc-id "$vpc_id" -–debug botocore.parsers - DEBUG - Response body: <?xml version="1.0" encoding="UTF-8"?> <CreateSecurityGroupResponse> <requestId>request-id</requestId> <return>true</return> <groupId>sg-5ec80f37</groupId> </CreateSecurityGroupResponse>
  • 94. Parse command Format response Validate/Serialize parameters HTTP request/response Parse response $ aws ec2 create-security-group --group-name re:Invent --description 're:Invent demo' --vpc-id "$vpc_id" -–debug botocore.parsers - DEBUG - Response body: <?xml version="1.0" encoding="UTF-8"?> <CreateSecurityGroupResponse> <requestId>request-id</requestId> <return>true</return> <groupId>sg-5ec80f37</groupId> </CreateSecurityGroupResponse>
  • 95. Parse command Format response Validate/Serialize parameters HTTP request/response Parse response $ aws ec2 create-security-group --group-name re:Invent --description 're:Invent demo' --vpc-id "$vpc_id" -–debug botocore.retryhandler - DEBUG - No retry needed. botocore.parsers - DEBUG - Response body: <?xml version="1.0" encoding="UTF-8"?> <CreateSecurityGroupResponse> <requestId>request-id</requestId> <return>true</return> <groupId>sg-5ec80f37</groupId> </CreateSecurityGroupResponse>
  • 96. Parse command Format response Validate/Serialize parameters HTTP request/response Parse response $ aws ec2 create-security-group --group-name re:Invent --description 're:Invent demo' --vpc-id "$vpc_id" -–debug botocore.retryhandler - DEBUG - No retry needed. botocore.parsers - DEBUG - Response body: <?xml version="1.0" encoding="UTF-8"?> <CreateSecurityGroupResponse> <requestId>request-id</requestId> <return>true</return> <groupId>sg-5ec80f37</groupId> </CreateSecurityGroupResponse>
  • 100. Important Takeaways • --debug • Search for Traceback’s • Leverage knowledge of AWS CLI architecture
  • 101. The Effective AWS CLI User Tenets The effective AWS CLI user:  Is resourceful with tooling
  • 102. • BASH: variables, pipes, functions
  • 103. • BASH: variables, pipes, functions • UNIX commands: xargs, sort, tail
  • 104. • BASH: variables, pipes, functions • UNIX commands: xargs, sort, tail • External tools: jp, jpterm, aws-shell
  • 105. • BASH: variables, pipes, functions • UNIX commands: xargs, sort, tail • External tools: jp, jpterm, aws-shell • DIY (Do it yourself) tooling
  • 106. alias
  • 107. [alias] st = status ci = commit br = branch co = checkout ~/.gitconfig
  • 108. [alias] st = status ci = commit br = branch co = checkout ~/.gitconfig $ git co some-branch
  • 109. [alias] st = status ci = commit br = branch co = checkout ~/.gitconfig $ git co some-branch
  • 110. [alias] st = status ci = commit br = branch co = checkout ~/.gitconfig $ git co some-branch $ git checkout some-branch
  • 113. Demo $
  • 114. Important Takeaways • alias • Leverage BASH and external tooling
  • 115. The Effective AWS CLI User Tenets The effective AWS CLI user:  Understands performance implications
  • 116. Performance Implications  Client-side vs. server-side filtering  Pagination  aws s3 cp
  • 117. Performance Implications  Client-side vs. server-side filtering
  • 118. $ aws ec2 describe-images --query 'Images[?starts_with(to_string(Name),`amzn-ami-hvm-`)]'
  • 119. $ aws ec2 describe-images --query 'Images[?starts_with(to_string(Name),`amzn-ami-hvm-`)]'
  • 120. $ aws ec2 describe-images --query 'Images[?starts_with(to_string(Name),`amzn-ami-hvm-`)]'
  • 121. $ aws ec2 describe-images --query 'Images[?starts_with(to_string(Name),`amzn-ami-hvm-`)]' $ aws ec2 describe-images --filters Name=name,Values='amzn-ami-hvm-*' --query 'Images'
  • 122. $ aws ec2 describe-images --query 'Images[?starts_with(to_string(Name),`amzn-ami-hvm-`)]' $ aws ec2 describe-images --filters Name=name,Values='amzn-ami-hvm-*' --query 'Images'
  • 123. $ aws ec2 describe-images --query 'Images[?starts_with(to_string(Name),`amzn-ami-hvm-`)]' $ aws ec2 describe-images --filters Name=name,Values='amzn-ami-hvm-*' --query 'Images'
  • 125. $ aws ec2 describe-snapshots
  • 126. $ aws ec2 describe-snapshots { "Snapshots”: [ { "SnapshotId": "snap-0", ... }, ... ] } }
  • 127. $ aws ec2 describe-snapshots { "Snapshots”: [ { "SnapshotId": "snap-0", ... }, ... ] } } { "Snapshots”: [ { "SnapshotId": "snap-1000", ... }, ... ] } }
  • 128. $ aws ec2 describe-snapshots { "Snapshots”: [ { "SnapshotId": "snap-0", ... }, ... ] } } { "Snapshots”: [ { "SnapshotId": "snap-1000", ... }, ... ] } } { "Snapshots”: [ { "SnapshotId": "snap-2000", ... }, ... ] } }
  • 129. $ aws ec2 describe-snapshots { "Snapshots”: [ { "SnapshotId": "snap-0", ... }, ... ] } } { "Snapshots”: [ { "SnapshotId": "snap-1000", ... }, ... ] } } { "Snapshots”: [ { "SnapshotId": "snap-2000", ... }, ... ] } } { "Snapshots”: [ { "SnapshotId": "snap-0", ... }, ... { "SnapshotId": "snap-1000", ... }, ... { "SnapshotId": "snap-2000", ... }, ... ] } }
  • 130. $ aws ec2 describe-snapshots --output text --query 'Snapshots[].SnapshotId' { "Snapshots”: [ { "SnapshotId": "snap-0", ... }, ... ] } } { "Snapshots”: [ { "SnapshotId": "snap-1000", ... }, ... ] } } { "Snapshots”: [ { "SnapshotId": "snap-2000", ... }, ... ] } }
  • 131. $ aws ec2 describe-snapshots --output text --query 'Snapshots[].SnapshotId' { "Snapshots”: [ { "SnapshotId": "snap-0", ... }, ... ] } } { "Snapshots”: [ { "SnapshotId": "snap-1000", ... }, ... ] } } { "Snapshots”: [ { "SnapshotId": "snap-2000", ... }, ... ] } }
  • 132. { "Snapshots”: [ { "SnapshotId": "snap-0", ... }, ... ] } } { "Snapshots”: [ { "SnapshotId": "snap-1000", ... }, ... ] } } { "Snapshots”: [ { "SnapshotId": "snap-2000", ... }, ... ] } } $ aws ec2 describe-snapshots --output text --query 'Snapshots[].SnapshotId' snap-0 ... snap-1000 ... snap-2000 ... }
  • 135. upload Thread pool 6 5 4 3 2 1 Thread 1 Thread 2 Thread 3 Parts
  • 136. upload Thread pool 6 5 4 3 2 1 Thread 1 Thread 2 Thread 3 Parts
  • 142. [default] region = us-east-1 s3 = max_concurrent_requests = 20 multipart_chunksize = 16MB multipart_threshold = 64MB max_queue_size = 10000 ~/.aws/config
  • 143. [default] region = us-east-1 s3 = max_concurrent_requests = 20 multipart_chunksize = 16MB multipart_threshold = 64MB max_queue_size = 10000 ~/.aws/config
  • 144. max_concurrent_requests Thread pool 6 5 4 3 2 1 Thread 1 Thread 2 Thread 3 Parts
  • 147. Demo $
  • 148. Important Takeaways • Use server-side filtering when possible • Pagination with --output text • s3 configuration options
  • 149. The Effective AWS CLI User Tenets The effective AWS CLI user:
  • 150. The Effective AWS CLI User Tenets The effective AWS CLI user:  Uses an iterative workflow  Example: --generate-cli-skeleton output
  • 151. The Effective AWS CLI User Tenets The effective AWS CLI user:  Troubleshoots well  Example: --debug
  • 152. The Effective AWS CLI User Tenets The effective AWS CLI user:  Is resourceful with tooling  Example: alias
  • 153. The Effective AWS CLI User Tenets The effective AWS CLI user:  Understands performance implications  Example: s3 configurations
  • 154. Remember to complete your evaluations!