CloudFormation templates, when written with portability in mind, allow you to write one template and deploy it multiple times in many different environments. This is made possible in part by CloudFormation’s Pseudo Parameters. These parameters inject values like AWS Account ID, the current region, or CloudFormation stack ID during template deployment.
Wouldn’t it be great if you could utilize similar functionality in bash/PowerShell scripts on your EC2 instances? Well, you’re in luck. Thanks to EC2 Instance Metadata, you can!
What is EC2 Instance Metadata?
From the documentation, “Instance metadata is data about your instance that you can use to configure or manage the running instance.” This includes information like the instance ID, public and private IP addresses, current IAM role, security groups, account ID, region, and more! All this information can easily be gathered by invoking a REST API local and is unique to the instance itself.
How can I access it?
The API can be accessed using tools that send GET requests (e.g., curl on *nix-based systems, or Invoke-RestMethod in PowerShell on Windows) to an address. The latest API version can be invoked by sending a GET request to the following URI:
http://169.254.169.254/latest/
While this is the main URI for the API, there are three principal sections we can gather data from. There are dynamic/, meta-data/, and user-data/ (if applicable). Each one of these response items has items that we can query within it.
Dynamic: Retrieves information about the instance identity documents. This includes information like the current account ID, region, etc., as well as authenticity certificates to verify that the information contained in the documents is indeed valid (see more here).
Meta-data: Retrieves information about the instance. This includes information about the instance type, various information relating to networking, the current AMI, the IAM role associated with the instance, and spot instance information (if you’re utilizing spot instances).
User-data: Retrieves the UserData script that was passed to the EC2 instance on launch. This section will not appear if you did not launch your instance with any UserData script.
If you query for an attribute that does not exist, you will receive a “404 – Not found” response. A request of general metadata resources (ending in a /) returns a list of available resources within that section of the metadata. For example, if we were to send a request to “http://169.254.169.254/latest/meta-data/”, we’d receive a response with a list of attributes that we can query.
Getting Data
To gather information from instance metadata, you’ll need to have the URI path set to the resource you want information from. You can visit the official documentation for a complete list of instance metadata categories, but here are some examples of requests for certain metadata. Outputs of the requests are bolded.
Public IPv4 Address (if the instance is allocated an Elastic IP)
Bash:
$ curl http://169.254.169.254/latest/meta-data/public-ipv4
34.XXX.XXX.XXX
PowerShell:
PS > Invoke-RestMethod http://169.254.169.254/latest/meta-data/public-ipv4
54.XXX.XXX.XXX
IAM Instance Profile
Bash:
$ curl http://169.254.169.254/latest/meta-data/iam/info
{ "Code" : "Success", "LastUpdated" : "2018-08-15T23:19:38Z", "InstanceProfileArn" : "arn:aws:iam::111122223333:instance-profile/MyInstanceProfile", "InstanceProfileId" : " AIPA1234567890ABCDEFG " }
PowerShell:
PS > Invoke-RestMethod http://169.254.169.254/latest/meta-data/iam/info
Code LastUpdated InstanceProfileArn InstanceProfileId ---- ----------- ------------------ ----------------- Success 2018-08-15T23:30:10Z arn:aws:iam::111122223333:... AIPA1234567890ABCDEFG
In PowerShell, Invoke-RestMethod returns the JSON object and turns it into a native PowerShell object. If you were to store this data in a variable, you could reference attributes. For example: $myvariable.InstanceProfileArn would return the ARN of the instance profile.
Current Region and Account
Bash:
$ curl -s http://169.254.169.254/latest/dynamic/instance-identity/document | jq -r .region
us-west-2
$ curl -s http://169.254.169.254/latest/dynamic/instance-identity/document | jq -r .accountId
111122223333
PowerShell:
PS > $(Invoke-RestMethod http://169.254.169.254/latest/dynamic/instance-identity/document).region
us-east-1
PS > $(Invoke-RestMethod http://169.254.169.254/latest/dynamic/instance-identity/document).accountId
111122223333
Querying the instance identity document returns a JSON snippet. In this example, we parse the JSON with different methods based on our scripting language. In Bash, we can pipe the output to Python or jq to select the value we want from the returned JSON object. With PowerShell, the Invoke-RestMethod returns a native PSObject from the JSON in which we can select certain attributes to output the desired value.
Using Metadata in Scripts!
Now that we know how to access metadata, let’s implement it in scripts! You’ll need to know how to store the output of commands as variables in your scripting language of choice. Both Bash and PowerShell use similar syntax to get this done. Here’s an example of the syntax:
VARIABLE=$(some_command_with_output)
How the data is stored depends on the scripting language you’re using (e.g., Bash most often stores output as strings, PowerShell could store output as a string or an object). Below is an example of how to store the current AWS region as a variable in Bash or PowerShell:
Bash:
$ REGION=$(curl -s http://169.254.169.254/latest/dynamic/instance-identity/document | jq -r .region)
$ echo $REGION
us-west-2
PowerShell:
PS > $region = $(Invoke-RestMethod http://169.254.169.254/latest/dynamic/instance-identity/document).region
PS > Write-Host $region
us-east-1
A Real-World Example
Below is a simplified version of a script that helped a customer tag root volumes of their Linux instances created via CloudFormation (at the time of this writing, CloudFormation does not allow you to tag root volumes within a template). This script is used as part of the instance UserData script and the instance has a role assigned with the appropriate permissions to describe instances and tag volumes. Above, I’ve shown how to use jq to select JSON attributes. In this script, we will be utilizing Python (which comes pre-installed on Amazon Linux) to parse JSON data.
This one-line Python statement can be used to parse attributes from JSON objects returned from the API.
python -c "import sys, json; print(json.load(sys.stdin)['region'])"
Utilizing the sys package, we read in the piped JSON output. With json.load(), we can convert it to a native Python dictionary. We can then select the “region” attribute using regular Python dictionary syntax. This is all wrapped in a print() statement to make sure we output the attribute to be stored in our variable. In the script below, we use this style of JSON parsing for both meta data and AWS CLI responses.
Example Script:
# Gather info from instance metadata and AWS CLI INSTANCE_ID=$(curl -s http://169.254.169.254/latest/meta-data/instance-id) REGION=$(curl -s http://169.254.169.254/latest/dynamic/instance-identity/document | python -c "import sys, json; print(json.load(sys.stdin))['region']") INFO=$(aws ec2 describe-instances --instance-ids $INSTANCE_ID --region $REGION) # Get Volume ID and Mount Point ROOT_VOL_ID=$(echo $INFO | python -c "import sys, json; print(json.load(sys.stdin)['Reservations'][0]['Instances'][0]['BlockDeviceMappings'][0]['Ebs']['VolumeId'])" ) ROOT_VOL_MOUNT=$(echo $INFO | python -c "import sys, json; print(json.load(sys.stdin)['Reservations'][0]['Instances'][0]['BlockDeviceMappings'][0]['DeviceName'])" ) # Apply Tags aws ec2 create-tags --resources $ROOT_VOL_ID --tags Key=Name,Value="$INSTANCE_ID / $ROOT_VOL_MOUNT / root" --region $REGION
Wrapping Up
Instance metadata holds a variety of useful information about your EC2 instances. This can be queried using a REST API directly from the instance. When used in scripts, instance metadata allows you to easily write one script that can be run in a dynamic set of environments. This helps your scripts and infrastructure to be more portable—just like CloudFormation templates!
If you are interested in learning more about how 1Strategy can help optimize your AWS cloud journey and infrastructure, please contact us for more information at info@1Strategy.com.
