No matter what you’re building, there is always an element of resource administration. The Cloud has done many amazing things over the years, but not all of us are completely serverless and many companies still have VMs to keep their systems running. When I’m in architecture discussions with clients I like to ask how they’re managing their EC2 instances. Many have great confidence in their use of a bastion host.
The traditional method for safely managing your EC2 instances was to create another EC2 instance that acted as your bastion host in a public subnet so you could access it over the internet. This is the AWS way, and many companies like GoToMyPC and LogMeIn offer services that solve this specific problem to great success, though you will pay for it on a monthly basis.
This process worked well for many years but not without its issues. For instance, how are you accessing your EC2 instances? If you’re using SSH keys, how many people in your office have the same SSH keys? Do you have one set for everyone, or a set for each admin? What happens when you need to change the keys? If an admin leaves the company, do you change the keys immediately? Are you keeping each EC2 private key on each instance updated as well? I sure hope you’re not storing your keys directly on your bastion host!
The traditional bastion approach to managing EC2 resources means you are bound to have a nightmare of managing SSH keys, ingress rules, and access for all of your resources, as well. For a small team this may not be an issue, but for a big team it can be a huge time suck. From a security perspective, you’re also leaving a lot up to chance if you’re not very closely managing your SSH access and widening your attack surface by keeping your EC2 bastion host running. Yes, using SSH agent forwarding and SSH config files are all best practices, but you still run the risk of those keys getting into the wrong hands.
Enter AWS Systems Manager.
About a year ago, AWS released a new service called AWS Systems Manager (SSM) that allows users to “…automate operational tasks across your AWS resources.” EC2 Systems Manager has historically been the place to complete many of these tasks, but it is now part of SSM. Where I see Systems Manager consistently used in our engagements is as a replacement for a bastion host. The best part: it’s a free service.
To summarize why you should care, here are a few things you’re spending too much time on that SSM can now take off your plate:
Windows Patching
Oh, the dreaded Windows Patch Tuesdays. Who loves waiting until after normal business hours to patch your EC2 Windows machines? NOBODY.
SSM has solved this for you by creating Maintenance Windows and Patch Groups. Maintenance Windows allow you to create a window of time on a regular cadence (say the first Tuesday of the month for Windows updates) to scan for OS updates and install them. You can organize machines into Patch Groups or use tags to target them based on environment, or select manually. Once updates are installed, the machine will reboot if necessary as many times as it needs.
Deploying Patches
Ideally, you’re installing OS updates on non-production machines first and vetting any compatibility issues before applying to production. In SSM, you can do this by requiring an approval of the patch baseline beforehand, so no patches will be installed on your production instances until they have been approved. Set your approval rules based on product, update classification (driver updates, security updates, etc.), and severity. If the update doesn’t match these rules, it will not be applied. This saves you the minutia of researching each OS update to understand if it’s critical to your system or not. You can even blacklist specific OS patches that you know are hazardous to your system by using the Rejected Patches section of the Patch Baseline wizard.
Storing and Versioning Common Shell Commands
Many teams have a collection of standard commands they run regularly. SSM provides a way to store commands in what’s called “command documents.” These documents are self-contained scripts that can be run from SSM on whatever target instances you choose. AWS has provided a huge list of common documents that you may find useful, such as AWS-RunPatchBaseline which will scan a machine for OS updates and install them for you. Create your own commands for your infrastructure, store them as a document, and tag them for even tighter control on who is allowed to run them.
Managing Admin Access
Instead of managing an array of SSH keys, you can now simply control who has access to SSM. You can also record user actions in SSM using CloudTrail, so you will always have a record of activity for your admins for audit purposes. Stop handing out SSH keys and control access to your admin activities like you do for everything else in AWS: using IAM.
You can still conduct shell sessions and use a command line interface to interact with your instances just as you would in the traditional sense. Session Manager provides a browser-based interactive shell that allows you to run all of the Linux terminal commands that your heart desires. You can also log your command history to S3 and/or CloudWatch.
Monitor Your Instances
By default, SSM will write logs to a file on the local machine. This is useful, but again, you would have to SSH or remote into the instance to see them. With SSM you can use the CloudWatch agent to record patching activity to S3 and set alarms in case things go wrong.
You can also manage all of your instances through the use of Systems Manager Inventory. This feature allows you to pull meta-data about all of your instances into a single dashboard that shows which OS version is on each machine, how many installs have occurred and when, etc.
Securely Store Passwords and Config Values
Use of SSM Parameters lets you store sensitive passwords in SSM and allows you to reference them from within a command document. This reduces the need to embed sensitive information into your scripts and makes updating them in the future much easier.
We hope you consider using Systems Manager to manage your virtual machines going forward. As of September 2018, AWS has some great features planned for this service, including:
- SSH Client – You will be able to create SSH sessions atop Session Manager without opening up any inbound ports.
- On-Premises Access – You’ll have the ability to access your on-premises instances (which must be running the SSM Agent) via Session Manager.
