Automating a build of an EC2 based developer environment

Feb 11, 2020

I love to cut some code at any opportunity, which isn’t a new thing for me, I’ve always been a developer at my core. Given that I’ve spent so many years in this space, it’s also not surprising to know that I’ve given plenty of thought over the years to the topic of developer productivity as well - and that was the theme of my thoughts that led me to the solution I built which is the focus of this blog post. I’ve previously written about some of the ways I had set up remote development environments for Visual Studio Code with Cloud9, and I’ve been using remote SSH hosts for development since then. I’m a massive fan of it for a lot of reasons, firstly it’s nice having a really big development instance to cut code on, so my project builds quickly, even if I’m running around with my Surface Go laptop which doesn’t have a lot of power. Secondly, it is really handy not having to have a highly privileged AWS credential on my laptop with static keys so that my CLI commands can work. Now there are ways to avoid this (the preview of the AWS CLI v2 is great for doing SSO and using temporary credentials for example) but typically speaking most developers would use an IAM user with programmatic access, and that credential would be stored in your .aws/credentials file. By using a remote EC2 host I get the benefit of an IAM Instance Profile so I don’t need to hard code any credentials anywhere. Lastly, I like that I can have different EC2 instances with different versions of software installed on them to suit individual projects I’m working on, and just connect to the appropriate one when it’s time to code, as opposed to having lots of versions of libraries and runtimes installed on my local laptop, and then seeing weird issues when versions clash. That last one though is what got me thinking though, in terms of “if I’m spinning up a lot of these environments, I should probably automate this so I can get coding quicker when I start a new project!”.

The goal for what I wanted then was to automate the creation of my developer environments. In my head, I had a few goals for this before I set out:

Not a bad little shopping list to start with - so it was time to get cracking. I started with a CDK project as this is my preferred way to create CloudFormation templates (I write them in NodeJS for those curious about my language preferences). This got the bare bones of what I needed ready to go pretty quickly, with the EC2 instance, the EFS volume and the IAM roles all coming together pretty quickly. The first challenge for me was around how to automate the installation of specific versions of runtimes and packages on my EC2 instance. Typically speaking the UserData script is the way you would do this, but I needed a way to set parameters in the CloudFormation script that would let me opt-out of some packages, and specify a version number of the package if I did want it, so this required a different approach. What I ended up doing was using a Lambda backed custom CloudFormation resource to help here. If a package had a version string of ‘none’ the lambda function returned an empty string for me, if it was anything else it spits out the appropriate script to add to the user data for that package. The scripts themselves are passed as parameters to the custom resource for a couple of reasons. The first one was that it made it easy for me to break them up into individual chunks for each runtime, as opposed to one large UserData script with lots of conditional statements in it. The second was that my script pushed the lambda function past the 4096 characters allowed for an inline lambda function, which meant I would need to deal with zipping the function to publish it, and I would no longer just have one nice portable CloudFormation template file. That wasn’t desirable so the breaking things up approach felt better.

Next was the interesting challenge around automatically powering the instance on and off based on my SSH connections. Powering it off felt easier so I started there, I’m using a CloudWatch alarm to monitor the network in metic, working on the theory that while I’m connected via SSH in Visual Studio Code, my saving files and sending commands will create a level of ‘network in’ to the instance. The alarm couldn’t be set to 0 bytes of input because of the SSM agent, which likes to chat in very small amounts, so I set a threshold of 1 million bytes in an hour. I’m still testing this as I code to make sure it works, but it felt like a good starting point.

To enable the powering on of the instance though, I couldn’t rely on network metrics as while it’s not running, there isn’t any. After doing some homework, I came up with the idea to inject the call to ‘start-instances’ into the ProxyCommand attribute in my SSH config file. If you take the standard command that session manager recommends, you simply modify it to look like this:

    ProxyCommand C:\Windows\System32\WindowsPowerShell\v1.0\powershell.exe "aws ec2 start-instances --instance-ids %h;aws ssm start-session --target %h --document-name AWS-StartSSHSession --parameters portNumber=%p"

Now when Visual Studio Code starts the SSH connection, it also powers on the instance for me. In my testing, I’ve found that I will usually see an error about not being able to connect to the instance in the first attempt. This error happens because the Systems Manager endpoint responds to the SSH connection, and then provides an immediate error based on the instance not being available in SSM. I simply wait a few moments before hitting retry, and things happen normally from there!

At this point, I’ve ticked off all of my goals for automating the creation of my environments and life is good. Since this felt like something that would be useful to others, I decided to throw the code up on my GitHub repo (along with a little more detail in the documentation) so that if this is something that sounds interesting to you, you can help yourself to the code. If you found it useful feel free to leave me a comment here. If you have suggestions for improvements or have issues with it, use the issues tab in GitHub so I can track them and we can discuss them there too!

 

Comments

You're signed in as | Sign out

Submitting comment...

There are no comments on this post yet. Be the first to leave one!