Automatic Node Deploys to Elastic Beanstalk

One of my favorite good ideas to ignore is the maxim that you should have your deployment pipeline ready to go before you start writing code. There's always some wrinkle you couldn't have anticipated anyway, so while it sounds good on paper I just don't think it's the best possible use of time. But with anything sufficiently complicated, there's a point where you just have to buckle down and automate rather than waste time repeating the same steps yet again (or, worse, forgetting one). I hit that point recently: the application isn't in production yet, so I'd been "deploying" by means of pulling the repo on an EC2 server, installing dependencies and building in-place, then killing and restarting the node process with nohup. Good enough for demos, not sustainable long-term. Also, I might have in fact missed a step Friday before last and not realized things were mostly broken until the following Monday.

I'd been using CircleCI to build and test the application already, so I wanted to stick with it for deployment as well. However, this precluded using the same EC2 instance: the build container would need to connect to it to run commands over SSH, but this connection would be coming from any of a huge possible range of build container IP addresses. I didn't want to open the server up to the whole world to accommodate the build system. Eventually I settled on Elastic Beanstalk, which can be controlled through the AWS command-line interface with the proper credentials instead of the morass of VPCs and security groups. Just upload a zip file!

The cost of using EBS, it turned out, was that while it made difficult things easy it also made easy things difficult. How do you deploy the same application to different environments? You don't. Everything has to be in that zip file, and if that includes any per-environment configuration then the right config files had better be where they're expected to be. This is less than ideal, but at least it can be scripted. Here's the whole thing (assuming awscli has already been installed):

# what time is it?
TIMESTAMP=$(date +%Y%m%d%H%M%S)

# work around Elastic Beanstalk permissions for node-gyp (bcrypt)
echo "unsafe-perm=true" > .npmrc

# generate artifacts
npm run build

# download config
aws s3 cp s3://elasticbeanstalk-bucket-name/app/development.config.json .

# zip everything up
zip -r app-dev.zip . \
  --exclude "node_modules/*" ".git/*" "coverage/*" ".nyc_output/*" "test/*" ".circleci/*"

# upload to s3
aws s3 mv ./app-dev.zip s3://elasticbeanstalk-bucket-name/app/app-dev-$TIMESTAMP.zip

# create new version
aws elasticbeanstalk create-application-version --region us-west-2 \
  --application-name app --version-label development-$TIMESTAMP \
  --source-bundle S3Bucket=elasticbeanstalk-bucket-name,S3Key=app/app-dev-$TIMESTAMP.zip

# deploy to dev environment
# --application-name app is not specified because apt installs
# an older version of awscli which doesn't accept that option
aws elasticbeanstalk update-environment --region us-west-2 --environment-name app-dev \
  --version-label development-$TIMESTAMP

The TIMESTAMP ensures the build can be uniquely identified later. The .npmrc setting is for AWS reasons: as detailed in this StackOverflow answer, the unfortunately-acronymed node-gyp runs as the instance's ec2-user account and doesn't have permissions it needs to compile bcrypt. If you're not using bcrypt (or another project that involves a node-gyp step on install), you don't need that line.

The zip is assembled in three steps:

  1. npm build compiles stylesheets, dynamic Pug templates, frontend JavaScript, and so forth.
  2. The appropriate environment config is downloaded from an S3 bucket.
  3. Everything is rolled together in the zip file, minus the detritus of source control and test results.

Finally, the Elastic Beanstalk deploy happens in two stages:

  1. aws elasticbeanstalk create-application-version does what it sounds like: each timestamped zip file becomes a new "version". These don't map exactly to versions as more commonly understood thanks to the target environment configuration, so naming them for the target environment and giving the timestamp helps identify them.
  2. aws elasticbeanstalk update-environment actually deploys the newly-created "version" to the destination environment.

Obviously, when it comes time to roll the project out to production, I'll factor the environment out into a variable to download and upload the appropriate artifacts. But even in its current state, this one small script has almost made deployment continuous: every pushed commit gets deployed to Elastic Beanstalk with no manual intervention, unless there are database changes. That's next.