automating maven releases with circleci

Automating Maven Releases with CircleCI

Maven's probably the only all-in-one build tool I've ever really appreciated. I'll probably come to like make eventually and cement my status as old-before-her-time *nix crone, but I haven't had a reason to really dig into it yet so Maven it is. And I'm back at a mostly-Java shop, so let's have some fun!

This week's goal: automating releases from our CircleCI instance. Sounds simple enough, right? Bump the version, cut a tag, publish. How hard could it be?

Well, first off, we're using git-flow, or at least we're preserving master for releases and working off a separate verify branch. Budget git-flow, if you will. That's one complication, since the release has to be tagged on master but verify also needs to be updated so the two don't diverge.

If you're familiar with Maven you may already have guessed the second complication. It's trickier. Maven doesn't work in nice, straightforward semver: Maven accepts several different versioning schemes and has a special SNAPSHOT qualifier for non-release builds. If you're working towards a 1.0 release, your version number is 1.0-SNAPSHOT. After you cut the release, you resume development with 1.1-SNAPSHOT (or 2.0-SNAPSHOT if it really needs a rework already). And so on. It's not meant to be automated, because releases are a big deal in the Maven world and you're expected to have a plan for what you're going to do next instead of reacting to whether you fixed bugs, introduced features, or broke compatibility. And honestly, there are some compelling arguments for doing it this way.

I'm not going to go into them because I'm one half of the software team by myself and they're less applicable working on proprietary stuff at this scale. So let's get to automating!

Workflow

We're using Circle v2 and its workflow feature to organize the build. Every branch gets built: verify and master get deployed to Artifactory, while release triggers its own job, which latter is the linchpin of the whole structure.

workflows:
  version: 2
  build-and-deploy:
    jobs:
      - build
      - deploy:
          requires:
            - build
          filters:
            branches:
              only: /^(master|verify)$/
      - release:
          requires:
            - build
          filters:
            branches:
              only: /^release$/

Just Build

I'll be honest, I copied & pasted most of this job definition right out of the docs:

steps:
  - checkout
  - restore_cache:
      keys:
      - v1-dependencies-{{ checksum "pom.xml" }}
      # fallback to using the latest cache if no exact match is found
      - v1-dependencies-
  - run: mvn clean install
  - save_cache:
      paths:
        - ~/.m2
      key: v1-dependencies-{{ checksum "pom.xml" }}
  - persist_to_workspace:
      <<: *source

We're caching our dependencies because that's how one does it; mvn clean install is likely overkill (we probably don't need to bother with installing the dependency to the local Maven cache) but it builds and runs our tests and generates the artifact. The only really interesting part here is that we're persisting the important files to a workspace so we can recover it later -- *source refers to another YAML block with a root string and list of paths.

And Deploy

steps:
  - attach_workspace:
      at: .
  - run:
      name: Deploy to Artifactory
      command: mvn deploy

Here's where we use that workspace. Whenever this job runs, it'll reattach the file structure we saved from the build job. mvn deploy still runs all the intermediary lifecycle stages because that's how Maven rolls, but we don't need to check out the code again.

We've got our POMs set up with the artifactory-maven-plugin so all we have to do to publish is issue mvn deploy. That makes that easy, at least; there's the Artifactory CLI if you prefer, but Maven's whole deal is managing everything so as far as I'm concerned we should let it.

There's just one piece missing, though: how do we actually release a new version of the artifact and set up to begin on the next?

The Release Trigger

One of the ideas of git-flow is that when you're gearing up for a release, you cut a new branch that only contains work towards that release. This is great if you're working on multiple versions of the code simultaneously and releases can take awhile, so you might cherry-pick a bugfix from current development into a legacy release branch to ensure it doesn't affect a subset of your users. Since we're not a product company, we don't really have to worry about that. We're always working on the next release, and it drops when it's ready to drop.

This is going to get complicated. Here's the release build steps in full:

steps:
  - checkout
  - run:
      name: Cut new release
      command: |
        # assemble current and new version numbers
        OLD_VERSION=$(mvn -s .circleci/settings.xml -q \
          -Dexec.executable="echo" -Dexec.args='${project.version}' \
          --non-recursive org.codehaus.mojo:exec-maven-plugin:1.3.1:exec)
        NEW_VERSION="${OLD_VERSION/-SNAPSHOT/}"
        echo "Releasing $OLD_VERSION as $NEW_VERSION"

        # ensure dependencies use release versions
        mvn -s .circleci/settings.xml versions:use-releases

        # write release version to POM
        mvn -s .circleci/settings.xml versions:set -DnewVersion="$NEW_VERSION"

        # setup git
        git config user.name "Release Script"
        git config user.email "builds@understoryweather.com"

        # commit and tag
        git add pom.xml
        git commit -m "release: $NEW_VERSION"
        git tag "$NEW_VERSION"

        # land on master and publish
        git checkout master
        git merge --no-edit release
        git push origin master --tags

        # increment minor version number
        MAJ_VERSION=$(echo "$NEW_VERSION" | cut -d '.' -f 1)
        MIN_VERSION=$(echo "$NEW_VERSION" | cut -d '.' -f 2)
        NEW_MINOR=$(($MIN_VERSION + 1))
        DEV_VERSION="$MAJ_VERSION.$NEW_MINOR-SNAPSHOT"

        # ready development branch
        git checkout verify
        git merge --no-edit release
        mvn -s .circleci/settings.xml versions:set -DnewVersion="$DEV_VERSION"
        git add pom.xml
        git commit -m "ready for development: $DEV_VERSION"
        git push origin verify

        # clean up release branch
        git push origin :release

It's not messy, but that's... a lot of bash script. But just like any sufficiently complicated database task involves writing SQL, any sufficiently complicated ops task involves bash. Let's break it down:

Getting Version Numbers

# assemble current and new version numbers
OLD_VERSION=$(mvn -s .circleci/settings.xml -q \
  -Dexec.executable="echo" -Dexec.args='${project.version}' \
  --non-recursive org.codehaus.mojo:exec-maven-plugin:1.3.1:exec)
NEW_VERSION="${OLD_VERSION/-SNAPSHOT/}"
echo "Releasing $OLD_VERSION as $NEW_VERSION"

Note the -s .circleci/settings.xml: since Circle's just spinning up a basic OpenJDK image, we have a settings.xml checked into source control. Credentials are interpolated through environment variables, but it's still not great; at some point, I'll want to come back and create a custom Docker image to centralize our configuration.

Maven stores version numbers in the POM. We could pull them out with XPath, but since this is Maven, there's a plugin for that. The OLD_VERSION is the current value; since we're always releasing from the verify branch, this is guaranteed to be a snapshot version, and we need to strip that qualifier off to get NEW_VERSION for the release.

Update Versions

# ensure dependencies use release versions
mvn -s .circleci/settings.xml versions:use-releases

# write release version to POM
mvn -s .circleci/settings.xml versions:set -DnewVersion="$NEW_VERSION"

We don't have a ton of Java libraries, but there are enough that release management is (obviously) a concern. The first statement here makes sure that when we release, we aren't depending on a snapshot version of another of our libraries. The second actually sets the version field in the POM to the release version we generated just now.

You may be asking: why didn't I just alias mvn to mvn -s .circleci/settings.xml? And the answer is: I did, and spent half a day trying to figure out why it didn't work. I don't know if it's this particular image or Circle in general or what, but aliases are just ignored.

Release!

# setup git
git config user.name "Release Script"
git config user.email "builds@understoryweather.com"

# commit and tag
git add pom.xml
git commit -m "release: $NEW_VERSION"
git tag "$NEW_VERSION"

# land on master and publish
git checkout master
git merge --no-edit release
git push origin master --tags

Since we're going to be committing code, we need to do a little more git configuration to attribute the commits properly. This is another element I could streamline with a custom build image later on.

Next, we commit the updated POM and create a tag. When we merge (with --no-edit since the script can't change the commit message), the release commit and tag will land on the master branch. Then it's just a matter of pushing to the origin.

Next Up...

We've released, but we're not quite done. If we left it here, the next release from the verify branch would run into merge conflicts since master has an updated version in the POM. To prevent that, we have to merge back into verify. Preferably with a snapshot version qualifier, because Maven.

# increment minor version number
MAJ_VERSION=$(echo "$NEW_VERSION" | cut -d '.' -f 1)
MIN_VERSION=$(echo "$NEW_VERSION" | cut -d '.' -f 2)
NEW_MINOR=$(($MIN_VERSION + 1))
DEV_VERSION="$MAJ_VERSION.$NEW_MINOR-SNAPSHOT"

I switched us over to two-part version numbers strictly out of convenience. Since Maven expects you to know what you're working towards, going from 1.0 to 1.1 is a lot more realistic than trying to suss out whether you're looking at 1.0.1 or 1.1.0 next. We can always update the version ourselves if we decide the next release should actually be 2.0, but I'm trying to minimize human involvement here.

# ready development branch
git checkout verify
git merge --no-edit release
mvn -s .circleci/settings.xml versions:set -DnewVersion="$DEV_VERSION"
git add pom.xml
git commit -m "ready for development: $DEV_VERSION"
git push origin verify

Merging release into verify saves us from any potential merge conflicts down the line, since the same release commit now exists both on master and in verify. The script then adds a second commit to verify with the new snapshot version and sends it all up to the origin.

# clean up release branch
git push origin :release

Finally: when a trigger goes off, it resets. We don't want the release branch to hang around long-term. If we did, we'd have to push the release commit up to the origin to avoid merge conflicts in future, and doing that would kick off an infinite loop since the release job is watching this branch. So instead we just delete it from the origin, since it's done everything it needed to do.

Setting it Off

git checkout -b release
git push origin release

That's the payoff. Whenever we're ready to drop a new version, all that has to happen is a new branch named release. You can even do it through the GitHub UI if you're so inclined, in two clicks and seven letters. Once release builds and deletes itself, the ordinary build and deploy jobs take over on both updated master and verify branches. Within a few minutes we've got a release and the first snapshot towards the next landing in Artifactory!