1 – Converting to Gitlab CI

How to convert my static website from a shared VM hosted solution to AWS S3 with AWS CloudFront!

Series introduction

Welcome to the first installment of our series on migrating a static website from a shared VM to AWS.

Converting Current Setup to Gitlab CI

In this article, we will dive into the process of transitioning my current setup to utilize Gitlab CI, setting the foundation for a more streamlined and automated workflow. By the end of this article, you will have a clear understanding of how I migrated from manual deployment to an automated CI/CD pipeline that integrates seamlessly with the existing VM solution. And, hopefully, you can use this for your own static websites!

The Old Situation

The static website is currently hosted on a shared VM, presenting certain challenges that we aim to overcome. This VM does not offer the luxury of SSH connections, limiting us to using FTP for deployment. Moreover, our local development environment is based on Windows OS with Ruby installed, which enables us to run Jekyll locally for development and testing purposes, but this is not ideal. To facilitate the deployment process, we have been using buddy.works, a CI/CD solution that involves building our Jekyll site and copying the output to the VM using FTP. Our VM setup includes two key directories: httpdocs housing our primary site, keet.me, and staging.keet.me for our staging environment with an identically named URL.

Embracing Automation with Gitlab CI

In order to modernize our workflow and pave the way for more advanced deployments, we are moving towards an automated CI/CD approach, starting with Gitlab CI. Our strategy revolves around trunk-based development, characterized by short-lived feature branches and a central main branch serving as the base for both the staging and production environments.

We dive a bit more into Trunk-Based Development, and how we take it into account, down below.

Designing the CI/CD Pipeline

Our new CI/CD pipeline design comprises two fundamental Steps: build and deploy. The build Step is set to run automatically for the main branch and Merge Requests, ensuring that code changes are continuously validated.

The deploy Step, on the other hand, follows distinct rules. After a Merge Request (MR) is merged into the main branch, an automatic deployment to the staging environment is triggered. However, to deploy changes to the production environment, a manual trigger is required. This trigger is initiated when a “tag” is created on the main branch, with tags adhering to the semantic versioning (semver) convention.

Staging and production pipelines

In the image you can see two executed pipelines. The bottom one was run for staging, you can see it executed on the main branch, for commit faac2942. The top pipeline was built automatically, however deployed manually. The pop-up in the image is due to there previously being a ‘blocked’ pipeline icon, which was ‘played’ to deploy the new version to the production environment. The top pipeline was created because of a tag being (manually) created on the same faac2942 commit, namely 0.1.1.

Considerations for a Seamless Transition

As we transition to Gitlab CI, several considerations are vital to ensure a seamless and successful migration:

Trunk-Based Development (TBD): Embrace the principles of TBD for agility and efficient collaboration.
Pipeline Rules: Clearly define the rules for automatic builds and deployments to maintain consistency.
Semver Tagging: Create tags on the “main” branch following the semantic versioning standard to signify releases accurately.
Gitlab Deployments: Generate a “release” in Gitlab Deployments for each deployment, enabling efficient tracking and documentation of changes.

Trunk-Based Development

In case you are not familiar with Trunk Based Development, you can find a great primer on it here (also the source of the below image), below you also see a visual representation.

Trunk-based development

Nobody really agrees on what is best for releasing to production, as there are a few options, though two stick out:

Create a release branch and tag it
Tag the main branch

Instead of having a discussions on the merits of either, we are taking the route of applying a tag on the main branch.

To sum up what we have in our design thus far:

Feature branches, for development
main branch, for the staging environment
tagged commit, for the production environment

Pipeline rules

Now that we know how the code will be generated and flow through the branching structure, we can apply rules of what can deploy when, and to where. This is also where we start creating a Gitlab CI file .gitlab-ci.yml in our project.

Let us start the file and start adding to it in a few steps for the initial environment setup and the rules of what to do when. The website is built using Jekyll, which provides an image on Docker Hub which you can use to build your own project. Our initial .gitlab-ci.yml file thus starts of like this:

image: jekyll/jekyll:4.2.2

We also have 2 Steps we need to execute:

build
deploy

The deploy Step needs output from the build Step, namely the website files. So, we add two stages to the configuration and add a dependency to the deploy Step, by configuring a needs rule. In order to be able to see it all working, I have included a script in both Steps below which you can safely execute, if you’re working along.

image: jekyll/jekyll:4.2.2

stages:
  - build
  - deploy
  
build:
  stage: build
  script:
    - echo "This is the build Step"

deploy:
  stage: deploy
  needs:
    - job: build
  script:
    - echo "This is the deploy Step"

If you are working along and commit the above to your repository, when viewing an executed Pipeline, you can select the tab “Job dependencies” and turn on “Show dependencies” option. The result should display the Pipeline configuration like shown below.

Trunk-based development

Lastly, rules. It would be unwise to simply run the whole thing every time. From the chapter above, we know we have three distinct situations:

Feature branches, for development
main branch, for the staging environment
tagged main branch, for the production environment

We do not want to deploy feature branches, at all. However, we do want to build the project to see if the build works, prior to merging new code, or content, into the main branch. This means that we need to execute the build Step when there is a Merge Request, and when there is a change on a Merge Request. (For all possible events, refer to the Gitlab CI Pipeline Merge Request documentation) Because this applies to everything in the pipeline configuration, we need to make this a “workflow” rule. The Gitlab CI workflow section controls when a pipeline is created (Full documentation). This translates to the following configuration.

workflow:
    rules:
        - if: $CI_PIPELINE_SOURCE == "merge_request_event"

The above has an implicit when: on_success attached to it, that is the default. The above workflow.rules will ensure that entire workflow is kicked off for every Merge Request event. Bear in mind, we did just state that we do not want to deploy a branch. Neither do we want to deploy the contents of a Merge Request, it might not be ready for deployment.

That means we need to prevent the deploy Step from executing. To prevent the Step deploy from executing, extend it with the following:

deploy:
    # previously created stuff
    rules:
        # Never deploy an MR
        - if: $CI_PIPELINE_SOURCE == "merge_request_event"
          when: never

Next, the main branch, after something is merged, needs to deploy to the staging environment. To avoid conflicts between environments of those working on the website, keeping the staging environment up to date is paramount. That means that there should not be a manual Step requirement. Summing up, after the main branch is updated, the staging environment needs to be automatically updated. Add another rule to deploy.rules to make this happen:

deploy:
    # previously created stuff
    rules:
        # MR rule above
        # Automatic deployment to staging
        - if: $CI_COMMIT_BRANCH == "main"

Lastly, we do not want the production environment to be updated automatically. We want to be in full control over when a deployment to production happens, and later we will also want to have a changelog to accommodate releases. For this, we need manual deployments to the production environment, but only when a commit is tagged. Expand the deploy.rules even further with:

deploy:
    rules:
      # Manual deployment to production
      - if: $CI_COMMIT_TAG =~ $SEMVER_REGEX
        when: manual

You might have noticed we used three different variables to compare against. Refer to the Gitlab documentation for all available predefined variables, but a quick explanation of the ones used:

$CI_PIPELINE_SOURCE is used to figure out the source of the pipeline. In our case, if it equals merge_request_event it concerns a Merge Request (whether a new one or an updated one), and we need to trigger the pipeline workflow
$CI_COMMIT_BRANCH is to compare against branch names. In our case, for the staging environment, we will only have a single branch: main.
$CI_COMMIT_TAG using this tag (available only when a tag is added, so not in a regular pipeline for an MR) causes the pipeline to be triggered if the tag complies with value of $SEMVER_REGEX

As the name implies, $SEMVER_REGEX contains a regex. The regex you can find on semver.org by following this link. To use it in your Gitlab CI configuration, create the following at the top of the .gitlab-ci.yml file:

variables:
  # MAKE SURE YOU CHECK THE LINK ABOVE IN CASE THIS REGEX IS UPDATED
    SEMVER_REGEX: /^(?P<major>0|[1-9]\d*)\.(?P<minor>0|[1-9]\d*)\.(?P<patch>0|[1-9]\d*)(?:-(?P<prerelease>(?:0|[1-9]\d*|\d*[a-zA-Z-][0-9a-zA-Z-]*)(?:\.(?:0|[1-9]\d*|\d*[a-zA-Z-][0-9a-zA-Z-]*))*))?(?:\+(?P<buildmetadata>[0-9a-zA-Z-]+(?:\.[0-9a-zA-Z-]+)*))?$/

A pipeline for production is now triggered only if a tag is created that follows the rules of Semantic Versioning (semver), for example 1.0.0 or 0.2.4.

With this, we have all the rules in place for our pipeline. If you have been working along, your .gitlab-ci.yml file should now look like this:

image: jekyll/jekyll:4.2.2

variables:
  SEMVER_REGEX: /^(?P<major>0|[1-9]\d*)\.(?P<minor>0|[1-9]\d*)\.(?P<patch>0|[1-9]\d*)(?:-(?P<prerelease>(?:0|[1-9]\d*|\d*[a-zA-Z-][0-9a-zA-Z-]*)(?:\.(?:0|[1-9]\d*|\d*[a-zA-Z-][0-9a-zA-Z-]*))*))?(?:\+(?P<buildmetadata>[0-9a-zA-Z-]+(?:\.[0-9a-zA-Z-]+)*))?$/

workflow:
  rules:
    - if: $CI_PIPELINE_SOURCE == "merge_request_event"

stages:
  - build
  - deploy
  
build:
  stage: build
  script:
    - echo "This is the build Step"

deploy:
  stage: deploy
  needs:
    - job: build
  script:
    - echo "This is the deploy Step"
  rules:
    # Never deploy an MR
    - if: $CI_PIPELINE_SOURCE == "merge_request_event"
      when: never
    # Automatic deployment to staging
    - if: $CI_COMMIT_BRANCH == "main"
    # Manual deployment to production
    - if: $CI_COMMIT_TAG =~ $SEMVER_REGEX
      when: manual

Building

The Steps for Gitlab CI we created so far are great, but still empty. They only echo a statement each, which is not useful. The deploy Step needs the files to deploy from the build Step. Let us build those. If you are working along, this is where you might want to start deviating, as some of the following items and code are specific for this website.

For unknown reason, the theme used by the website is not available when calling a bundle exec jekyll build command, so we need to install this manually. For that, we need to make sure that the image we are using (jekyll:jekyll) has the language ruby available for us to use. Update the script to install Jekyll and the website theme, then ‘build’ the website.

build:
  stage: build
  script:
    - echo "This is the build Step"
    # Install the Ruby language
    - apk add ruby
    # Install the "bundle" package, "jekyll" and the Jekyll theme we are using
    - gem install bundle jekyll jekyll-theme-chirpy:5.3.2
    # Execute the "bundle install" to install project dependencies
    - bundle install
    # Run a file ownership correction - I have had problems with weird ownership, your mileage may vary
    - chown -R jekyll:jekyll /srv/jekyll
    # Execute "build" to generate the website files. The output goes to "_site" (created if it does not exist)
    - bundle exec jekyll build

Note my last comment in the YAML above: The output goes to "_site" (created if it does not exist). This is important, as any content, changes, configurations, et cetera, are lost after the Step completes. Meaning the files of the ‘built’ website are not available for the deploy Step. To make them available we need to add them to the Pipeline Artifacts. Extend the build Step with the following configuration.

build:
    # The above configuration
    artifacts:
        expire_in: 1 month
        paths:
            - _site

This additional configuration makes it so that the output of _site is added to the build Artifacts. The expire_in ensures that the contents get deleted after 1 month. If you run a build with the above configuration (by creating or updating a Merge Request) you can browse to, and through, the Artifact(s) by going to your Gitlab Project, then follow %project% > Pipelines > #%pipeline% > build, on this screen select Browse in the right column. You should see something akin to the below image.

Browse Artifact archive

Deploying

We now have files to deploy and rules of when to deploy. The only thing missing is the ability to deploy.

Things to know, the Jekyll image used is based on Alpine Linux. That means the image is very minimal when it comes to included software. As such, the Jekyll image, which we are also using as the pipeline base image, does not support FTP. In turn, that means we have to install it. Because this is not something we consider part of ‘normal operations’, we put this in the before_script section of a Gitlab CI Step configuration.

deploy:
    # What exists already
    before_script:
        # Update and upgrade Alpine, then install cURL. Upgrade necessary due to buy in a dependency.
        # Also installing "lftp", an FTP client to be used for file transfer for deployments
        - echo "start before_script"
        - apk update
        - apk upgrade
        - apk add lftp

Now, we are ready to deploy! Change the deploy.script section to the following:

deploy:
    script:
      - echo "Will deploy files to $HOST/$PROJECT_DIR/"
      # the line below is explained at the end of the section
      - lftp -e "set ssl:verify-certificate false; cd $PROJECT_DIR; ls; mirror -R _site/ .; ls; exit" -u $USER,$PASSWORD ftp://$HOST/

If you have been working along, here we suddenly have a bunch of new variables being used. These are variables that might change, such as a user, or maybe are dependent on to which environment we are going to deploy to, such as the $PROJECT_DIR variable. There is also $PASSWORD, a secret.

In case the value of these variables ever changes, for example to set a new password for the user to use, or maybe even fully replace the user with another, we do not want to update the configuration of .gitlab-ci.yml. Instead, if you declare them using the Gitlab GUI you can set these on your project, and they are automatically available when Gitlab CI is being executed. The location to set these in the GUI is Gitlab > %project% > Settings > CI/CD, then “expand” the Variables section. The variables for this project look like this:

CI/CD variables

Note: in the screenshot the blank squares are removed prefixes. Just ignore them :)

Notice how some variables have staging or production listed under “Environments”? These are only available when running a Pipeline for those environments. This ties in with something from all the way in the beginning of this article: we are differentiating between directories on the VM host.

From the screenshot above you can clearly see there is only one login, so both staging and production are updated using the same credentials. I know, this is bad practice, but this is not an environment I have set up, maintain, update, or control. I can only modify the content. This bad security practice is part of the reasons to have this series about a move to AWS.

Back to deploying. As there is a difference in where we are deploying to, we have two options:

Create separate Steps for deployment, such as deploy-staging and deploy-production
Use a variable to distinguish which environment we are targeting

The first option we can easily use with Workflow Rules. However, it would mean a copy/paste of the configuration code, as there is literally no difference between staging and production, excepting the directory in which the files are hosted.

So, option 2, use a variable to distinguish what we are targeting. Let us first set a default value for this new variable. Do this by adding the following line to the variables section, where also the variable SEMVER_REGEX is declared:

variables:
  GITLAB_ENV: staging

The default value is staging. This is intentionally not production, so that in case we are messing around with Gitlab CI configurations, we do not accidentally destroy the production environment (still possible on VM setup such as this, but besides the point).

Next, we could determine in %step%.rules sections which environment we are targeting. However, it is much nicer to do this globally. Update the workflow.rules to the following:

workflow:
    rules:
        - if: $CI_PIPELINE_SOURCE == "merge_request_event"
          variables:
              GITLAB_ENV: "merge request"
        - if: $CI_COMMIT_BRANCH == "main"
          variables:
              GITLAB_ENV: "staging"
        - if: $CI_COMMIT_TAG =~ $SEMVER_REGEX
          variables:
              GITLAB_ENV: "production"

Using the workflow.rules for this is much nicer than determining it per Step, as this sets the variable only once. Technically you do not need to set it for the staging environment, as we already have that as a default value in the variables section.

Lastly, having set this variable we still do not get the variable values we declared in the GUI CI/CD variables. For that, we also need to set the Step to target an environment. Add the following line to the deploy Step:

deploy:
    # Other deploy things
    environment: $GITLAB_ENV

Now a deployment can be fully executed. However, allow me to explain the following command, which copies the files of the website from the pipeline to the directory on the VM.

      - lftp -e "set ssl:verify-certificate false; cd $PROJECT_DIR; ls; mirror -R _site/ .; ls; exit" -u $USER,$PASSWORD ftp://$HOST/

Earlier, we installed lftp to have software capable of using the FTP protocol. The -e flag allows for the execution of a string of commands, the section in "". The -u flag is for user and password, and at the end of the command the FTP protocol URL is located. The commands executed on the VM machine, in order are:

set ssl:verify-certificate false
cd $PROJECT_DIR
ls
mirror -R _site/ .
ls
exit

Due to my hosting providers’ setup, I need to disable verification of SSL certificates. Without disabling that, any commands are simply ignored. After disabling SSL certificate verification, the pipeline navigates into the $PROJECT_DIR. The value of this variable is stored in the GUI of Gitlab CI/CD, and it is dependent on the targeted environment, which is set using $GITLAB_ENV.

We then mirror the files recursively, which copies from source to target. The source we created as an artifact earlier, namely the _site directory in the build Step. The target directory we just navigated into, that makes our current target ..

Before and after the mirror command I have included an ls command, which is to list the directories’ contents. Listing it in this way, you have an overview of the before and after mirror state in the output of your pipeline Job.

At the end there is exit, to close the FTP connection. The lftp documentation (and StackOverflow Q&A’s) state that instead of lftp -e it should be possible to use lftp -c which should automatically exit when commands have run. However, this only gave me grief, and this (temporary) workaround works just fine.

Releasing

We are done with building and deploying to both staging and production. The one thing missing is registering when a deployment has happened. To accommodate that, we need to create a “Gitlab Release”. Via the GUI you can find the deployments of a project under Gitlab > %project% > Deploy > Releases.

Here you will find the full documentation for Gitlab Releases. A Release contains a few items, handy for developers and stakeholders. Namely, an installation package of the project so anyone can run it locally, but also a changelog on what was released. A changelog needs to be created to add in the description of a Release, for this first step of converting to Gitlab CI this is out of scope.

According to the documentation, to create a Release, you need to use the “release-cli” command. Again, we are using the Jekyll image, so this software is not installed. We need this command only in the deploy Step, so let us update the deploy.before_script configuration to install this. Note: the apk add line (#9) is updated to now also install curl.

deploy:
    # Other deploy stuff
    before_script:
        # Update and upgrade Alpine, then install cURL. Upgrade necessary due to buy in a dependency.
        # Also installing "lftp", an FTP client to be used for file transfer for deployments
        - echo "start before_script"
        - apk update
        - apk upgrade
        - apk add curl lftp
        # Install Release-CLI: https://docs.gitlab.com/ee/user/project/releases/release_cli.html#install-on-unixlinux
        - curl --location --output /usr/local/bin/release-cli "https://gitlab.com/api/v4/projects/gitlab-org%2Frelease-cli/packages/generic/release-cli/latest/release-cli-linux-amd64"
        - chmod +x /usr/local/bin/release-cli
        - release-cli -v

With that done, we can now trigger the creation of a Release. Because Gitlab supports this (near) natively (excepting the CLI tool we just installed), we have a new Release made by adding a deploy.release section, like below.

deploy:
    # Other deploy stuff
    release:
        name: "Release $GITLAB_ENV $RELEASE_TAG"
        description: "$RELEASE_TAG"
        tag_name: '$RELEASE_TAG'
        ref: '$RELEASE_TAG'

Another variable we need to declare! Let us quickly make this one, it is the last hurdle. In the variables section, declare the variable and set the value to the predefined Gitlab variable $CI_COMMIT_SHORT_SHA. As a pipeline is always triggered on a commit (whether for an MR, a branch, or a tag), the value of this predefined variable contains the first 8 characters of the commit SHA.

variables: 
    # Other variables
    RELEASE_TAG: $CI_COMMIT_SHORT_SHA

This does present a (small) problem though. If we have released to staging, a Release exists with a “short commit sha” tag, for example a1b2c3d4. When we try to release to production, an attempt will be made to create another Release with that SHA, causing the Job (Step) of the Pipeline to fail on the attempt to create a duplicate Release.

For the production environment we earlier configured that it will trigger a Pipeline for a deployment on the creation of a tag. Let us use the tag! In the workflow.rules section, specifically the section for production, overwrite the value of $RELEASE_TAG with the tag name/value. Update that rule to the following.

workflow:
    rules:
        # Other rules
        - if: $CI_COMMIT_TAG =~ $SEMVER_REGEX
          variables:
              GITLAB_ENV: "production"
              RELEASE_TAG: $CI_COMMIT_TAG

Full Gitlab CI Configuration

In case you encountered an unexplained exception, or maybe made a copy/paste mistake, here is the full configuration:

image: jekyll/jekyll:4.2.2

variables:
    GITLAB_ENV: staging
    RELEASE_TAG: $CI_COMMIT_SHORT_SHA
    SEMVER_REGEX: /^(?P<major>0|[1-9]\d*)\.(?P<minor>0|[1-9]\d*)\.(?P<patch>0|[1-9]\d*)(?:-(?P<prerelease>(?:0|[1-9]\d*|\d*[a-zA-Z-][0-9a-zA-Z-]*)(?:\.(?:0|[1-9]\d*|\d*[a-zA-Z-][0-9a-zA-Z-]*))*))?(?:\+(?P<buildmetadata>[0-9a-zA-Z-]+(?:\.[0-9a-zA-Z-]+)*))?$/

workflow:
    rules:
        - if: $CI_PIPELINE_SOURCE == "merge_request_event"
          variables:
              GITLAB_ENV: "merge request"
        - if: $CI_COMMIT_BRANCH == "main"
          variables:
              GITLAB_ENV: "staging"
        - if: $CI_COMMIT_TAG =~ $SEMVER_REGEX
          variables:
              GITLAB_ENV: "production"
              RELEASE_TAG: $CI_COMMIT_TAG

stages:
    - build
    - deploy

build:
    stage: build
    script:
        - echo "This is the build Step"
        # Install the Ruby language
        - apk add ruby
        # Install the "bundle" package, "jekyll" and the Jekyll theme we are using
        - gem install bundle jekyll jekyll-theme-chirpy:5.3.2
        # Execute the "bundle install" to install project dependencies
        - bundle install
        # Run a file ownership correction - I have had problems with weird ownership, your mileage may vary
        - chown -R jekyll:jekyll /srv/jekyll
        # Execute "build" to generate the website files. The output goes to "_site" (created if it does not exist)
        - bundle exec jekyll build
    artifacts:
        expire_in: 1 month
        paths:
            - _site

deploy:
    stage: deploy
    environment: $GITLAB_ENV
    needs:
        - job: build
    before_script:
        # Update and upgrade Alpine, then install cURL. Upgrade necessary due to buy in a dependency.
        # Also installing "lftp", an FTP client to be used for file transfer for deployments
        - echo "start before_script"
        - apk update
        - apk upgrade
        - apk add curl lftp
        # Install Release-CLI: https://docs.gitlab.com/ee/user/project/releases/release_cli.html#install-on-unixlinux
        - curl --location --output /usr/local/bin/release-cli "https://gitlab.com/api/v4/projects/gitlab-org%2Frelease-cli/packages/generic/release-cli/latest/release-cli-linux-amd64"
        - chmod +x /usr/local/bin/release-cli
        - release-cli -v
    script:
        - echo "Will deploy files to $HOST/$PROJECT_DIR/"
        - lftp -e "set ssl:verify-certificate false; cd $PROJECT_DIR; ls; mirror -R _site/ .; ls; exit" -u $USER,$PASSWORD ftp://$HOST/
    rules:
        # Never deploy an MR
        - if: $CI_PIPELINE_SOURCE == "merge_request_event"
          when: never
        # Automatic deployment to staging
        - if: $CI_COMMIT_BRANCH == "main"
        # Manual deployment to production
        - if: $CI_COMMIT_TAG =~ $SEMVER_REGEX
          when: manual
    release:
        name: "Release $GITLAB_ENV $RELEASE_TAG"
        description: "$RELEASE_TAG"
        tag_name: '$RELEASE_TAG'
        ref: '$RELEASE_TAG'

Note: If you use the above, you need to provide the variables via the GUI (Settings > CI/CD > "expand" variables) that are missing in the script.

Wrapping up

A lot was done for this conversion, we have:

Designed a CI/CD pipeline that matched the old situation
- The design took into account Trunk-Based Development (TBD)
- Worked out when (not) to deploy Pipeline results
We created a Gitlab CI file to configure the running (when/how) of Pipelines
- We create Workflow rules of when to create a Pipeline
- We created Rules in the Deploy Step for when to deploy to an Environment
- We configured Gitlab CI/CD Variables in the GUI for specific Environments
- Each deployment to an environment generates a Release package, including installable files and a changelog (to be created).

All this put together allows for what we see below: a Gitlab Release for version 0.2.0 created on a commit (90f41c51), containing the installable files and the changelog. As the code to generate a proper changelog is yet to be created we only have the version in the Release description.

Production release

Conclusion

In this article, we have laid the groundwork for migrating our static website from a VM solution to an AWS solution by changing the deployment process to Gitlab CI. By shifting to an automated pipeline and adhering to Trunk-Based Development principles, we are setting the stage for smoother deployments and enhanced collaboration.

Stay tuned for more exciting developments as we continue our migration to the cloud!

Handy links

Check out all articles in the series: ‘Migrating to AWS’