Table Of Contents
- The Task At Hand
- Exploring The Environment
- Into The Rabbit-Hole : Setting Up The Pipeline
- Out Of The Frying Pan And Into The Fire: Creating Our Own Task
- The Berry On Top: Our Physical Feedback
- The Good, The Bad & … The Ugly?
7 weeks ago I started my internship at Ordina Mechelen. I had several project options available all looking to touch new and unknown tech that might be relevant for future operations.
My inner Judas spoke to me when I saw a listing about a project that would shoot toy rockets at developers if they broke a build, and it would provide a great opportunity to pull myself out of my comfort zone focusing more on Devops and Cloud platforms rather than pure programming. Sadly due to a global hardware shortage the toy rocket launcher was not available for delivery anymore, so I decided to use a raspberry pi to fetch the build logs and convert them into audio using google text to speech.
The Task At Hand
The project described a ci-cd pipeline that would trigger a raspberry pi once a build fails, then in response the pi would operate a toy rocket launcher unit that would target the developer responsible for breaking the build. As such the final project can be broken down into these steps:
- Create a sacrificial bare-bones spring boot project to put through the ringer
- Create a pipeline which performs some cookie cutter tasks
- Expand pipeline to setup AWS infrastructure
- Expand pipeline to provision said infrastructure
- Create a custom pipeline task to fetch and push build logs
- Configure raspberry pi to host a node.js server
- Listen for new log files
- If new log files have been found, search for error messages and convert those via text to speech(TTS)
- Broadcast the failures of your colleagues
Exploring The Environment
I’m going to dive right into the meat and potatoes of this project and write about the pipeline since the spring boot application does nothing more than display some basic html and runs a couple of unit tests. The devops environment that was used consisted of a couple of things.
A simple git instance on the Azure platform used for version control of our spring boot project.
The bread and butter of our operation. Using a data serialization language called yaml we are able to define each individual task we want applied to our code.
A place to store key value pairs that we can group and later reference in our yaml file.
Predefined connections to internal (Azure) or external services from which we can later extract credentials to gain access in the pipeline.
Into The Rabbit-Hole: Setting Up The Pipeline
In this section I will initially explain how to set up some basic tasks within our yaml file for building and testing. Next, I’ll move on to containerizing our build. Finally, I’ll explain how I have used IaC (Infrastructure as Code) to firstly spin up an RDS (AWS) instance based on Postgres and secondly use Helm to deploy the application to an EKS (AWS) cluster owned by JWorks.
Before We Dive In
Before we dive in, there are some nice to know things about the inner workings and structure of a pipeline defined in a yaml file. I’ll briefly go over some key definitions, so you can follow along when each individual task gets explained.
- Triggers :
- Triggers are (like the word implies) what starts a pipeline
- These can be changes in a branch like main or develop but could also be specific events happening in another pipeline such as failed jobs/stages or a specific variable value
- Variables :
- A hardcoded variable defined at the start of your pipeline
- A variable group containing secret values like tokens that can not be acquired using service connections
- Both of these options can be defined at the same time or individually and used anywhere in the pipeline including in additional task commands if the task allows it.
- Stages :
- Defines what steps your pipeline should take in what order.
- Encapsulation of stage sections
- Stage :
- A section of your pipeline
- Can be given a name of your choosing eg: Test,Docker,CloudSetup …
- Encapsulation for jobs
- Jobs :
- Encapsulation of job sections
- Job :
- Can be given a name of your choosing just like the stage section
- Can be given a pool attribute to define which agent OS has to run this job
- Encapsulation of one or more task sections
- Task :
- A pre-built or custom-made task to be performed on your pipeline run.
- Has a variety of attributes that can be manually filled up such as credentials or dictating your preferred working directories
- Agents :
- A machine hosted by the cloud provider (Azure in our case) that runs on a specific OS
- Defined in the pool attribute of a job or at the start of a pipeline.
- Mostly a clean slate with only some essential software pre-installed eg: Docker
- Gets wiped after use in order to accommodate the next job.
All of the above gets combined into a structure that resembles following code.
variables: - group: my-variable-group trigger: main pool: vmImage: ubuntu-latest stages: - stage: jobs: - job: myJob steps: - script: echo "Hello"
Following are the pipeline tasks used to test our java project, generate a test rapport and then build it using Maven.
To test our application we will be using JUnit because of the pre-existing support given by Azure. This will also generate a test rapport during each pipeline run based on the unit tests defined in our project.
A standard task using the Maven goal of ‘package’ which returns a JAR file that can later be used by Docker.
Copying And Creating An Artifact
In order to use our build across multiple agents we need to create an artifact out of the build. This artifact gets stored within the pipeline and can get called upon whenever we please. We do this by first copying our build to a directory on our agent that functions as the default staging directory for artifacts. By using a second task we take that build and publish it to our pipeline storage.
Here we use the artifact we created during our last job to create an image and push it to Dockerhub.
Fetching Our Artifact
First we have to fetch the artifact that we uploaded to our pipeline and place it in the appropriate directory on our new agent.
Creating An Image
Then we use said artifact together with a Dockerfile that was previously placed within our java project to create and upload an image to Dockerhub.
Setting Up A Cloud Database
In this job we will use Terraform to set up an RDS (AWS) database based on Postgres for our containerized java application. Our Terraform files are stored in the java project under a folder called infrastructure and can be called upon directly, alternatively a remote folder containing Terraform files can be specified if you want or need to split up your project files. Credentials needed to get access to AWS services come from a manually pre-defined service connection.
In order to use Terraform on our agent we have to first install it to our agent since it is not supplied by default.
This task performs several initialization steps in order to prepare the current working directory for use with Terraform.
Plans out what configurations and steps will be made once the apply command is given.
Excecuting our Terraform plan defined in the previous step.
Provisioning A Cloud Cluster
Now that we have an image of our app and place to store data to only one crucial step remains, launching our application. Our application gets deployed to EKS (Elastic Kubernetes Service) which is another AWS service designed for running cloud based Kubernetes. In order to do so a Helm chart has been made which like the Terraform files is stored under the infrastructure directory of our project. These charts are defined in a yaml format where specifications for Kubernetes are being made eg: Name of the app, Kind , Amount of replicas, Image to use …
Using our Helm chart and a service connection allowing us to deploy to the Jworks cluster, we deploy our application, which gets pulled from Dockerhub, to the “stage-thomas-more” namespace within EKS.
Giving A Signal
Now that everything has been set up and all the services are up and running it’s time to give our developers a heads-up. I did this using a Telegram bot that will broadcast a message for every build that has run. The bot token was stored in the library as a secret key-value pair.
Creating Our Bot
This is a prerequisite if you want to work with Telegram since a bot token and a chat id are required to function. Telegram has a neat tutorial on how to create your own bot using the “Botfather” which you can find here : The Botfather
Sending Out Notifications
Now that we have our bot token and a chat id we can define a message that gets sent everytime the task is reached.
A Visual Representation
Displayed below you will find two images representing the pipeline and the goals they accomplish on the Cloud.
For now pay no attention to the little logo displaying Eric Cartman, this is the image I used to represent my custom task which we will get to in the following section.
Out Of The Frying Pan And Into The Fire: Creating Our Own Task
During week 5 all of the above was learned, implemented and configured to a working state so a question was asked by Frederick Bousson the solutions lead at the Jworks Ordina Unit if it was possible to create a custom task for use in the pipeline.
Up for another challenge I stepped into the pretty unexplored (And not fully documented) lands of developing a task for a pipeline. The main objective of the task is to get the pipeline logs from a predefined pipeline run or from the current one as default. Those logs get filtered and send to a DynamoDB instance hosted on AWS.
- TFX : A packaging tool
- Microsoft VSS Web Extension SDK package
- A Visual Studio publisher account (free)
Getting The Logs
This was done using the Azure Devops REST API: Documentation
To get builds, a couple of things are required:
- The Azure Devops organization name
- The project name
- The build number
While the values for point one and two were pretty straightforward gaining access to the build logs required a build id instead of a build number , and since the build id can not be traced through the UI of Azure Devops some nested API requests were required.
The Authorization was gained through the creation of a PAT token that can be used as a header in the GET Request.
Filtering And Saving Failures
Now that we have our desired logs all that remains is filtering, formatting and sending those logs to DynamoDB. In order to complete this operation the following steps were taken:
- Set up authorization in a way that allows developers to use their AWS service connection.
- Use the AWS SDK combined with the credentials from the service connection to authorize the user.
- Filter the received logs which had a format of plain text using regex to find possible error messages
- Format the error messages together with the developer responsible for the build and the time of build.
- Push our formatted object to DynamoDB
Eh Voilà! Our extension does everything we hoped it would do, so it’s time implement it in our pipeline.
In my publisher account:
In the pipeline:
Want to check out the code? Take a look at the project repo : Github
The Berry On Top : Our Physical Feedback
Almost there! Finally, it is time for the actual dirty work namely snitching on our dear colleagues. To do this I only needed two things, a raspberry pi and a bluetooth speaker.
Dollar Store Google Assistant
Initially we had planned to use a toy rocket launcher as our physical feedback machine. That idea was scrapped however because of a global shortage (or tremendous price increase) for all hardware components.
Next the Google Assistant came to mind which is embedded in Android devices or a Google home speaker. The problem with this idea unfortunately was that the Google Assistant was never designed to take in text input through an API because devices running Google Assistant had no direct endpoints. Now we could in fact work around this and set up a Home automation system like Home Assistant or Node Red but this would mean that our speaker could never change location without reconfiguring access to its new network.
A final solution I came up with in secret while my mentor was away on a conference in Valencia, was to run a Node.js server on a Raspberry Pi that after booting takes a reference log from our DynamoDB database. Every minute it would get the latest log available and compare it to the log it had stored, this way when a difference in logs had been found it knew it was time to call out the developer.
Through a nifty npm package we are able to send our text message that we want converted to an audio file and get an url in return pointing to an audio file. Using Node file system I programmatically created a text file where each audio url alternated with a path to a local audio file gets written on a line in the previously created text file. What I had just created was a playlist which dictated the order of audio files to be played. All I had to do now was hook it up to a media player.
note: This not the ideal solution, but it allowed the Pi and speaker to be portable to any location as long as we could connect it to wi-fi.
A succesfull pipeline run broadcast(Sound up):
A failed pipeline run broadcast(Sound up):
The Good, The Bad & … The Ugly? : Summary
Like I mentioned at the start of this blog, there were a number of different project options to choose from as an internship topic. The reason I went with the ci-cd route is that I knew it would pull me out of my comfort zone and broaden my view on development in general.
While having touched on basic CLI environments like Docker or Bash it always felt cryptic to use a multitude of flags and if you messed up the error messages weren’t all that clear compared to an error with a web framework like React for example.
After 7 weeks of submerging myself mostly in configuration files like .yaml and .tf(Terraform) combined with CLI tools such as kubectl for Kubernetes, psql for Postgres and AWS CLI for AWS Configuration, I’m glad to say that i feel a lot more at home touching on tools not necessarily meant for pure programming.
For a total recap of tools and software used during the project, I put together the following image.
As a final sendoff I want to say a quick thank you to my mentor Nick Geudens for guiding me through the jungle of DevOps and AWS and to Frederick Bousson for coming up with the project and allowing me the opportunity to execute it.