I talk with many of my fellow engineers at conferences and other events throughout the year. One thing I like demonstrating is how they can implement a continuous integration/continuous deployment (CI/CD) pipeline into a codebase with very little effort. In this post I will walk through some demo code and the CircleCI config that I use in the demonstration. Following these steps will show you how to implement CI/CD pipelines into your code base.
This post will cover:
A simple unit test for a Python Flask application
How to implement a CI/CD pipeline in the codebase using a CircleCI config file in the project
Building a Docker image
Pushing the Docker image to Docker Hub
Kicking off a deployment script which will run the application in Docker container on a Digital Ocean server
Prerequisites
Before we get started, you will need:
Project environment variables set up so that they specify your Docker Hub username and password in the CircleCI dashboard
SSH access to a cloud server. You can add a SSH key to your account using the CircleCI portal. For this post I will be using a Digital Ocean server, but you can use whichever server/cloud provider you prefer.
A deployment script on the host server to use for deploying this application. Here is an example deployment script deploy_app.sh
After you have all the pre-requisites complete you are ready to proceed to the next section.
Using the sample application
For this post I’ll be using a simple Python Flask and you can find the complete source code for this project here and git clone it locally. The app is a simple web server that renders html when a request is made to it. The Flask application lives in the hello_world.py file:
from flask import Flask
app = Flask(__name__)
def wrap_html(message):
html = """
<html>
<body>
<div style='font-size:120px;'>
<center>
<image height="200" width="800" src="https://infosiftr.com/wp-content/uploads/2018/01/unnamed-2.png">
<br>
{0}<br>
</center>
</div>
</body>
</html>""".format(message)
return html
@app.route('/')
def hello_world():
message = 'Hello DockerCon 2018!'
html = wrap_html(message)
return html
if __name__ == '__main__':
app.run(host='0.0.0.0', port=5000)
The key takeaway in this code the message variable within the hello_world() function. This variable specifies a string value, and the value of this variable will be tested for a match in a unit test.
Testing the code
Code must be tested to make sure that high quality, stable code is released to the public. Python comes with a testing framework named unittest that I will be using for this tutorial. Now that we have a complete Flask application and it needs a companion unit test that will test the application and ensure it’s functioning as designed. The unit test file test_hello_world.py is the unit test for our hello_world.py app. Now, let’s walk through the code.
import hello_world
import unittest
class TestHelloWorld(unittest.TestCase):
def setUp(self):
self.app = hello_world.app.test_client()
self.app.testing = True
def test_status_code(self):
response = self.app.get('/')
self.assertEqual(response.status_code, 200)
def test_message(self):
response = self.app.get('/')
message = hello_world.wrap_html('Hello DockerCon 2018!')
self.assertEqual(response.data, message)
if __name__ == '__main__':
unittest.main()
import hello_world
import unittest
Importing the hello_world application using the import statement gives the test access to the code in the hello_world.py. Next, import the unittest modules and start defining test coverage for the application.
class TestHelloWorld(unittest.TestCase): The TestHelloWorld is instantiated from the base class unittest.Test, which is the smallest unit of testing. It checks for a specific response to a particular set of inputs. The unittest framework provides a base class, TestCase, that you will use to create new test cases.
def setUp(self):
self.app = hello_world.app.test_client()
self.app.testing = True
The class-level method setUp() is called to prepare the test fixture. It is called immediately before calling the test method. In this example, we create and define a variable named app and instantiate it as app.test_client() object from the hello_world.py code.
def test_status_code(self):
response = self.app.get('/')
self.assertEqual(response.status_code, 200)
The method test_status_code() specifies an actual test case in code. This test case makes a get request to the Flask application and captures the app’s response in the response variable. The self.assertEqual(response.status_code, 200) compares the value of the response.status_code result to the expected value of 200, which signifies the get request was successful. If the server responds with a status_code other than 200, the test will fail.
def test_message(self):
response = self.app.get('/')
message = hello_world.wrap_html('Hello DockerCon 2018!')
self.assertEqual(response.data, message)
Another method, test_message(), specifies a different test case. This test case is designed to check the value of the message variable that is defined in the hello_world() method from the hello_world.py code. Like the previous test, a get call is made to the app and results are captured in a response variable:
message = hello_world.wrap_html('Hello DockerCon 2018!')
The message variable is assigned the resulting html from the hello_world.wrap_html() helper method, as defined in the hello_world app. The string Hello DockerCon 2018 is supplied to the wrap_html() method which is then injected and returned in html. The test_message() verifies that the message variable in the app matches the expected string in this test case. If the strings do not match, the test will fail.
Implementing the CI/CD pipeline
Now that we are clear on the application and its unit tests, it is time to implement a CI/CD pipeline into the codebase. Implementing a CI/CD pipeline using CircleCI is very simple, but before continuing, make sure you do the following:
Setting up a CI/CD pipeline
Once your project is set up in the CircleCI platform, any commits pushed upstream will be detected and CircleCI will execute the job defined in your config.yml file.
Create a new directory in the repo’s root and add a yaml file within this new directory. The new assets must follow these naming schema - directory: .circleci/ file: config.yml in your project’s git repository. This directory and file basically define your CI/CD pipeline adn configuration for the CircleCI platform.
Configuration files
The config.yml file is where all of the CI/CD magic happens. After this code block which shows the example file, I will briefly explain what is going on within the syntax.
version: 2
jobs:
build:
docker:
- image: circleci/python:2.7.14
environment:
FLASK_CONFIG: testing
steps:
- checkout
- run:
name: Setup VirtualEnv
command: |
echo 'export TAG=0.1.${CIRCLE_BUILD_NUM}' >> $BASH_ENV
echo 'export IMAGE_NAME=python-circleci-docker' >> $BASH_ENV
virtualenv helloworld
. helloworld/bin/activate
pip install --no-cache-dir -r requirements.txt
- run:
name: Run Tests
command: |
. helloworld/bin/activate
python test_hello_world.py
- setup_remote_docker:
docker_layer_caching: true
- run:
name: Build and push Docker image
command: |
. helloworld/bin/activate
pyinstaller -F hello_world.py
docker build -t ariv3ra/$IMAGE_NAME:$TAG .
echo $DOCKER_PWD | docker login -u $DOCKER_LOGIN --password-stdin
docker push ariv3ra/$IMAGE_NAME:$TAG
- run:
name: Deploy app to Digital Ocean Server via Docker
command: |
ssh -o StrictHostKeyChecking=no root@hello.dpunks.org "/bin/bash ./deploy_app.sh ariv3ra/$IMAGE_NAME:$TAG"
The jobs: key represents a list of jobs that will be run. A job encapsulates the actions to be executed. If you only have one job to run, you must give it the key name build:. There are more details available about jobs and builds here.
The build: key is composed of a few elements:
docker:steps:
The docker: key tells CircleCI to use a Docker executor, which means our build will be executed using Docker containers.
image: circleci/python:2.7.14 specifies the Docker image that the build must use.
steps:
The steps: key is a collection that specifies all of the commands that will be executed in this build. The first action that happens, the - checkout command, performs a git clone of your code into the execution environment.
The - run: keys specify commands to execute within the build. Run keys have a name: parameter where you can label a grouping of commands. For example, name: Run Tests groups the test related actions. This grouping helps organize and display build data within the CircleCI dashboard.
Note: Each run block is equivalent to separate, individual shells or terminals. Commands that are configured or executed will not persist in later run blocks. Use the $BASH_ENV workaround in the tips & tricks section of the docs.
- run:
name: Setup VirtualEnv
command: |
echo 'export TAG=0.1.${CIRCLE_BUILD_NUM}' >> $BASH_ENV
echo 'export IMAGE_NAME=python-circleci-docker' >> $BASH_ENV
virtualenv helloworld
. helloworld/bin/activate
pip install --no-cache-dir -r requirements.txt
The command: key for this run block has a list of commands to execute. These commands set the $TAG & IMAGE_NAME custom environment variables that will be used throughout this build. The remaining commands set up the python virtualenv and installs the Python dependencies specified in the requirements.txt file.
- run:
name: Run Tests
command: |
. helloworld/bin/activate
python test_hello_world.py
In this run block, the command executes tests on our application. If these tests fail, the entire build will fail. Developers will need to fix their code and recommit.
- setup_remote_docker:
docker_layer_caching: true
This run block specifies the setup_remote_docker: key, a feature that enables building, running and pushing images to Docker registries from within a Docker executor job. When docker_layer_caching is set to true, CircleCI will try to reuse Docker Images (layers) built during a previous job or workflow. That is, every layer you built in a previous job will be accessible in the remote environment. However, in some cases your job may run in a clean environment, even if the configuration specifies docker_layer_caching: true.
The setup_remote_docker: feature is required because we are building a Docker image for our app and pushing that image to Docker Hub.
- run:
name: Build and push Docker image
command: |
. helloworld/bin/activate
pyinstaller -F hello_world.py
docker build -t ariv3ra/$IMAGE_NAME:$TAG .
echo $DOCKER_PWD | docker login -u $DOCKER_LOGIN --password-stdin
docker push ariv3ra/$IMAGE_NAME:$TAG
The Build and push Docker image run block specifies the commands that package the application into a single binary using pyinstaller. It then continues on to the Docker image building process.
docker build -t ariv3ra/$IMAGE_NAME:$TAG .
echo $DOCKER_PWD | docker login -u $DOCKER_LOGIN --password-stdin
docker push ariv3ra/$IMAGE_NAME:$TAG
These commands build the docker image based on the Dockerfile included in the repo. Instructions for building the Docker image can be found here: Dockerfile.
FROM python:2.7.14
RUN mkdir /opt/hello_word/
WORKDIR /opt/hello_word/
COPY requirements.txt .
COPY dist/hello_world /opt/hello_word/
EXPOSE 80
CMD [ "./hello_world" ]
The echo $DOCKER_PWD | docker login -u $DOCKER_LOGIN --password-stdin command uses the $DOCKER_LOGIN and $DOCKER_PWD env variables set in the CircleCI dashboard as credentials to login & push this image to Docker Hub.
- run:
name: Deploy app to Digital Ocean Server via Docker
command: |
ssh -o StrictHostKeyChecking=no root@hello.dpunks.org "/bin/bash ./deploy_app.sh ariv3ra/$IMAGE_NAME:$TAG"
The final run block deploys our new code to a live server running on the Digital Ocean platform, s make sure that you have created a deploy script on the remote server. The ssh command accesses the remote server and executes the deploy_app.sh script, including ariv3ra/$IMAGE_NAME:$TAG, which specifies the image to pull and deploy from Docker Hub.
After the job completes successfully, the new application should be running on the target server you specified in your config.yml file.
Summary
My goal for this tutorial is to guide you through implementing a CI/CD pipeline into your code. Although this example is built using Python technologies, the general build, test and deployment concepts can easily be implemented using any language or framework you prefer. You can expand on the simple examples in this tutorial and tailor them to your own pipelines.
I am Sunil kumar, Please do follow me here and support #devOps #trainwithshubham #github #devopscommunity #devops #cloud #devoparticles #trainwithshubham
Connect with me over linkedin : linkedin.com/in/sunilkumar2807