Dilaver DEMİREL
How to set up a CI (Continuous Integration) environment with Docker?
I want to create a sample pipeline. The steps within this pipeline can be defined as follows:
Note: This sample application is created as a Maven project.
- Checkout of the code
- Build (Maven) the project
- Run the unit tests in the project
- Run the static code analysis
To perform these operations, I need the following:
- A computer with Docker installed
- A Docker image for Jenkins
- A Maven Docker image to use when building the project
- A SonarQube Docker image to process and visualize the static code analysis
Assuming I have a computer with Docker installed, I can proceed to step 2. The Docker image I will use in this step is “jenkinsci/blueocean”. However, to retrieve the code, I need to make some changes to the Docker image for the SSH operations I require. To easily establish an SSH connection from within the container, I need to add the SSH private key to the image and also add the address I will connect to to “known_hosts”.
Accordingly, the docker file is created as follows:
FROM jenkinsci/blueocean
ARG ssh_prv_key
ARG ssh_pub_key
USER root
RUN mkdir -p /root/.ssh && \
chmod 0700 /root/.ssh && \
ssh-keyscan -p 2022 [host-name] > /root/.ssh/known_hosts
RUN echo "$ssh_prv_key" > /root/.ssh/id_rsa && \
echo "$ssh_pub_key" > /root/.ssh/id_rsa.pub && \
chmod 600 /root/.ssh/id_rsa && \
chmod 600 /root/.ssh/id_rsa.pub
USER jenkins
The following section allows you to add the address specified by [host-name] to the “known_hosts” file.
RUN mkdir -p /root/.ssh && \
chmod 0700 /root/.ssh && \
ssh-keyscan -p 2022 [host-name] > /root/.ssh/known_hosts
The following section allows you to add the SSH private and public keys, provided using the parameters “$ssh_prv_key” and “$ssh_pub_key”, to the “.ssh” directory.
RUN echo "$ssh_prv_key" > /root/.ssh/id_rsa && \
echo "$ssh_pub_key" > /root/.ssh/id_rsa.pub && \
chmod 600 /root/.ssh/id_rsa && \
chmod 600 /root/.ssh/id_rsa.pub
You can use the following command to build a Docker image:
docker build -t custom-jenkins-blueocean --build-arg
ssh_prv_key="$(cat /[your-path]/id_rsa)" --build-arg
ssh_pub_key="$(cat /[your-path]/id_rsa.pub)" --squash.
To run a container from the image we just created, you can use the following command:
docker volume create jenkins-data
docker run --name my-jenkins -u root --rm -d -p 8080:8080 -p 50000:50000 -v jenkins-data:/var/jenkins_home -v /var/run/docker.sock:/var/run/docker.sock custom-jenkins-blueocean
docker logs -f my-jenkins
After Jenkins starts, it asks for the admin password to ensure it opens as an administrator. You can obtain this information by following these steps:
docker exec -it my-jenkins bash
vi /var/jenkins_home/secrets/initialAdminPassword
In the next step, we can specify that Jenkins should install the default plugins. Jenkins is now ready to use.
We now see that the Jenkins environment is ready. The next step is to create a pipeline to handle CI/CD tasks for our project. You can use the following repository to test it;
https://github.com/dilaverdemirel/simple-java-maven-app
The Jenkinsfile within this repository will allow us to automatically create our pipeline.
pipeline {
agent {
docker {
image 'maven:3-alpine'
args '-v /root/.m2:/root/.m2 -v /var/run/docker.sock:/var/run/docker.sock --privileged'
}
}
stages {
stage('Build') {
steps {
sh 'mvn -B -DskipTests clean package'
}
}
stage('Test') {
steps {
sh 'mvn test'
}
post {
always {
junit 'target/surefire-reports/*.xml'
}
}
}
stage('Deliver') {
steps {
sh './jenkins/scripts/deliver.sh'
}
}
}
}
When we examined the script;
agent {
docker {
image 'maven:3-alpine'
args '-v /root/.m2:/root/.m2'
}
}
The section above provides the environment in which we will run the "stages" defined in the pipeline script. The stages are run within a Docker container created with a "maven:3-alpine" image. The "maven:3-alpine" image provides us with a Maven 3 installed environment. Our goal is to run our project, created as a Maven project, through the build/test/deliver steps and perform the defined checks and tasks. For this, we needed the Maven tool, which we accomplished with the "maven:3-alpine" image.
In this way, we built and tested the project with Docker without installing Maven.
At this point, if you are using a custom Maven repository, you will need to modify the Maven "settings.xml" file. You can use the following Docker file for this:
#Dockerfile
FROM maven:3.6.1-jdk-8-slim
COPY settings.xml /usr/share/maven/conf/
The "settings.xml" file, necessary for Maven, must also be in the directory where the Dockerfile is located. You can build the image with the following command:
docker build -t maven-jdk-8-slim .
To use the custom image you created, you must change the agent section in the "Jenkinsfile" as follows:
agent {
docker {
image 'maven-jdk-8-slim'
args '-v /root/.m2:/root/.m2'
}
}
In summary, we set up the Jenkins environment with Docker and built our project using a Maven Docker image. We didn't manually install any other tools besides Docker.
What is Video?
It is the creation of a visual image that appears to be moving, formed by a series of images arranged in sequence. The FPS (Frames per second) value is very important for videos.
Furthermore, the compression of the image during transmission provides great convenience in today's technologies. For example:
1 Pixel = 8 bits * 3 (RGB) => 24 bits
A 1024x1024 pixel image occupies approximately 24 Mbit of space (1K * 1K * 24).
If we were transmitting this image without compression/encoding, we would need 24 Mbit of bandwidth for 1 frame per second (1 FPS).
This is an unacceptable FPS value for moving images.
Now that we have learned about images and videos, we can move on to the algorithmic side of object detection.
CNN (Convolitional Neural Network)
The CNN (ConvNet) algorithm is an algorithm used to detect objects in images and videos.
CNN algorithms use unique features of objects in an image to distinguish (learn) from each other.
The CNN algorithm itself could be the subject of a separate article. Since our topic isn't solely CNN, let's discuss some basic information about the algorithm.
Structure:
Convolution Layer: Detects unique features on the image/frame.
Non-Linearity Layer: The image matrix is normalized by passing it through an activation function. The chosen activation function affects the speed of the neural network training that will take place later.
Pooling Layer: Reduces the number of weights in the matrix. Some users increase the size of the filter matrix used in the convolution layer instead of using the pooling layer.
Max Pooling
Fully-Connected Layer: The events up to this layer represent the preparation of the image for the artificial neural network. This layer is where the system is trained, i.e., where the artificial neural network operates. It can be said that the output of the CNN algorithm is this layer.
Now we can move on to YOLO, the main character of our article 🙂
What is YOLO?
YOLO is an algorithm that uses convolutional neural networks to detect objects. Its full name is "You Only Look Once."
This is because the algorithm can detect objects very quickly and in a single pass.
The reason YOLO is faster than other algorithms is that it processes the entire image through a neural network in one go.
The YOLO algorithm surrounds the detected objects in images with a bounding box.
YOLO divides the input image into NxN grids. These grids can be 5x5, 9x9, 17x17…
Each grid checks whether there is an object within it, and if it thinks there is an object, it considers whether the object's center point is within its area.
The grid that decides that the object has a center point must find the object's class, height, and width and draw a bounding box around that object.
YOLO Grid System (Representative)
Multiple grids may assume an object is within themselves. This creates unnecessary bounding boxes on the screen.
All bounding boxes have a confidence score.
To prevent this, the Non-Maximum Suppression algorithm is used.
In short, the Non-Maximum Suppression algorithm draws the bounding box with the highest confidence value for detected objects on the image.
Non-Max Suppression
The following graphs show the object detection performance of YOLO and some other algorithms for the MS COCO dataset.
https://arxiv.org/abs/2004.10934
As seen in the graphs, considering a case where the number of classifiers is equal, YOLOv4 is almost three times better than its competitors.
Now let's briefly discuss the object recognition stages with YOLO. As an example, let's examine a project that detects masked and unmasked faces.
1) Data Collection/Labeling
You can perform data labeling via https://www.makesense.ai/. Labeling is important so that we can provide the algorithm with training data that will allow it to distinguish between masked and unmasked people and train itself. Besides Make Sense, you can find many other platforms for image labeling. I prefer Make Sense because it provides the labeled data in the format required by YOLO. Below, we see example images for the labeling phase.
As an example, I labeled two images and when I exported the result in YOLO format, I had two different matrices for the two images as shown below.
After labeling the images, I trained my YOLO model via Google Colab through the DarkNet. I am sharing some screenshots of the training stages with you.
The DarkNet requires certain files from us to run the YOLO algorithm. One of these is the .cfg file.
The config file requests features that will affect the success, speed, etc., of the neural network within YOLO.
Another file that YOLO expects from us is the .names file. The .names file holds the names of the classes. For mask detection, it includes both Mask and No-Mask options.