Skip to main content

16 posts tagged with "DevOps"

View All Tags

healthchecks

· One min read

Open in Notion

Running in Docker​

docker run -d \
  --name=hc \
  -p 8010:8000 \
  --restart unless-stopped \
  -e DB=sqlite \
  -e DB_NAME=/data/hc.sqlite \
  -e DEBUG=False \
  -e SITE_ROOT=http://10.224.59.43:8010 \
  -v ~/healthchecks-data:/data \
healthchecks

docker exec -it hc /opt/healthchecks/manage.py createsuperuser

Podman

· 2 min read

Open in Notion

The alternative of docker as docker is no longer free for enterprise users.

Installation​

bookmark

Install from download​

wget https://github.com/containers/podman/releases/latest/download/podman-remote-static-linux_amd64.tar.gz
tar -xvzf podman-remote-static-linux_amd64.tar.gz
sudo mv ./bin/podman-remote-static-linux_amd64 /usr/local/bin/podman
sudo chmod +x /usr/local/bin/podman

Examples​

# If you want to keep your existing volumns, you should use --userns=keep-id,
# otherwise, it might throw "operation not permitted" when start a container
podman run -d --name my-container --userns=keep-id \
    -v /your-path:/path
	image-name

Configure sources​

Open /etc/containers/registries.conf and uncomment below line, so that you can pull images from other sources.

unqualified-search-registries = ["registry.fedoraproject.org", "registry.access.redhat.com", "docker.io"]

Jenkins in Podman​

Run podman in jenkins pipeline if jenkins is running in container​

Mount the socket of podman into jenkins container.

docker run -d \
  --name my-jenkins \
  -p 8080:8080 \
  -v /home/<host_user>/jenkins_home:/var/jenkins_home \
  -v /run/user/1000/podman/podman.sock:/run/user/1000/podman/podman.sock \
  bndynet/jenkins

Then run it in pipeline:

pipeline {
    agent any
    stages {
        stage('Run Podman') {
            steps {
                sh 'export PODMAN_HOST=unix:///run/user/1000/podman/podman.sock && podman run --rm hello-world'
            }
        }
    }
}

Let jenkins account run podman​

sudo usermod --add-subuids 100000-165535 --add-subgids 100000-165535 jenkins

If the jenkins user does not exist, please create one:

sudo useradd -m -s /bin/bash jenkins
sudo passwd jenkins  # set password（optional）

Restart jenkins

sudo systemctl restart jenkins
# or podman restart jenkins-container

Automatically Start​

Podman provides generate systemd command, that can generate systemd service file to restart when server starts.

podman generate systemd --name my-container --restart-policy=always > /etc/systemd/system/my-container.service

Then enable and start service:

systemctl daemon-reload
systemctl enable my-container
systemctl start my-container

In this way, my-container will start automatically, it likes docker --restart unless-stopped.

Q&A​

Can not “podman machine start”?​

You need to remove the current machine via podman machine rm. and reinit via podman machine init.

Update Jenkins inside Docker Container

· One min read

Open in Notion

First identify your image.

$ docker ps --format "{{.ID}}: {{.Image}} {{.Names}}"3d2fb2ab2ca5: jenkins-docker jenkins-docker_1

Then login into the image as root.

$ docker container exec -u 0 -it jenkins-docker_1 /bin/bash

Now you are inside the container, download the jenkins.war file from the official site like.

# wget http://updates.jenkins-ci.org/download/war/2.176.1/jenkins.war

Replace the version with the one that fits to you.

The next step is to move that file and replace the oldest one.

# mv ./jenkins.war /usr/share/jenkins/

Then change permissions.

# chown jenkins:jenkins /usr/share/jenkins/jenkins.war

The last step is to logout from the container and restart it.

$ docker restart jenkins-docker_1

You can verify that update was successful by access to you Jenkins url.

Useful Scripts for Jenkins

· One min read

Open in Notion

Decrypt credentials defined in Jenkins and list values.​

def creds = com.cloudbees.plugins.credentials.CredentialsProvider.lookupCredentials(
    com.cloudbees.plugins.credentials.common.StandardUsernameCredentials.class,
    Jenkins.instance,
    null,
    null
);
for (c in creds) {
  if (c.properties.privateKeySource) {
	  println("ID: " + c.id)
    println("UserName: " + c.username)
    println("Description: " + c.description)
	  println("Private Key: " + c.getPrivateKey() )
  }
}

Decrypt the password if you got its HASH.​

println(hudson.util.Secret.decrypt("{HASHxxxxx=}"))
# or /////
println(hudson.util.Secret.fromString("{HASHxxxxx=}").getPlainText())

Trigger build by url​

# Run when the anonymous user has the build permission
curl http://127.0.0.1:8080/job/Trigger_Remote_Demo/build?token=My-token
# Send request with credentials
curl -u username:api_token http://127.0.0.1:8080/job/Trigger_Remote_Demo/build
# Build with Parameter
curl -u username:api_token http://127.0.0.1:8080/job/Trigger_Remote_Demo/buildWithParameters?para1=val1&para2=val2

Jenkins

· 3 min read

Open in Notion

Start Jenkins​

Startup script to set the System Properties​

Step 1:

  • Locate the Jenkins home directory. You can look into the existing system properties for ‘jenkins_home’ or execute ‘echo $JENKINS_HOME’ inside the jenkins server to get the Jenkins home directory.
  • Say if your Jenkins home directory is ‘/var/jenkins_home’. Create a new directory with name ‘init.groovy.d’.

Step 2:

  • Now change your working directory to ‘/var/jenkins_home/init.groovy.d’
  • Within init.groovy.d directory create a new groovy script. Say ‘startup-properties.groovy’

Step 3:

  • Copy the below content to the ‘startup-properties.groovy’ file
import jenkins.model.Jenkins
import java.util.logging.LogManager

/* Jenkins home directory */
def jenkinsHome = Jenkins.instance.getRootDir().absolutePath
def logger = LogManager.getLogManager().getLogger("")

/* Replace the Key and value with the values you want to set.*/
/* System.setProperty(key, value)*/
System.setProperty("hudson.model.WorkspaceCleanupThread.disabled", "true")
logger.info("Jenkins Startup Script: Successfully updated the system properties. Script location: ${jenkinsHome}/init.groovy.d")

Visit https://docs.oracle.com/en/java/javase/11/docs/api/java.base/java/lang/System.html?is-external=true#setProperty(java.lang.String,java.lang.String) to see how to use setProperty.

Step 4:

Restart the Jenkins server, you can manually restart the server using :

<Jenkins-server-URL>/restart or <Jenkins-server-URL>/safeRestart

  • Use safeRestart when you have some job still running. It will wait till all the jobs are completed.

Step 5:

Check the logs after Jenkins restart: Manage Jenkins → System Log. You can see while server is starting, its executing the startup-properties.groovy script which is inside init.groovy.d directory.

1*kjzZCxTixWCYsQbIcfRkzA.png

For example, run the below scripts in Script Console to check the System Properties, you will see the required key and values.

println(System.getProperty("hudson.model.WorkspaceCleanupThread.disabled"))

Now every time server is restarted the script will run and set the system properties for you. You don’t need to worry for setting these system properties manually every time server restart.

https://wiki.jenkins-ci.org/display/JENKINS/Features-controlled-by-system-properties.html

Start docker container with system properties​

link_preview

docker run --name my-jenkins -p 8080:8080 -p 50000:50000 -v /your/jenkins_home:/var/jenkins_home \
  -d \
  -v /var/run/docker.sock:/var/run/docker.sock \
  --env JAVA_OPTS="-Dhudson.model.WorkspaceCleanupThread.disabled=\"true\"" \
  --restart unless-stopped \
  --user root \
  bndynet/jenkins

Plugin: Configuration as Code (aka.JCasC )​

Apply configuration from Git

https://<USERNAME>:<PERSON_ACCESS_TOKEN>@github.com/your-repo/jenkins-config.git

Add Nodes​

Step 1. In node server, install java by https://sdkman.io/

curl -s "https://get.sdkman.io" | bash
sdk install java 17.0.14-tem

Step 2. Add java path for all users (/etc/environment)

JAVA_HOME=/home/<user>/.sdkman/candidates/java/current
PATH="/home/<user>/.sdkman/candidates/java/current/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin"

Step 3. Generate ssh key in jenkins server, if you are running jenkins on container, you should enter the container to do that.

ssh-keygen -t rsa -b 4096 -f /var/jenkins_home/id_rsa

Step 4. Copy pub key to node server

ssh-copy-id -i /var/jenkins_home/.ssh/id_rsa.pub username@hostname

Step 5. Add Credentials in jenkins UI with the private key

Step 6. Add Node in jenkins UI with Launch agens via SSH, and choose above Credentials

Q: Could not copy remoting.jar into '/home/jenkins' on agent​

If below error thrown:

Permission denied (SSH_FX_PERMISSION_DENIED: The user does not have sufficient permissions to perform the operation.)

GitHub Actions

· One min read

Open in Notion

Context in Workflow​

More contexts at https://docs.github.com/en/actions/learn-github-actions/contexts

  • Url for repo, for example: https://github.com/bndynet/bndynet ${{ github.server_url }}/${{ github.repository }}
  • Url for pull request ${{ github.server_url }}/${{ github.repository }}/pull/${{ github.event.pull_request.number }}
  • Url for issue ${{ github.server_url }}/${{ github.repository }}/issues/${{ github.event.issue.number }}

About CRON​

# ┌───────────── minute (0 - 59)
# │ ┌───────────── hour (0 - 23)
# │ │ ┌───────────── day of the month (1 - 31)
# │ │ │ ┌───────────── month (1 - 12)
# │ │ │ │ ┌───────────── day of the week (0 - 6)
# │ │ │ │ │                       
# │ │ │ │ │
# │ │ │ │ │
# * * * * * <command to execute>

Examples​

name: Test Build

on:  
  push:
  pull_request:
  schedule:
    - cron: '00 1 * * 1'  # At 01:00 on Mondays.
name: Sync Notion pages to posts

on:
  schedule:
    # Runs "At 20:00 on every day-of-week"
    - cron: '0 20 * * *'
    
jobs:
  # Build job
  sync:
    runs-on: ubuntu-latest
    steps:
      - name: Checkout
        uses: actions/checkout@v3

      - name: Generate posts
        uses: bndynet/github-action-notion@v1
        with:
          notion-token: ${{ secrets.NOTION_TOKEN}}
          root-page-id: ${{ secrets.NOTION_ROOT_PAGE_ID }}

      - name: Commit posts
        uses: EndBug/add-and-commit@v9  # https://github.com/marketplace/actions/add-commit
        with:
          add: '_posts'
          message: Sync Notion pages to posts by GitHub Actions
          committer_name: Bendy Zhang
          committer_email: email@your.com

Nginx

· 5 min read

Open in Notion

Run in Docker​

sudo docker pull nginx
sudo docker run --name my-nginx -p 8080:80 -v /some/content:/usr/share/nginx/html:ro -d nginx
sudo chmod -R 755 /some/content

Install Nginx on CentOS 8​

1. Update the System:​

First, update your system to ensure all packages are up to date.

sudo dnf update -y

2. Add the Nginx Repository​

CentOS 8 doesn't include the latest Nginx packages in its default repositories. To get the latest version, add the official Nginx repository.

Create a repository file for Nginx:

sudo nano /etc/yum.repos.d/nginx.repo

Add the following content to the file:

[nginx-stable]
name=nginx stable repo
baseurl=http://nginx.org/packages/centos/$releasever/$basearch/
gpgcheck=1
enabled=1
gpgkey=https://nginx.org/keys/nginx_signing.key

[nginx-mainline]
name=nginx mainline repo
baseurl=http://nginx.org/packages/mainline/centos/$releasever/$basearch/
gpgcheck=1
enabled=0
gpgkey=https://nginx.org/keys/nginx_signing.key

The nginx-stable repository is enabled by default, while the nginx-mainline repository is not. You can enable the nginx-mainline repository by setting enabled=1 if you prefer to use the mainline version of Nginx.

3. Install Nginx:​

Install Nginx using the dnf package manager.

sudo dnf install nginx -y

4. Start and Enable Nginx:​

Once the installation is complete, start the Nginx service and enable it to start on boot.

sudo systemctl start nginx
sudo systemctl enable nginx

5. Verify Nginx Installation:​

You can verify that Nginx is running by checking its status.

sudo systemctl status nginx

6. Adjust Firewall Settings:​

If you have a firewall enabled, you'll need to allow traffic on HTTP (port 80) and HTTPS (port 443) ports.

sudo firewall-cmd --permanent --zone=public --add-service=http
sudo firewall-cmd --permanent --zone=public --add-service=https
sudo firewall-cmd --reload

Configuration for static website based on framework​

server {
    listen 80;
    server_name yourdomain.com; 

    root /var/www/angular-app; 
    index index.html;
    
    location / {
        try_files $uri /index.html;
    }

    error_page 404 /index.html;

    gzip on;
    gzip_types text/plain text/css application/json application/javascript text/xml application/xml application/xml+rss text/javascript;
}

Logs​

sudo tail -f /var/log/nginx/error.log

Distribute Domains to Ports​

server {
    listen 80;
    server_name domain1.com;
 
    location / {
        proxy_pass http://localhost:8080;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    }
}
 
server {
    listen 80;
    server_name domain2.com;
 
    location / {
        proxy_pass http://localhost:8081;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    }
}

Use map:

http {
    map $host $backend {
        default localhost:80;
        domain1.com localhost:8080;
        domain2.com localhost:8081;
    }

    server {
        listen 80;

        location / {
            proxy_pass http://$backend;
            proxy_set_header Host $host;
            proxy_set_header X-Real-IP $remote_addr;
            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
            proxy_set_header X-Forwarded-Proto $scheme;
        }
    }
}

Certificate in Nginx​

1. Obtain an SSL/TLS Certificate​

You can obtain a certificate from a Certificate Authority (CA) like Let's Encrypt, which provides free SSL/TLS certificates. Using Certbot, you can automate the process of obtaining and renewing certificates

Install Certbot:

sudo dnf install certbot python3-certbot-nginx -y

Obtain a Certificate: Replace your_domain.com with your actual domain name.

sudo certbot --nginx -d your_domain.com

Follow the prompts to complete the certificate issuance process. Certbot will automatically configure Nginx for SSL.

2. Manually Configuring Nginx with an SSL/TLS Certificate​

If you have an existing certificate or need to configure it manually, follow these steps:

Create a Configuration File for Your Site:

sudo nano /etc/nginx/conf.d/your_domain.conf

Add SSL Configuration: Replace your_domain.com with your actual domain name and specify the paths to your certificate and key files.

server {
    listen 80;
    server_name your_domain.com;
    return 301 https://$host$request_uri;  # Redirect HTTP to HTTPS
}

server {
    listen 443 ssl;
    server_name your_domain.com;

    ssl_certificate /etc/letsencrypt/live/your_domain.com/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/your_domain.com/privkey.pem;

    ssl_protocols TLSv1.2 TLSv1.3;
    ssl_prefer_server_ciphers on;
    ssl_ciphers 'ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384';
    ssl_session_cache shared:SSL:10m;
    ssl_session_timeout 10m;

    location / {
        proxy_pass http://backend_server;  # Change to your backend application address
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
    }
}

3. Configure Firewall​

Ensure your firewall allows HTTPS traffic:

sudo firewall-cmd --permanent --zone=public --add-service=https
sudo firewall-cmd --reload

4. Test and Reload Nginx​

Test Nginx Configuration:

sudo nginx -t

Reload Nginx to Apply Changes:

sudo systemctl reload nginx

Q&A​

Error about socket 80​

nginx: [emerg] socket() [::]:80 failed (97: Unknown error)

Nginx try to listen the IPV6 [::]:80. Comment below line from the configuration /etc/nginx/sites-available/default.

# listen [::]:80 default_server;

Permission denied​

Let www-data user have the permission to read the folder.

sudo chown -R www-data:www-data /your/site
sudo chmod 755 /your/site

If you still have same error, maybe your path under some user home folder. You should change the home folder permission.

sudo chown -R username:www-data /home/username
sudo chmod -R 750 /home/username
sudo chmod -R 755 /home/username/public_html

Docker Q&A

· One min read

Open in Notion

IPv4 forwarding is disabled. Networking will not work​

Solution 1: This won’t work every time

// run on host server
sysctl -w net.ipv4.ip_forward=1
sudo systemctl restart network

Solution 2: Append net.ipv4.ip_forward=1 to /etc/sysctl.conf file.

Solution 3: Append parameter —net=host to docker run, that will use host ports and ignore the -p parameters for example:

docker run -it --network host –name mynginx nginx

/var/jenkins_home/copy_reference_file.log Permission denied​

$ docker run -ti -p 8080:8080 -p 50000:50000 -v /opt/jenkins:/var/jenkins_home jenkins touch: cannot touch ‘/var/jenkins_home/copy_reference_file.log’: Permission denied Can not write to /var/jenkins_home/copy_reference_file.log. Wrong volume permissions?

If you meet the above error, please figure out your volume mapping permissions.

sudo chown -R 1000:1000 /opt/jenkins

ELK

· 2 min read

Open in Notion

Refer https://elk-docker.readthedocs.io/#installation

$ sudo docker run -p 5601:5601 -p 9200:9200 -p 5044:5044 -it --name elk sebp/elk

Running the container using Docker Compose

elk:
  image: sebp/elk
  ports:
    - "5601:5601"
    - "9200:9200"
    - "5044:5044"
$ sudo docker-compose up elk

Creating a dummy log entry​

$ sudo docker exec -it <container-name> /bin/bash
# /opt/logstash/bin/logstash --path.data /tmp/logstash/data \
    -e 'input { stdin { } } output { elasticsearch { hosts => ["localhost"] } }'

http://localhost:9200/_search?pretty&size=1000 You will see:

{
  ...
  "hits": {
    ...
    "hits": [ {
      "_index": "logstash-...",
      "_type": "logs",
      ...
      "_source": { "message": "this is a dummy entry", "@version": "1", "@timestamp": ... }
    } ]
  }
}

Urls or Ports​

Logstash

Commands​

/opt/logstash/bin/logstash --debug
/opt/logstash/bin/logstash -f your-config-file
/opt/logstash/bin/logstash-plugin list

Configurations​

input {
  http {
    host => "0.0.0.0"
    port => 5044
    type => http
    response_headers => {
      "Access-Control-Allow-Origin" => "*"
      "Content-Type" => "text/plain"
      "Access-Control-Allow-Headers" => "Origin, X-Requested-With, Content-Type,
       Accept"
    }
  }
  tcp {
    host => "0.0.0.0"
    port => 5045
    codec => json_lines
    type => logback
  }
}

filter {
  if [headers][request_method] == "OPTIONS" {
    drop {}
  }
}

output {
  if [type]=="http" and [headers.request_method]!="OPTIONS" {
    elasticsearch {
      hosts => ["127.0.0.1:9200"]
      manage_template => false
      #index => "%{APP_NAME}-%{[@metadata][beat]}-%{+YYYY.MM.dd}"
      index => "http-%{+YYYY.MM.dd}"
    }
  }
  if [type]=="logback" {
    elasticsearch {
      hosts => ["127.0.0.1:9200"]
      manage_template => false
      #index => "%{APP_NAME}-%{[@metadata][beat]}-%{+YYYY.MM.dd}"
      index => "logback-%{APP_NAME}-%{+YYYY.MM.dd}"
    }
  }
}