Kubernetes Jobs and Istio

Note: the Job in the title refers to the Job resource in a Kubernetes cluster.

At the time the Istio sidecar doesn’t play well with a Job or a Cronjob, because the istio-proxy might not be ready when the Job starts (which causes connection issues for the job) and won’t exit after the job finishes (which causes the Job stuck and won’t be marked as complete).

Here’s a simple Bash script for a Job assuming the Job’s container has Bash and curl

apiVersion: batch/v1
kind: Job
metadata:
  name: db-migrate
spec:
  template:
    metadata:
      name: db-migrate
    spec:
      restartPolicy: Never
      containers:
        - name: db-migrate
          image: "some-image-with-curl:v0.1"
          command:
            - /bin/bash
            - -c
            - |
              # wait for the istio-proxy to become ready
              until curl -fsI http://localhost:15021/healthz/ready; do
                echo 'Waiting for Sidecar...'
                sleep 1
              done
              # do the job here
              bundle exec rails db:migrate
              # ask the istio-proxy to exit
              curl -fsI -X POST http://localhost:15020/quitquitquit

And if the job container image doesn’t have bash or curl, I used a curl image as another sidecar to get the job done

---
apiVersion: batch/v1
kind: Job
metadata:
  name: db-migrate
spec:
  template:
    metadata:
      name: db-migrate
    spec:
      restartPolicy: Never
      volumes:
        - name: flags
          emptyDir: {}
      containers:
        - name: curl
          image: curlimages/curl:7.78.0
          command:
            - /bin/sh
            - -c
            - |
              # test istio-proxy
              until curl -fsI http://localhost:15021/healthz/ready; do
                echo 'Waiting for Sidecar...'
                sleep 1
              done
              # touch the flag in tmp dir
              touch /tmp/flags/istio-proxy-ready
              # then wait for the job to finish
              until [ -f /tmp/flags/done ]; do
                echo 'Waiting for the job to finish...'
                sleep 1
              done
              # ask istio-proxy to exit
              curl -fsI -X POST http://localhost:15020/quitquitquit
          volumeMounts:
            - name: flags
              mountPath: /tmp/flags
        - name: db-migrate
          image: "some-image-without-curl:v0.1"
          command:
            - /bin/bash
            - -c
            - |
              # wait for the flag of istio-proxy
              until [[ -f /tmp/flags/istio-proxy-ready ]]; do
                echo 'Waiting for Sidecar...'
                sleep 1
              done
              # do the job
              bundle exec rails db:migrate
              # set the flag so curl can shut down istio-proxy
              touch /tmp/flags/done
          volumeMounts:
            - name: flags
              mountPath: /tmp/flags

🙂

Mining Ethereum with AMD 6600 XT on Ubuntu Linux

Warning: Ethereum(ETH) will migrate to PoS(Proof of Stake) algorithm in near future, maybe in a year. So jumping into ETH mining now might or might not be profitable. Also, I do NOT condone crypto mining with fossil fuel based energy source!

Prerequisite: A PC running on Ubuntu 20.04 with stable Internet connection

AMD RX 6600 XT is latest addition to the Big Navi product family. Below are the steps I managed to do ETH mining with a 6600 XT on Ubuntu 20.04.

Installing AMD Radeon Pro driver

The driver for 6600 XT can be downloaded here, and the official instructions for installing the pro driver is here. And the exact commands I used to install are below

# after the driver is downloaded, extract it with tar
$ tar -xvf amdgpu-pro-21.30-1290604-ubuntu-20.04.tar.xz
...
# install the pro version with OpenCL support
$ cd amdgpu-pro-21.30-1290604-ubuntu-20.04/
$ ./amdgpu-pro-install -y --opencl=rocr,legacy
# add current user to video and render group
$ sudo usermod -a -G video $LOGNAME
$ sudo usermod -a -G render $LOGNAME
# reboot to load the driver
$ sudo reboot
...
# install clinfo to check if OpenCL is enabled properly
$ sudo apt install clinfo
$ clinfo
...
Number of devices                                 1
  Device Name                                     gfx1032
...
# success!

Build rocm_smi_lib to set power limit, memory clock speed, etc

With nVidia’s Linux driver, there’s a bundled tool nvidia-smi to control the card’s power limit, etc. But that’s missing from AMD’s driver bundle. Thanks to Radeon Open Compute the rocm_smi_lib is here. It’s source code so I need to build it myself, but it’s quite straight forward.

# install dependencies 
$ sudo apt install cmake g++
# clone the github repo
$ git clone https://github.com/RadeonOpenCompute/rocm_smi_lib.git
# start the build
$ cd rocm_smi_lib/
$ mkdir build
$ cd build
$ cmake ..
...
$ make
...
$ sudo make install
...
# set max power to 55W for the first card
$ /opt/rocm/bin/rocm-smi -d 0 --setpoweroverdrive 55
# check result
$ /opt/rocm/bin/rocm-smi
======================= ROCm System Management Interface =======================
================================= Concise Info =================================
GPU  Temp   AvgPwr  SCLK    MCLK     Fan     Perf  PwrCap  VRAM%  GPU%  
0    55.0c  54.0W   920Mhz  1000Mhz  22.75%  auto  55.0W    61%   99%   
================================================================================
============================= End of ROCm SMI Log ==============================

Install and Start TeamRed Miner

This is the most easy part but has to be done the last 🙂 The latest release can be downloaded here and commands to get the mining started are below

# in Downloads directory
$ tar xvf teamredminer-v0.8.4-linux.tgz
...
$ mv teamredminer-v0.8.4-linux ~/
$ cd ~/teamredminer-v0.8.4-linux/
# in start_eth.sh, replace the ETH wallet address to yours otherwise you will be mining for someone else :)
$ vim start_eth.sh
...
# start mining!
$ ./start_eth.sh
[2021-08-22 10:06:25] Pool asia1.ethermine.org received new job. (job_id: 0x65d72f5b396..., diff 1.000 / 4295 MH)
...

If this actually helped you, you’re welcome to support my blogging by running your mining rig with my wallet( 0x0016a5e2d5a32d40f89475218427d35798d05d8a ) for a few hours. Or maybe just click on an ad. which is much easier.

🙂

EDIT: It seemed when 2 6600 XT cards are plugged in only 1 of them becomes available to OpenCL.

Combine GitHub Actions with FluxCD

I started my hobby project SausLink( a tinyURL like URL shortener ) a while ago, as it was really boring during covid lockdowns. The web app itself isn’t anything cutting-edge but I intended to implement full git-ops for this project.

For the CI pipeline, I chose GitHub Actions because it’s easy and also free for open source projects( for a limited builds per day ). The pipeline will be triggered by pushing to dev branch and will build a docker image and then tag it with a GitHub Actions job number. The last step will update the kustomize patch file to refer to the latest tag.

Here’s the pipeline definition along with my comments

# this is .github/workflows/dev.yaml
name: Build the dev image

on:
  push:
    branches: [ dev ]
  pull_request:
    branches: [ dev ]

jobs:
  build:
    runs-on: [ ubuntu-20.04 ]
    steps:
      # Checks-out your repository under $GITHUB_WORKSPACE, so your job can access it
      - uses: actions/[email protected]

      # save a tag like dev-123 to environment variable called build_number
      - name: build number
        run: |
          echo "build_number=dev-${GITHUB_RUN_ID}" >> $GITHUB_ENV

      # build the docker image and tag it with build_number
      - name: Docker build saus-gunicorn
        run: |
          docker build -t ghcr.io/raynix/saus:$build_number .

      # login to ghcr.io and push the docker image
      - name: Docker push
        run: |
          echo ${{ secrets.GHCR_PAT }} | docker login ghcr.io -u $GITHUB_ACTOR --password-stdin
          docker push ghcr.io/raynix/saus:$build_number

      # update the kustomize patch to use the latest tag, using sed command and regex capture
      - name: Update deployment version
        run: |
          sed -i -E "s|(image: ghcr.io/raynix/saus:).*|\1$build_number|g" .kustomize/dev.sausl.ink/patch.yaml

      # add an auto commit with the updated patch file, so the FluxCD can pick it up and deploy with the latest tag
      - name: Auto commit & push changes
        run: |
          git config --global user.name 'raynix Bot'
          git config --global user.email '[email protected]'
          git commit -am "Automated commit"
          git push

At the CD pipeline run by FluxCD v2, a Kustomization resource is defined as follow

---
# this is the definition of the git repo where the kustomize templates are located
apiVersion: source.toolkit.fluxcd.io/v1beta1
kind: GitRepository
metadata:
  name: saus-dev
  namespace: flux-system
spec:
  interval: 4m0s # check the repo every 4 minutes, don't be greedy
  ref:
    branch: dev # fixed to dev branch for dev environment
  url: https://github.com/raynix/saus.git

---
# this is the kustomization definition
apiVersion: kustomize.toolkit.fluxcd.io/v1beta1
kind: Kustomization
metadata:
  name: saus-dev
  namespace: flux-system
spec:
  healthChecks:
  - kind: Deployment
    name: saus
    namespace: saus-dev
  interval: 4m0s # reconcile the deployment every 4 minutes
  path: .kustomize/dev.sausl.ink # directory where the main kustomization.yaml resides in
  prune: true
  sourceRef:
    kind: GitRepository
    name: saus-dev
  timeout: 2m0s
  validation: client
  force: true # force to re-deploy objects which have immutable fields such as jobs

For more FluxCD experiments please see my previous post here. The SausLink’s source code is here. Feel free to fork it and host your own URL shortener 🙂

1-Step Away From Fossil Fuel: Sanden Heat Pump

Sanden Heat Pump Hot Water system freshly installed

After weeks of waiting, thanks to lock downs in Victoria, I finally have a 315L Sanden Heat Pump installed, replacing my who-knows-how old gas instantaneous water heater. The job has been done professionally by SolarFlow, kudos to the team!

I can label myself an environmentalist who keeps trying to minimize my own carbon footprint but this heat pump is also a logical and economical decision.

First of all, a heat pump works like a reverse air conditioner: it extracts heat from the outside and heats up inside. In other words, unlike gas or old school electric water heater, heat pump doesn’t use energy to generate heat but only transfer it. During my little research on heat pump products, I noticed that amongst all heat pumps Sanden stands out as the most efficient one: with 1 unit of energy consumed it can transfer up to 6 units of energy, a 600% coefficient of performance(COP)! This means comparing to my gas water heater which probably has <60% of efficiency the heat pump is a tenfold more efficient. And high efficiency means less running cost.

With solar panels the perk of being an electric appliance is that when the sun is shining, the heat pump runs(almost) for free. Just to be clear, I still need to pay for water regardless weather. With a rough calculation, I paid about 500 dollars a year for hot water with the old gas heater, so this heat pump can possibly pay for itself in 10 years. ( I found it interesting when it is a V8 diesel SUV no one asks if it will pay for itself later but people do ask this question if it’s renewable related products )

Fingers crossed though, the tank is warranted for 15 years but the heat pump itself only has 6 years of warranty.

The temperature of the hot water out of the tank feels like 60°C, which means it needs to mix with some cold water to achieve a comfortable temperature, so the 315L tank practically can supply me much more than 315L of hot water depending on the desired temperature.

Last but not least it’s also quieter when running. I could definitely tell if the gas heater was running 5 meters away from it, but won’t be so sure about the heat pump.

Overall I’m quite pleased with this upgrade 🙂