Run ArgoCD with Istio Service Mesh in a Kubernetes Cluster

It’s been quite a while since I installed Flux CD V2 in my garage Kubernetes lab, as there’s a lot of debate going on between Flux and ArgoCD I decided to give ArgoCD a go. The other reason to try ArgoCD is that it supports Jsonnet.

By default installation, ArgoCD will use self-signed TLS certificate and enforce TLS connection, which means users get to see the security warning and have to trust the certificate to continue. Naturally with Istio handles ingress and TLS termination, I would like to enable Istio sidecar for ArgoCD and run it in HTTP mode.

Here are the steps to configure and install ArgoCD along side with Istio:

Enalbe Istio Sidecar

I choose to enable automatic Istio sidecar injection for ArgoCD’s namespace.

# create the namespace, by default it's argocd
kubectl create namespace argocd
# turn on istio injection
kubectl label namespace argocd istio-injection=enabled

Install ArgoCD the Normal Way

kubectl apply -n argocd -f https://raw.githubusercontent.com/argoproj/argo-cd/stable/manifests/install.yaml

Disable TLS for argocd-server Deployment

This can be done before or after the deployment being applied to the cluster in the above step, eg. edit the install.yaml before the apply command or use kubectl edit deployment command afterwards. It may probably be easier if using Helm for this tweak.

# kubectl edit deployment argocd-server
# and add --insecure argument
...
      containers:
      - command:
        - argocd-server
      - args:
        - --insecure
...
# then save and exit. A new pod with --insecure will start and replace the old one

Sample Gateway Schema

apiVersion: networking.istio.io/v1beta1
kind: Gateway
metadata:
  name: argocd-gateway
spec:
  selector:
    istio: ingressgateway
  servers:
    - hosts:
        - argo.example.com
      port:
        name: https
        number: 443
        protocol: HTTPS
      tls:
        mode: SIMPLE
        # argo-cert is a tls secret in istio-system namespace, containing a valid TLS cert for the domain name argo.example.com
        credentialName: argo-cert
    - hosts:
        - argo.example.com
      port:
        name: http
        number: 80
        protocol: HTTP
      tls:
        httpsRedirect: true

I use cert-manager and let’s encrypt to provision free TLS certificates for my personal projects. For more info please see this.

Sample VirtualService Schema

apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
  name: argocd
spec:
  gateways:
    - argocd-gateway
  hosts:
    - argo.example.com
  http:
    - route:
      - destination:
          host: argocd-server
          port:
            number: 80

If the DNS is already working and pointing to the Istio ingress gateway, I can see ArgoCD in my browser with a valid TLS certificate.

🙂

Kubernetes Jobs and Istio

Note: the Job in the title refers to the Job resource in a Kubernetes cluster.

At the time the Istio sidecar doesn’t play well with a Job or a Cronjob, because the istio-proxy might not be ready when the Job starts (which causes connection issues for the job) and won’t exit after the job finishes (which causes the Job stuck and won’t be marked as complete).

Here’s a simple Bash script for a Job assuming the Job’s container has Bash and curl

apiVersion: batch/v1
kind: Job
metadata:
  name: db-migrate
spec:
  template:
    metadata:
      name: db-migrate
    spec:
      restartPolicy: Never
      containers:
        - name: db-migrate
          image: "some-image-with-curl:v0.1"
          command:
            - /bin/bash
            - -c
            - |
              # wait for the istio-proxy to become ready
              until curl -fsI http://localhost:15021/healthz/ready; do
                echo 'Waiting for Sidecar...'
                sleep 1
              done
              # do the job here
              bundle exec rails db:migrate
              # ask the istio-proxy to exit
              curl -fsI -X POST http://localhost:15020/quitquitquit

And if the job container image doesn’t have bash or curl, I used a curl image as another sidecar to get the job done

---
apiVersion: batch/v1
kind: Job
metadata:
  name: db-migrate
spec:
  template:
    metadata:
      name: db-migrate
    spec:
      restartPolicy: Never
      volumes:
        - name: flags
          emptyDir: {}
      containers:
        - name: curl
          image: curlimages/curl:7.78.0
          command:
            - /bin/sh
            - -c
            - |
              # test istio-proxy
              until curl -fsI http://localhost:15021/healthz/ready; do
                echo 'Waiting for Sidecar...'
                sleep 1
              done
              # touch the flag in tmp dir
              touch /tmp/flags/istio-proxy-ready
              # then wait for the job to finish
              until [ -f /tmp/flags/done ]; do
                echo 'Waiting for the job to finish...'
                sleep 1
              done
              # ask istio-proxy to exit
              curl -fsI -X POST http://localhost:15020/quitquitquit
          volumeMounts:
            - name: flags
              mountPath: /tmp/flags
        - name: db-migrate
          image: "some-image-without-curl:v0.1"
          command:
            - /bin/bash
            - -c
            - |
              # wait for the flag of istio-proxy
              until [[ -f /tmp/flags/istio-proxy-ready ]]; do
                echo 'Waiting for Sidecar...'
                sleep 1
              done
              # do the job
              bundle exec rails db:migrate
              # set the flag so curl can shut down istio-proxy
              touch /tmp/flags/done
          volumeMounts:
            - name: flags
              mountPath: /tmp/flags

🙂

Combine GitHub Actions with FluxCD

I started my hobby project SausLink( a tinyURL like URL shortener ) a while ago, as it was really boring during covid lockdowns. The web app itself isn’t anything cutting-edge but I intended to implement full git-ops for this project.

For the CI pipeline, I chose GitHub Actions because it’s easy and also free for open source projects( for a limited builds per day ). The pipeline will be triggered by pushing to dev branch and will build a docker image and then tag it with a GitHub Actions job number. The last step will update the kustomize patch file to refer to the latest tag.

Here’s the pipeline definition along with my comments

# this is .github/workflows/dev.yaml
name: Build the dev image

on:
  push:
    branches: [ dev ]
  pull_request:
    branches: [ dev ]

jobs:
  build:
    runs-on: [ ubuntu-20.04 ]
    steps:
      # Checks-out your repository under $GITHUB_WORKSPACE, so your job can access it
      - uses: actions/[email protected]

      # save a tag like dev-123 to environment variable called build_number
      - name: build number
        run: |
          echo "build_number=dev-${GITHUB_RUN_ID}" >> $GITHUB_ENV

      # build the docker image and tag it with build_number
      - name: Docker build saus-gunicorn
        run: |
          docker build -t ghcr.io/raynix/saus:$build_number .

      # login to ghcr.io and push the docker image
      - name: Docker push
        run: |
          echo ${{ secrets.GHCR_PAT }} | docker login ghcr.io -u $GITHUB_ACTOR --password-stdin
          docker push ghcr.io/raynix/saus:$build_number

      # update the kustomize patch to use the latest tag, using sed command and regex capture
      - name: Update deployment version
        run: |
          sed -i -E "s|(image: ghcr.io/raynix/saus:).*|\1$build_number|g" .kustomize/dev.sausl.ink/patch.yaml

      # add an auto commit with the updated patch file, so the FluxCD can pick it up and deploy with the latest tag
      - name: Auto commit & push changes
        run: |
          git config --global user.name 'raynix Bot'
          git config --global user.email '[email protected]'
          git commit -am "Automated commit"
          git push

At the CD pipeline run by FluxCD v2, a Kustomization resource is defined as follow

---
# this is the definition of the git repo where the kustomize templates are located
apiVersion: source.toolkit.fluxcd.io/v1beta1
kind: GitRepository
metadata:
  name: saus-dev
  namespace: flux-system
spec:
  interval: 4m0s # check the repo every 4 minutes, don't be greedy
  ref:
    branch: dev # fixed to dev branch for dev environment
  url: https://github.com/raynix/saus.git

---
# this is the kustomization definition
apiVersion: kustomize.toolkit.fluxcd.io/v1beta1
kind: Kustomization
metadata:
  name: saus-dev
  namespace: flux-system
spec:
  healthChecks:
  - kind: Deployment
    name: saus
    namespace: saus-dev
  interval: 4m0s # reconcile the deployment every 4 minutes
  path: .kustomize/dev.sausl.ink # directory where the main kustomization.yaml resides in
  prune: true
  sourceRef:
    kind: GitRepository
    name: saus-dev
  timeout: 2m0s
  validation: client
  force: true # force to re-deploy objects which have immutable fields such as jobs

For more FluxCD experiments please see my previous post here. The SausLink’s source code is here. Feel free to fork it and host your own URL shortener 🙂

Use Variables with Kustomize, Part 2

I was looking at the Kustomize variable trick I did a year ago and I think I’ve learned some new tricks worth noting down.

Variables are very handy most of the times, here’s a pattern to define a variable to be used in Kustomize templates and set its value via annotations.

# base/gateway.yaml
apiVersion: networking.istio.io/v1beta1
kind: Gateway
metadata:
  name: wordpress-gateway
spec:
  selector:
    istio: ingressgateway
  servers:
  - hosts:
    # the domain name will be set by the variable
    - $(DOMAIN)
    tls:
      mode: SIMPLE
      credentialName: $(CERT)
    port:
      name: https
      number: 443
      protocol: HTTPS

# base/virtual-service.yaml
apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
  name: wordpress-vs
spec:
  gateways:
    - wordpress-gateway
  hosts:
    # same domain here so no need to repeat the domain name, also the virtual service will always match the gateway
    - $(DOMAIN)
  http:
    - route:
      - destination:
          host: wordpress

# base/config.yaml, ensure variables are enabled for Istio resources
varReference:
  - path: spec/hosts
    kind: VirtualService
  - path: spec/servers/hosts
    kind: Gateway
  - path: spec/servers/tls/credentialName
    kind: Gateway

# at last the base/kustomize.yaml which defines 2 variables: DOMAIN and CERT
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
configurations:
  - config.yaml
resources:
  - gateway.yaml
  - virtual-service.yaml
  # the content of deployment isn't given here, any valid deployment resource should do
  - deployment.yaml 
vars:
  - name: DOMAIN
    objref:
      apiVersion: apps/v1
      kind: Deployment
      name: wordpress
    fieldref:
      fieldpath: metadata.annotations.domain
  - name: CERT
    objref:
      apiVersion: apps/v1
      kind: Deployment
      name: wordpress
    fieldref:
      fieldpath: metadata.annotations.cert

# the above templates form a base Kustomize template and any Kustomize template extending the above base can have the variables set for real values
# site1/kustomize.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
namespace: wordpress-1
bases:
  - ../base

commonAnnotations:
  domain: site1.blog
  cert: site1-blog-cert

There you have it: When building the overlay template site1, variable DOMAIN will have the value of site1.blog; and CERT variable will be set to site1-blog-cert.

🙂