istio – raynix 筆記

September 23, 2021

Run ArgoCD with Istio Service Mesh in a Kubernetes Cluster

It’s been quite a while since I installed Flux CD V2 in my garage Kubernetes lab, as there’s a lot of debate going on between Flux and ArgoCD I decided to give ArgoCD a go. The other reason to try ArgoCD is that it supports Jsonnet.

By default installation, ArgoCD will use self-signed TLS certificate and enforce TLS connection, which means users get to see the security warning and have to trust the certificate to continue. Naturally with Istio handles ingress and TLS termination, I would like to enable Istio sidecar for ArgoCD and run it in HTTP mode.

Here are the steps to configure and install ArgoCD along side with Istio:

Enalbe Istio Sidecar

I choose to enable automatic Istio sidecar injection for ArgoCD’s namespace.

# create the namespace, by default it's argocd
kubectl create namespace argocd
# turn on istio injection
kubectl label namespace argocd istio-injection=enabled

Install ArgoCD the Normal Way

kubectl apply -n argocd -f https://raw.githubusercontent.com/argoproj/argo-cd/stable/manifests/install.yaml

Disable TLS for argocd-server Deployment

This can be done before or after the deployment being applied to the cluster in the above step, eg. edit the install.yaml before the apply command or use kubectl edit deployment command afterwards. It may probably be easier if using Helm for this tweak.

# kubectl edit deployment argocd-server
# and add --insecure argument
...
      containers:
      - command:
        - argocd-server
      - args:
        - --insecure
...
# then save and exit. A new pod with --insecure will start and replace the old one

Sample Gateway Schema

apiVersion: networking.istio.io/v1beta1
kind: Gateway
metadata:
  name: argocd-gateway
spec:
  selector:
    istio: ingressgateway
  servers:
    - hosts:
        - argo.example.com
      port:
        name: https
        number: 443
        protocol: HTTPS
      tls:
        mode: SIMPLE
        # argo-cert is a tls secret in istio-system namespace, containing a valid TLS cert for the domain name argo.example.com
        credentialName: argo-cert
    - hosts:
        - argo.example.com
      port:
        name: http
        number: 80
        protocol: HTTP
      tls:
        httpsRedirect: true

I use cert-manager and let’s encrypt to provision free TLS certificates for my personal projects. For more info please see this.

Sample VirtualService Schema

apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
  name: argocd
spec:
  gateways:
    - argocd-gateway
  hosts:
    - argo.example.com
  http:
    - route:
      - destination:
          host: argocd-server
          port:
            number: 80

If the DNS is already working and pointing to the Istio ingress gateway, I can see ArgoCD in my browser with a valid TLS certificate.

🙂

August 29, 2021August 29, 2021

Kubernetes Jobs and Istio

Note: the Job in the title refers to the Job resource in a Kubernetes cluster.

At the time the Istio sidecar doesn’t play well with a Job or a Cronjob, because the istio-proxy might not be ready when the Job starts (which causes connection issues for the job) and won’t exit after the job finishes (which causes the Job stuck and won’t be marked as complete).

Here’s a simple Bash script for a Job assuming the Job’s container has Bash and curl

apiVersion: batch/v1
kind: Job
metadata:
  name: db-migrate
spec:
  template:
    metadata:
      name: db-migrate
    spec:
      restartPolicy: Never
      containers:
        - name: db-migrate
          image: "some-image-with-curl:v0.1"
          command:
            - /bin/bash
            - -c
            - |
              # wait for the istio-proxy to become ready
              until curl -fsI http://localhost:15021/healthz/ready; do
                echo 'Waiting for Sidecar...'
                sleep 1
              done
              # do the job here
              bundle exec rails db:migrate
              # ask the istio-proxy to exit
              curl -fsI -X POST http://localhost:15020/quitquitquit

And if the job container image doesn’t have bash or curl, I used a curl image as another sidecar to get the job done

---
apiVersion: batch/v1
kind: Job
metadata:
  name: db-migrate
spec:
  template:
    metadata:
      name: db-migrate
    spec:
      restartPolicy: Never
      volumes:
        - name: flags
          emptyDir: {}
      containers:
        - name: curl
          image: curlimages/curl:7.78.0
          command:
            - /bin/sh
            - -c
            - |
              # test istio-proxy
              until curl -fsI http://localhost:15021/healthz/ready; do
                echo 'Waiting for Sidecar...'
                sleep 1
              done
              # touch the flag in tmp dir
              touch /tmp/flags/istio-proxy-ready
              # then wait for the job to finish
              until [ -f /tmp/flags/done ]; do
                echo 'Waiting for the job to finish...'
                sleep 1
              done
              # ask istio-proxy to exit
              curl -fsI -X POST http://localhost:15020/quitquitquit
          volumeMounts:
            - name: flags
              mountPath: /tmp/flags
        - name: db-migrate
          image: "some-image-without-curl:v0.1"
          command:
            - /bin/bash
            - -c
            - |
              # wait for the flag of istio-proxy
              until [[ -f /tmp/flags/istio-proxy-ready ]]; do
                echo 'Waiting for Sidecar...'
                sleep 1
              done
              # do the job
              bundle exec rails db:migrate
              # set the flag so curl can shut down istio-proxy
              touch /tmp/flags/done
          volumeMounts:
            - name: flags
              mountPath: /tmp/flags

🙂

May 14, 2021August 22, 2021

TLS Full Site Encryption with Istio and Let’s Encrypt

These are steps to easily install TLS certs to a Kubernetes cluster with Istio service mesh as ingress controller, provided by Let’s Encrypt‘s awesome certbot.

Installation of the certbot (on Ubuntu Linux 20.04LTS)

The certbot can be install via snap on Ubuntu Linux

sudo snap install --classic certbot
sudo ln -s /snap/bin/certbot /usr/local/bin/certbot
certbot --version
certbot 1.15.0

By default certbot needs to write to system directories which I thought unnecessary. I use this alias to run certbot as a normal user

mkdir ~/.certbot
alias certbot="certbot --config-dir ~/.certbot/ --work-dir ~/.certbot/ --logs-dir ~/.certbot"

Generate a new cert

Here’s an example to use certbot’s plugin to create certificate for domains hosted at CloudFlare. Here for more info on the plugin.

# install the plugin first
sudo snap set certbot trust-plugin-with-root=ok
sudo snap install certbot-dns-cloudflare

# save a cloudflare API token
echo "dns_cloudflare_api_token = xxxx" > ~/.cloudflare.ini

# generate the cert
# cert and key will be in ~/.certbot/live/raynix.info
certbot certonly --dns-cloudflare -d raynix.info -d '*.raynix.info' --dns-cloudflare-credentials ~/.cloudflare.ini
ls ~/.certbot/live/raynix.info/ -lht
total 4.0K
-rw-rw-r-- 1 ray ray 692 May 10 11:52 README
lrwxrwxrwx 1 ray ray  35 May 10 11:52 cert.pem -> ../../archive/raynix.info/cert1.pem
lrwxrwxrwx 1 ray ray  36 May 10 11:52 chain.pem -> ../../archive/raynix.info/chain1.pem
lrwxrwxrwx 1 ray ray  40 May 10 11:52 fullchain.pem -> ../../archive/raynix.info/fullchain1.pem
lrwxrwxrwx 1 ray ray  38 May 10 11:52 privkey.pem -> ../../archive/raynix.info/privkey1.pem

Install the cert to an Istio gateway

The cert and the key will be put into a Kubernetes secret in istio-system namespace

# assuming kubectl is installed and configured
kubectl create secret -n istio-system tls wild-cert --key ~/.certbot/live/raynix.info/privkey.pem --cert ~/.certbot/live/raynix.info/fullchain.pem

Now the Istio gateway object needs to use this secret as TLS credential

cat <<EOF >gw.yaml
apiVersion: networking.istio.io/v1beta1
kind: Gateway
metadata:
  name: wordpress-gateway
  namespace: wordpress
spec:
  selector:
    # default istio ingress gateway
    istio: ingressgateway
  servers:
  - hosts:
    - raynix.info
    port:
      name: https
      number: 443
      protocol: HTTPS
    tls:
      credentialName: wild-cert
      mode: SIMPLE
  - hosts:
    - raynix.info
    port:
      name: http
      number: 80
      protocol: HTTP
    tls:
      httpsRedirect: true

Then this can be locally tested with curl

curl -v -HHost:raynix.info --resolve "raynix.info:<TLS node port>:<node IP>" "https://raynix.info:<TLS node port>"

🙂