TLS Full Site Encryption with Istio and Let’s Encrypt

These are steps to easily install TLS certs to a Kubernetes cluster with Istio service mesh as ingress controller, provided by Let’s Encrypt‘s awesome certbot.

Installation of the certbot (on Ubuntu Linux 20.04LTS)

The certbot can be install via snap on Ubuntu Linux

sudo snap install --classic certbot
sudo ln -s /snap/bin/certbot /usr/local/bin/certbot
certbot --version
certbot 1.15.0

By default certbot needs to write to system directories which I thought unnecessary. I use this alias to run certbot as a normal user

mkdir ~/.certbot
alias certbot="certbot --config-dir ~/.certbot/ --work-dir ~/.certbot/ --logs-dir ~/.certbot"

Generate a new cert

Here’s an example to use certbot’s plugin to create certificate for domains hosted at CloudFlare. Here for more info on the plugin.

# install the plugin first
sudo snap set certbot trust-plugin-with-root=ok
sudo snap install certbot-dns-cloudflare

# save a cloudflare API token
echo "dns_cloudflare_api_token = xxxx" > ~/.cloudflare.ini

# generate the cert
# cert and key will be in ~/.certbot/live/raynix.info
certbot certonly --dns-cloudflare -d raynix.info -d '*.raynix.info' --dns-cloudflare-credentials ~/.cloudflare.ini
ls ~/.certbot/live/raynix.info/ -lht
total 4.0K
-rw-rw-r-- 1 ray ray 692 May 10 11:52 README
lrwxrwxrwx 1 ray ray  35 May 10 11:52 cert.pem -> ../../archive/raynix.info/cert1.pem
lrwxrwxrwx 1 ray ray  36 May 10 11:52 chain.pem -> ../../archive/raynix.info/chain1.pem
lrwxrwxrwx 1 ray ray  40 May 10 11:52 fullchain.pem -> ../../archive/raynix.info/fullchain1.pem
lrwxrwxrwx 1 ray ray  38 May 10 11:52 privkey.pem -> ../../archive/raynix.info/privkey1.pem

Install the cert to an Istio gateway

The cert and the key will be put into a Kubernetes secret in istio-system namespace

# assuming kubectl is installed and configured
kubectl create secret -n istio-system tls wild-cert --key ~/.certbot/live/raynix.info/privkey.pem --cert ~/.certbot/live/raynix.info/fullchain.pem

Now the Istio gateway object needs to use this secret as TLS credential

cat <<EOF >gw.yaml
apiVersion: networking.istio.io/v1beta1
kind: Gateway
metadata:
  name: wordpress-gateway
  namespace: wordpress
spec:
  selector:
    # default istio ingress gateway
    istio: ingressgateway
  servers:
  - hosts:
    - raynix.info
    port:
      name: https
      number: 443
      protocol: HTTPS
    tls:
      credentialName: wild-cert
      mode: SIMPLE
  - hosts:
    - raynix.info
    port:
      name: http
      number: 80
      protocol: HTTP
    tls:
      httpsRedirect: true

Then this can be locally tested with curl

curl -v -HHost:raynix.info --resolve "raynix.info:<TLS node port>:<node IP>" "https://raynix.info:<TLS node port>"

🙂

A Kubernetes ClusterSecret

No, at this moment ClusterSecret, unlike ClusterRole, doesn’t officially exist in any version of Kubernetes yet. I’ve seen some discussion like this, so looks like it will be a while to have a ClusterSecret.

But why do I need a ClusterSecret in the first place? The reason is very simple: To be DRY. Imagine I have a few apps deployed into several different namespaces and they all need to pull from my private docker registry. This looks like:

├── namespace-1
│   ├── image-pull-secret
│   └── deployment-app-1
├── namespace-2
│   ├── image-pull-secret
│   └── deployment-app-2
...

It’s a tad straight forward that all the image-pull-secret secrets are the same but as there’s no ClusterSecret they have to be duplicated all over the place. And to make things nicer, if the private registry changes its token, all of these secrets need to be updated at once.

Of course I’m not the first one to be frustrated by this and there are tools built by the community already. ClusterSecret operator is one of them. But when I looked at kubernetes-reflector I immediately liked its simple approach: it can reflects 1 source secret or configmap to many mirror ones in all namespaces! Also it’s easy to integrate with existing SealedSecret operator with reflector.

Here’s how to install kubernetes-reflector quickly with all default settings(copied from its README). I chose to save this file and let my FluxCD to install it for me.

kubectl apply -f https://github.com/emberstack/kubernetes-reflector/releases/latest/download/reflector.yaml

Now I can create a image pull secret for my private docker registry in kube-system namespace and then the reflector will copy it to a few namespaces which match the regex for the namespace whitelist.

The command to create a image pull secret is

kubectl create secret docker-registry image-pull-secret -n kube-system --docker-server=<your-registry-server> --docker-username=<your-name> --docker-password=<your-pword> --docker-email=<your-email>

The full sealed secret command will be

kubectl create secret docker-registry image-pull-secret -n kube-system --docker-server=<your-registry-server> --docker-username=<your-name> --docker-password=<your-pword> --docker-email=<your-email> | \
  kubeseal --controller-namespace=sealed-secrets --controller-name=sealed-secrets -o yaml > image-pull-secret.yaml

Then I’ll add a few magic annotation to let the reflector pick up the job

# this is image-pull-secret.yaml
apiVersion: bitnami.com/v1alpha1
kind: SealedSecret
metadata:
  creationTimestamp: null
  name: image-pull-secret
  namespace: kube-system
spec:
  encryptedData:
    .dockerconfigjson: AgA4E6mcpri...
  template:
    metadata:
      creationTimestamp: null
      name: image-pull-secret
      namespace: kube-system
      annotations:
        reflector.v1.k8s.emberstack.com/reflection-allowed: "true"
        reflector.v1.k8s.emberstack.com/reflection-auto-enabled: "true"
        reflector.v1.k8s.emberstack.com/reflection-auto-namespaces: "wordpress-.*"
status: {}

So when I deploy this file, first the SealedSecret operator will decrypt this into a normal secret with those annotations(note. adding annotations won’t break the encryption, but changing name or namespace could). And then the reflector will create the image-pull-secret secrets in all namespaces which start with wordpress- prefix.

Mission accomplished 🙂

Real Life Cost of a Tesla Model 3

It’s been a bit more than 2 weeks since I got my Model 3 as a result of my impulse-buying, so as an end user I think I’m qualified to do some cost analysis and projection based on the first 1000km I’ve done with the car.

Disclaimer: The following analysis is based on my calculation with my personal data. “Your mileage may vary”. Also this is only concerning money, all those driving experience and zero emission bluff are safely ignored here.

The major saving for driving a Tesla is from the running cost, ie. ‘fuel’ cost. Though the Model 3 Long Range claims a nice 657KM range for a fully charged battery, when I do school runs with AC set to a comfortable 20 degree, the estimated range is about 540KM, which translates to roughly 15kwh/100km.

If I replaced an EV with a Tesla, the saving won’t be great. Fortunately the one got replaced is a 3.0L Diesel SUV which has an average 8.3L/100km fuel economy( and I believe that’s a pretty good one among 3.0L SUVs).

Then the saving per 100km depends on the electricity and diesel fuel prices. There isn’t any way to get around the diesel price, ie. I always fill my tank at nearby petrol station. I just grab the “Don’t pay above” price from RACV‘s data and it’s 132.8c today. The Diesel SUV costs me AUD 11.02 per 100KM.

For electricity however, there are ways to save money: midnight electricity is usually cheap, and it can go even cheaper if charging with solar power during the day. So here’s a table for cost saving grouped by different source of electricity.

The other import saving comes from the 0 maintenance of a Tesla car. Like I stated in my previous click-bait style post, A Tesla car doesn’t need to see a mechanic. For my diesel SUV, there is a major maintenance every 20,000km and a minor one between the major ones and we usually do more than 20,000km a year. Also there were cost to replace parts such as coolant or catalytic converter so for the sake of simplicity I’ll put AUD 1500/year as an average cost and the real figure could be easily bigger.

Some common cost both ICE and EV cars have are ignored, such as rego, insurance, AC filter and tyre, etc. costs.

Electricity SourceRateSaving/100kmSaving/yr(25,000km)Combined Saving
Home/Grid off peak17cAUD 8.47AUD 2,118AUD 3,618
Home/Solar power10.2cAUD 9.49AUD 2,373AUD 3,873
Super charger42cAUD 4.72AUD 1,180AUD 2,680
25,000km/yr is much higher than average but we have decided to do school runs with a car thanks to covid

On the other hand, a Tesla costs a premium and my Model 3 Long Range figure is almost AUD 90,000. My SUV can be sold for at least AUD 20,000, so it’s an AUD 70,000 expense out of my pocket. Depending on finance options, below is a list of overall saving(combined saving – loss to interest)

Finance OptionRateLossOverall Estimated Saving
Cash + 1 year Term1%AUD 700AUD 1,980 ~ 3,173
Cash + Offset2.2%AUD 1,540AUD 1,180 ~ 2,373
Car Finance2%AUD 1,400AUD 1,280 ~ 2,473
Thanks to the super low interest rate, having a term deposit is the worst thing to do.

Of course there’s a possibility that someone bought 35 BTC with AUD 70k a few years ago, and that definitely is the best investment. But other than that, buying a Tesla this year actually saves me a few thousand dollars a year.

EDIT: Another faction is depreciation of the car’s value, here’s some simple research I did with carsales.com.au. Obviously all cars depreciate and it’s best to not own one if possible 😀 but looks like Tesla depreciate slower based on samples from the past 5 years.

ModelDepreciation/yrPrice When NewLoss/yr
Tesla Model S 75D 2016~7%AUD 158kAUD 11,840
BMW 4 435i F33 2016~11%AUD 128kAUD 14,840
Cash~1.68%AUD 90kAUD 1,518
The depreciation rate was calculated roughly based on 2016 car price when new and used car price found in carsales.com.au. Cash depreciation rate is the average inflation rate of 2016 – 2020

🙂

Hello World, Grafana Tanka

I liked YAML a lot, until it gets longer and longer, and even longer.

There are tools to make YAML ‘DRY’, the popular ones are Helm and Kustomize. But none of them can say it got the job done.

To be honest, I didn’t like Helm much from the start. Helm uses templating syntax similar to Jinja2, or PHP, or ASP 20 years ago. It tried to solve a recent problem using an ancient approach: generate text and substitute placeholders with values from variables. Obviously it works and a lot of people are using it, but this bothers me as there are issues that Helm can hardly resolve due to this design:

  • It’s a mixture of YAML and scripts. When working on it, developer has to maintain YAML indentations in the code fragments too, to ensure YAML can be generated as expected.
  • The values of a Helm chart are the parts that can be customized when re-using this chart, this hides the complexity of a full chart from end users but if what the user wanted wasn’t in the values, the user will have to fork the whole chart.

I liked Kustomize a bit more because it doesn’t break the elegance of YAML, ie. the base and overlay Kustomize templates are both valid YAML files. But this is also its Achilles’ heel because YAML is data, not code, it’s Ok to have a few variables, but loops, functions, modules? Those are too much to ask.

I was introduced to Jsonnet a while ago but why should I go back to JSON when I’m better with YAML? However I happened to get to know Grafana Tanka project, after a few minutes’ read I think Tanka has solved most problems which Helm and Kustomize did not. After I watch this presentation(to my surprise the views on this video is shockingly low) I decided to give it a go!

The first step is to install the tanka binary. I personally installed it as a golang module. The go get command looks like:

$ GO111MODULE=on go get github.com/grafana/tanka/cmd/tk
...
# install JsonnetBundler as it's recommended
$ GO111MODULE=on go get github.com/jsonnet-bundler/jsonnet-bundler/cmd/jb
...
tk --version
tk version dev

Then we can initialize a hello-world project for tanka:

$ mkdir tanka-hello
$ cd tanka-hello
$ tk init
GET https://github.com/ksonnet/ksonnet-lib/archive/0d2f82676817bbf9e4acf6495b2090205f323b9f.tar.gz 200
GET https://github.com/grafana/jsonnet-libs/archive/4d4b5b1ce01003547a110f93cc86b8b7afb282a6.tar.gz 200
Directory structure set up! Remember to configure the API endpoint:
`tk env set environments/default --server=https://127.0.0.1:6443`

The initial directory structure looks like this:

$ tree
.
├── environments
│   └── default
│       ├── main.jsonnet # <-- this is where we will do the hello-world
│       └── spec.json
├── jsonnetfile.json
├── jsonnetfile.lock.json
├── lib
│   └── k.libsonnet
└── vendor
    ├── github.com
    │   ├── grafana
    │   │   └── jsonnet-libs
    │   │       └── ksonnet-util
    │   │           ├── grafana.libsonnet
    │   │           ├── kausal.libsonnet
    │   │           ├── k-compat.libsonnet
    │   │           ├── legacy-custom.libsonnet
    │   │           ├── legacy-noname.libsonnet
    │   │           ├── legacy-subtypes.libsonnet
    │   │           ├── legacy-types.libsonnet
    │   │           └── util.libsonnet
    │   └── ksonnet
    │       └── ksonnet-lib
    │           └── ksonnet.beta.4
    │               ├── k8s.libsonnet
    │               └── k.libsonnet
    ├── ksonnet.beta.4 -> github.com/ksonnet/ksonnet-lib/ksonnet.beta.4
    └── ksonnet-util -> github.com/grafana/jsonnet-libs/ksonnet-util

To make this demo more intuitive, I ran an inotifywait command in left pane and vim environments/default/main.jsonnet in right pane

# the inotifywait command explained:
$ inotifywait -q -m -e moved_to  environments/default/ \ # every time the file is saved in vim, an event will be triggered here 
  |while read -r filename event; do \ # the changed filename and event name are read here but we don't really need to do anything about them
    echo -e '\n\n'; tk show environments/default/ # print out generated YAML file 
  done

As shown in the screenshot, after I understood minimum set of tanka syntax, I can get a Deployment YAML out of a few lines of Jsonnet code.

🙂