Playing with Kubernetes Ingress Controller

It’s very very easy to use Kubernetes(K8s) to provision an external service with AWS ELB, there’s one catch though(at least for now in 2018).

AWS ELB is usually used with an auto scaling group and a launch configuration. However with K8s, EC2 instances won’t get spun directly, only pods will, which is call Horizontal Scaling. K8s will issue AWS API calls to update the ELBs so there’s no need for auto scaling groups or launch configurations.

This worked like a charm until when things got busy. There was a brief down time on one of the ELBs managed by K8s, because all instances at the back of the ELB were marked as unhealthy! Of course they were healthy at that moment. With help from AWS Support team, the culprit seems to be similar to this case: https://github.com/kubernetes/kubernetes/issues/47067.

Luckily for me, I had a gut feel that the simple ELB implementation isn’t the best practice and started to adopt the K8s Ingress Controller. And in this case I believe ingress can avoid the down time because the routing is done internally in K8s cluster which doesn’t involving AWS API calls. Nonetheless ingress can use 1 ELB for many apps and that’s good because ELBs are expensive.

Here are steps to deploy an nginx ingress controller as an http(L7) load balancer:

Deploy the mandatory schema, the default replica number for the controller is 2, I changed it to 3 to have 1 in each availability zone:

kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/master/deploy/mandatory.yaml

Some customisation for L7 load balancer on AWS, remember to use your SSL cert if you need https termination:

kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/master/deploy/provider/aws/service-l7.yaml
kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/master/deploy/provider/aws/patch-configmap-l7.yaml

Then an ingress for an app can be deployed:

$ cat .k8s/prod/ingress.yaml 
---
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: my-ingress
  namespace: my-prod
  annotations:
    kubernetes.io/ingress.class: prod
spec:
  rules:
    - host: my.domain.elb
      http:
        paths:
          - path: /
            backend:
              serviceName: my-service
              servicePort: 80
    - host: my.domain.cdn
      http:
        paths:
          - path: /
            backend:
              serviceName: my-service
              servicePort: 80

Notes:

  • my-service is a common NodePort service and has port 80 exposed
  • io/ingress.class is for multiple ingress controllers in same k8s cluster, eg. 1 for dev and the other for prod
  • for now I have to duplicate the host block for each domain, because wildcard or regex are not supported by k8s ingress specification
  • at last, find the ELB this ingress controller created, then point my.domain.elb to it, then the CDN domain can use my.domain.elb as origin.

🙂

Time Machine for Arch Linux

I’ve been using Arch Linux for some years, and it’s still my favorite Linux distribution. The feature that distinguished Arch from others is its rolling release which means there’s no such a thing called version in Arch. Using latest packages in Arch is the norm.

However living on the edge means it’s not quite safe. After I installed a bunch of updates including Gnome Shell 3.28, my XPS 15 laptop had trouble to bring up external monitor. It even froze when I plug the HDMI in hot.

I tried to revert some packages like

sudo pacman -U /var/cache/packman/pkg/some-package-1.0.xx.pkg.tar.xz

But it didn’t solve the problem because there were hundreds of packages in last update.

Almost going to panic, I found this instruction to revert all packages to a snapshot in time. And it actually worked wonders for me.

Only surprise is when downgrading packages, I saw errors like

 package-name: /path/to/package-file exists in filesystem

Guess it’s a safe guarding mechanism of pacman but since I know what I was doing so I simply deleted those files. The final command is

sudo pacman -Syyuu

which will bring Arch Linux back to a point of time and the issue has been fixed 🙂

Kubernetes External Service with HTTPS

This is a quick example to assign an SSL certificate to a Kubernetes external service(which is an ELB in AWS). Tested with kops 1.8 and kubernetes 1.8.

---
apiVersion: v1
kind: Service
metadata:
 name: my-https-service
 namespace: my-project
 labels:
   app: my-website-ssl
 annotations:
   service.beta.kubernetes.io/aws-load-balancer-ssl-cert: "arn:aws:acm:ap-southeast-2:xxx:certificate/xxx..."
   service.beta.kubernetes.io/aws-load-balancer-backend-protocol: "http"
   service.beta.kubernetes.io/aws-load-balancer-ssl-ports: "https"
   service.beta.kubernetes.io/aws-load-balancer-connection-idle-timeout: '3600'
spec:
 type: LoadBalancer
 selector:
   app: my-website
 ports:
   - name: http
     port: 80
     targetPort: 80
   - name: https
     port: 443
     targetPort: 80

🙂