Posted on

Run Google Lighthouse in Docker Container

Thanks to my Colleague Simon’s suggestion, I was introduced to Google Lighthouse, an opensource nodejs framework to use Google Chrome to audit a website’s performance.

I like Lighthouse because:

  • opensource
  • good portability
  • can run as CLI command or as a nodejs module

Here’s a sample Dockerfile to have a container ready to run Lighthouse with Google Chrome for Linux.

FROM debian:stretch

USER root
WORKDIR /root
ENV CHROME_VERSION="google-chrome-stable"

# system packages
RUN apt update -qqy && \
  apt install -qqy build-essential gnupg wget curl jq

# nodejs 10
RUN curl -sL https://deb.nodesource.com/setup_10.x | bash - && \
  apt install -qqy nodejs && \
  npm install -g lighthouse

# google-chrome
RUN wget -q -O - https://dl-ssl.google.com/linux/linux_signing_key.pub | apt-key add - && \
  echo "deb http://dl.google.com/linux/chrome/deb/ stable main" >> /etc/apt/sources.list.d/google-chrome.list && \
  apt update -qqy && \
  apt install -qqy ${CHROME_VERSION:-google-chrome-stable}

# python3 (optional for metric processing)
RUN apt install -qqy python3 python3-pip && \
  pip3 install influxdb

# lighthouse
RUN useradd -ms /bin/bash lighthouse
USER lighthouse
WORKDIR /home/lighthouse

Then lighthouse can be executed in the container to audit $url:

CHROME_PATH=$(which google-chrome) lighthouse $url --emulated-form-factor=none --output=json --chrome-flags="--headless --no-sandbox"

The result json will be sent to stdout, and it can be easily piped to other scripts for post processing, eg. parse json and extract metrics, etc…

🙂

Posted on

A Simple Python Script to Check Difference between Directories

I needed to track differences between software release packages, so that if anything changed dramatically, eg. some file missing or much smaller than expected, I can then get a notification to review the new potentially flawed package.

I found that filecmp.dircmp class in Python is spot on for this job. Here’s my snippet to compare differences of 2 directories recursively:

!/usr/bin/env python3

import argparse
from filecmp import dircmp
from os.path import getsize

changed_files = {}
deleted_files = {}
added_files = {}

def diff_file_size(file1, file2):
  return getsize(file2) - getsize(file1)

def diff_report():
  for k, v in deleted_files.items():
    print(k, v)

  for k, v in added_files.items():
    print(k, v)

  for k, v in changed_files.items():
    print(k, v)

def compare_dir(dir):
  for changed_file in dir.diff_files:
    file1 = "{0}/{1}".format(dir.left, changed_file)
    file2 = "{0}/{1}".format(dir.right, changed_file)
    changed_files[ file2 ] = diff_file_size(file1, file2)

  for deleted_file in dir.left_only:
    file1 = "{0}/{1}".format(dir.right, deleted_file)
    deleted_files[ file1 ] = "DELETED!"

  for added_file in dir.right_only:
    file1 = "{0}/{1}".format(dir.right, added_file)
    added_files[ file1 ] = "ADDED!"

  for sub_dir in dir.subdirs.values():
    compare_dir(sub_dir)

def main():
  parser = argparse.ArgumentParser(description="Usage for diff_dir.py")
  parser.add_argument('--dir1', type=str, required=True)
  parser.add_argument('--dir2', type=str, required=True)
  args = parser.parse_args()

  dir = dircmp(args.dir1, args.dir2)
  compare_dir(dir)
  diff_report()

if name__ == '__main':
  main()

🙂

Posted on

Upload Limit in Kubernetes Nginx Ingress Controller

According to https://github.com/nginxinc/kubernetes-ingress/issues/21#issuecomment-408618569, this is how to lift the upload limit in Nginx Ingress Controller for Kubernetes after recent update to the project:

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: test-project-ingress
  namespace: test-project-dev
  annotations:
    kubernetes.io/ingress.class: dev
    nginx.ingress.kubernetes.io/proxy-body-size: 200m
spec:
  rules:
    - host: test-project.dev.com
      http:
        paths:
          - path: /
            backend:
              serviceName: test-project
              servicePort: 80

And for now the nginx pods have to be restarted before this can take effect. Hope this won’t be necessary in future

🙂

Posted on

Auto Scaling in Kubernetes 1.9

I updated my Kubernetes cluster from 1.8 to 1.9 recently, the upgrade process is very smooth, however the auto-scaling part seemed to be failing. Below are some notes on how I troubleshoot this issue.

First to ensure I have both kops and kubectl upgraded to 1.9 on my laptop:

Install kubectl 1.9:

curl -LO https://storage.googleapis.com/kubernetes-release/release/v1.9.10/bin/linux/amd64/kubect

Install kops 1.9: https://github.com/kubernetes/kops/releases/tag/1.9.2

I was doing some load testing and I discovered that no matter how busy the pods were, they weren’t scaled out. To see what’s happening with the horizontal pod autoscaler(HPA), I use the following command:

kubectl describe hpa
...
  ScalingActive  False   FailedGetResourceMetric  the HPA was unable to compute the replica count: unable to get metrics for resource cpu: unable to fetch metrics from API: the server could not find the requested resource (get pods.metrics.k8s.io)

After some googling around, it turns out that Kubernetes 1.9 uses new metrics server for HPA, and my cluster didn’t have it. Here’s how to install metrics server for kubernetes cluster: https://github.com/kubernetes-incubator/metrics-server

To make this troubleshooting more interesting, the metrics server encountered error too! Looks like:

Failed to get kubernetes address: No kubernetes source found.

Bug fix for the metrics server:  https://github.com/kubernetes-incubator/metrics-server/issues/105#issuecomment-412818944

In short, adding a overriding command in `deploy/1.8+/metrics-server-deployment.yaml` got it working:

        command:
        - /metrics-server
        - --source=kubernetes.summary_api:''

Install cluster autoscaler for kubernetes cluster: https://github.com/kubernetes/autoscaler/blob/master/cluster-autoscaler/cloudprovider/aws/examples/cluster-autoscaler-one-asg.yaml I used `- image: k8s.gcr.io/cluster-autoscaler:v1.1.3` for Kubernetes 1.9. This part was without any surprises and worked as expected.

Sample HPA schema:

---
apiVersion: autoscaling/v1
kind: HorizontalPodAutoscaler
metadata:
  name: test-hpa
  namespace: test
spec:
  scaleTargetRef:
    apiVersion: apps/v1beta1
    kind: Deployment
    name: test-deploy
  minReplicas: 2
  maxReplicas: 10
  targetCPUUtilizationPercentage: 50

🙂