Posted on

Nicer Deployment with Kubernetes

The default strategy to do rolling update in a Kubernetes deployment is to reduce the capacity of current replica set and then add the capacity to the new replica set. This probably means total processing power for the app could be hindered a bit during the deployment.

I’m a bit surprised to find that the default strategy works this way. But luckily it’s not hard to fine tune this. According to the doc here: https://kubernetes.io/docs/concepts/workloads/controllers/deployment/#rolling-update-deployment only a few lines is needed to change the strategy:

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: my-deploy
  namespace: my-project
spec:
  strategy:
    rollingUpdate:
      maxUnavailable: 0
      maxSurge: 40%
  revisionHistoryLimit: 3

By maxUnavailable: 0 this means the total capacity of the deployment will not be reduced. and maxSurge: 40% means new replica can reach 40% of the capacity of the current replica set, before the current one become the old one and drained.

Not a big improvement but revisionHistoryLimit: 3 will only keep 3 replica sets for purpose to roll back a deployment. The default for this is unlimited, which is quite over provisioned, from my point of view.

🙂

Posted on

Ansible, Packer and Configurations

I use Ansible as provisioner for Packer, to build AMIs to be used as a base image of our development environment. When Ansible is used by Packer, it’s not quite intuitive whether it’s using the same ansible.cfg when I run ansible-playbook command in a terminal.

Here’s how to make sure Ansible in Packer session will use the correct ansible.cfg file. 

First, an ENV is supplied in Packer’s template, because ENV precedes any other configuration that can be found:

    {
      "type": "ansible",
      "playbook_file": "../../ansible/apps.yml",
      "ansible_env_vars": ["ANSIBLE_CONFIG=/tmp/ansible.cfg"],
      "extra_arguments": [
        "--skip-tags=packer-skip",
        "-vvv"
      ]
    },

The line with “ANSIBLE_CONFIG=/tmp/ansible.cfg” will tell ansible to use /tmp/ansible.cfg.

With the ansible.cfg at /tmp, and the extra debug switch -vvv I can see in the output if the config file is picked up.


Posted on

Don’t Panic When Kubernetes Master Failed

It was business as usual when I was upgrading our Kubernetes cluster from 1.9.8 to 1.9.10, until it isn’t.

$ kops rolling-update cluster --yes
...
node "ip-10-xx-xx-xx.ap-southeast-2.compute.internal" drained
...
I1024 08:52:50.388672   16009 instancegroups.go:188] Validating the cluster.
...
I1024 08:58:22.725713   16009 instancegroups.go:246] Cluster did not validate, will try again in "30s" until duration "5m0s" expires: error listing nodes: Get https://api.my.kops.domain/api/v1/nodes: dial tcp yy.yy.yy.yy:443: i/o timeout.
E1024 08:58:22.725749   16009 instancegroups.go:193] Cluster did not validate within 5m0s

error validating cluster after removing a node: cluster did not validate within a duation of "5m0s"

From AWS console I can see the new instance for the master is running and the old one has been terminated. There’s 1 catch though, the IP yy.yy.yy.yy is not the IP of the new master instance!

I manually updated the api and api.internal CNAMEs of the Kubernetes cluster in Route 53 and the issue went away quickly. I assume for some reason the DNS update for the new master has failed, but happy to see everything else worked as expected.

🙂

Posted on

A Simple Python Script to Check Difference between Directories

I needed to track differences between software release packages, so that if anything changed dramatically, eg. some file missing or much smaller than expected, I can then get a notification to review the new potentially flawed package.

I found that filecmp.dircmp class in Python is spot on for this job. Here’s my snippet to compare differences of 2 directories recursively:

!/usr/bin/env python3

import argparse
from filecmp import dircmp
from os.path import getsize

changed_files = {}
deleted_files = {}
added_files = {}

def diff_file_size(file1, file2):
  return getsize(file2) - getsize(file1)

def diff_report():
  for k, v in deleted_files.items():
    print(k, v)

  for k, v in added_files.items():
    print(k, v)

  for k, v in changed_files.items():
    print(k, v)

def compare_dir(dir):
  for changed_file in dir.diff_files:
    file1 = "{0}/{1}".format(dir.left, changed_file)
    file2 = "{0}/{1}".format(dir.right, changed_file)
    changed_files[ file2 ] = diff_file_size(file1, file2)

  for deleted_file in dir.left_only:
    file1 = "{0}/{1}".format(dir.right, deleted_file)
    deleted_files[ file1 ] = "DELETED!"

  for added_file in dir.right_only:
    file1 = "{0}/{1}".format(dir.right, added_file)
    added_files[ file1 ] = "ADDED!"

  for sub_dir in dir.subdirs.values():
    compare_dir(sub_dir)

def main():
  parser = argparse.ArgumentParser(description="Usage for diff_dir.py")
  parser.add_argument('--dir1', type=str, required=True)
  parser.add_argument('--dir2', type=str, required=True)
  args = parser.parse_args()

  dir = dircmp(args.dir1, args.dir2)
  compare_dir(dir)
  diff_report()

if name__ == '__main':
  main()

🙂