-
Install Fluentd with Ansible
Fluentd has become the popular open source log aggregration framework for a while. I’ll try to give it a spin with Ansible. There are quite some existing Ansible playbooks to install Fluentd out there, but I would like to do it from scratch just to understand how it works. From the installation guide page, I…
-
The Burnout Effect
Back in October 2015 I got an offer from a big data startup, and after 1 year and 4 months I decided to move on. There’s a 3D printer and a drone in the office and the team was talking about Fallout 4 in the morning because it was just released. I thought the company…
-
Use MySQL Trigger To Do Incremental ETL
There’s a huge MySQL table that I need to ETL to Google BigQuery daily, about 1 billion rows. The rows are updated in a random fashion all the time so I can’t do incremental ETL by the recording the max primary key. Then my colleague brought up the trigger idea, which I believe is the…
-
Install OS X Sierra on VirtualBox on Ubuntu
Following the guide below I installed Sierra in a VirutalBox VM running on Ubuntu quite easily. https://techsviewer.com/install-macos-sierra-virtualbox-windows/ Update: sample vboxmanage commands. vboxmanage modifyvm Sierra –cpuidset 00000001 000106e5 00100800 0098e3fd bfebfbff vboxmanage setextradata Sierra “VBoxInternal/Devices/efi/0/Config/DmiSystemProduct” “iMac11,3” vboxmanage setextradata Sierra “VBoxInternal/Devices/efi/0/Config/DmiSystemVersion” “1.0” vboxmanage setextradata Sierra “VBoxInternal/Devices/efi/0/Config/DmiBoardProduct” “Iloveapple” vboxmanage setextradata Sierra “VBoxInternal/Devices/smc/0/Config/DeviceKey” “ourhardworkbythesewordsguardedpleasedontsteal(c)AppleComputerInc” vboxmanage setextradata Sierra “VBoxInternal/Devices/smc/0/Config/GetKeyFromRealSMC” 1…
-
Install OS X Sierra on Proxmox
My Proxmox running on an old Intel i7 has been upgraded to the latest: 4.4-5. When started the Sierra installer prepared following the above guide, the menu bar was stuck at “Language Chooser” which prevent the launch of DiskUtility or Terminal. The issue isn’t new and has its ticket here: https://github.com/kholia/OSX-KVM/issues/26 But I didn’t find…
-
Distribute cron jobs to hours/minutes with Ansible
This is a handy trick to run a batch of cron jobs on different hour/minute combination so they won’t collide with each other and cause some pressure on the server. The key is to use `with_indexed_items` and Jinja2 math: – name: ansible daily cronjob {{ item.1 }} cron: user=ansible name=ansible-daily-{{ item.1 }} hour={{ item.0…
-
MySQL/ Aurora to Google BigQuery
Google BigQuery(BQ) is Google’s column storage, it’s super fast when you need to query over many gigabytes of data. BQ defaults to utf8, so it makes sense the tables in MySQL/ Aurora are also utf8 encoded. I use Python and SQL Alchemy to load data to BQ. It’s a waste for SA’s ORM features but…
-
Change Ganeti's Network Configuration
“Ganeti is a cluster virtual server management software tool built on top of existing virtualization technologies such as Xen or KVM and other open source software. ” This is how I changed the secondary network configuration using Ganeti command line tools. 1, First, say I need to change the network from 10.0.0.0/24 to 10.1.0.0/16, I…