-
The Burnout Effect
Back in October 2015 I got an offer from a big data startup, and after 1 year and 4 months I decided to move on. There’s a 3D printer and a drone in the office and the team was talking about Fallout 4 in the morning because it was just released. I thought the company…
-
Use MySQL Trigger To Do Incremental ETL
There’s a huge MySQL table that I need to ETL to Google BigQuery daily, about 1 billion rows. The rows are updated in a random fashion all the time so I can’t do incremental ETL by the recording the max primary key. Then my colleague brought up the trigger idea, which I believe is the…
-
Install OS X Sierra on VirtualBox on Ubuntu
Following the guide below I installed Sierra in a VirutalBox VM running on Ubuntu quite easily. https://techsviewer.com/install-macos-sierra-virtualbox-windows/ Update: sample vboxmanage commands. vboxmanage modifyvm Sierra –cpuidset 00000001 000106e5 00100800 0098e3fd bfebfbff vboxmanage setextradata Sierra “VBoxInternal/Devices/efi/0/Config/DmiSystemProduct” “iMac11,3” vboxmanage setextradata Sierra “VBoxInternal/Devices/efi/0/Config/DmiSystemVersion” “1.0” vboxmanage setextradata Sierra “VBoxInternal/Devices/efi/0/Config/DmiBoardProduct” “Iloveapple” vboxmanage setextradata Sierra “VBoxInternal/Devices/smc/0/Config/DeviceKey” “ourhardworkbythesewordsguardedpleasedontsteal(c)AppleComputerInc” vboxmanage setextradata Sierra “VBoxInternal/Devices/smc/0/Config/GetKeyFromRealSMC” 1…
-
Install OS X Sierra on Proxmox
My Proxmox running on an old Intel i7 has been upgraded to the latest: 4.4-5. When started the Sierra installer prepared following the above guide, the menu bar was stuck at “Language Chooser” which prevent the launch of DiskUtility or Terminal. The issue isn’t new and has its ticket here: https://github.com/kholia/OSX-KVM/issues/26 But I didn’t find…
-
Distribute cron jobs to hours/minutes with Ansible
This is a handy trick to run a batch of cron jobs on different hour/minute combination so they won’t collide with each other and cause some pressure on the server. The key is to use `with_indexed_items` and Jinja2 math: – name: ansible daily cronjob {{ item.1 }} cron: user=ansible name=ansible-daily-{{ item.1 }} hour={{ item.0…
-
MySQL/ Aurora to Google BigQuery
Google BigQuery(BQ) is Google’s column storage, it’s super fast when you need to query over many gigabytes of data. BQ defaults to utf8, so it makes sense the tables in MySQL/ Aurora are also utf8 encoded. I use Python and SQL Alchemy to load data to BQ. It’s a waste for SA’s ORM features but…
-
Change Ganeti's Network Configuration
“Ganeti is a cluster virtual server management software tool built on top of existing virtualization technologies such as Xen or KVM and other open source software. ” This is how I changed the secondary network configuration using Ganeti command line tools. 1, First, say I need to change the network from 10.0.0.0/24 to 10.1.0.0/16, I…
-
MySQL VS. MariaDB 之日期格式差异
MariaDB Foundation 号称 MariaDB 是 MySQL 的替代品, 但我发现它们之间还是有些不兼容的地方(而且不在已列出的清单里). MariaDB 似乎对日期格式要求更为严格. 以下 SQL 的执行结果就很有不同, 在 MySQL 中 changed_at 是正常的 datetime, 但是在 MariaDB 中 changed_at 完全是 ‘0000-00-00 00:00:00’ 的样子. INSERT INTO … SELECT … GREATEST( COALESCE(`inspection`.`ts`, 0), COALESCE(`inspection_details`.`ts`, 0), COALESCE(`open_time_reason`.`ts`, 0) ) AS `changed_at` ON DUPLICATE KEY UPDATE … `changed_at` = VALUES(`changed_at`) 解决方法是添加一个明确的格式转换: CONVERT(… , DATETIME) CONVERT(…