Category: Systems Admin

  • Use MySQL Trigger To Do Incremental ETL

    There’s a huge MySQL table that I need to ETL to Google BigQuery daily, about 1 billion rows. The rows are updated in a random fashion all the time so I can’t do incremental ETL by the recording the max primary key. Then my colleague brought up the trigger idea, which I believe is the […]

  • Install OS X Sierra on VirtualBox on Ubuntu

    Following the guide below I installed Sierra in a VirutalBox VM running on Ubuntu quite easily. https://techsviewer.com/install-macos-sierra-virtualbox-windows/ Update: sample vboxmanage commands. vboxmanage modifyvm Sierra –cpuidset 00000001 000106e5 00100800 0098e3fd bfebfbff vboxmanage setextradata Sierra “VBoxInternal/Devices/efi/0/Config/DmiSystemProduct” “iMac11,3” vboxmanage setextradata Sierra “VBoxInternal/Devices/efi/0/Config/DmiSystemVersion” “1.0” vboxmanage setextradata Sierra “VBoxInternal/Devices/efi/0/Config/DmiBoardProduct” “Iloveapple” vboxmanage setextradata Sierra “VBoxInternal/Devices/smc/0/Config/DeviceKey” “ourhardworkbythesewordsguardedpleasedontsteal(c)AppleComputerInc” vboxmanage setextradata Sierra “VBoxInternal/Devices/smc/0/Config/GetKeyFromRealSMC” 1 […]

  • Install OS X Sierra on Proxmox

    My Proxmox running on an old Intel i7 has been upgraded to the latest: 4.4-5. When started the Sierra installer prepared following the above guide, the menu bar was stuck at “Language Chooser” which prevent the launch of DiskUtility or Terminal. The issue isn’t new and has its ticket here: https://github.com/kholia/OSX-KVM/issues/26 But I didn’t find […]

  • Distribute cron jobs to hours/minutes with Ansible

    This is a handy trick to run a batch of cron jobs on different hour/minute combination so they won’t collide with each other and cause some pressure on the server. The key is to use `with_indexed_items` and Jinja2 math: – name: ansible daily cronjob {{ item.1 }}   cron: user=ansible name=ansible-daily-{{ item.1 }} hour={{ item.0 […]

  • MySQL/ Aurora to Google BigQuery

    Google BigQuery(BQ) is Google’s column storage, it’s super fast when you need to query over many gigabytes of data. BQ defaults to utf8, so it makes sense the tables in MySQL/ Aurora are also utf8 encoded. I use Python and SQL Alchemy to load data to BQ. It’s a waste for SA’s ORM features but […]

  • Change Ganeti's Network Configuration

    “Ganeti is a cluster virtual server management software tool built on top of existing virtualization technologies such as Xen or KVM and other open source software. ” This is how I changed the secondary network configuration using Ganeti command line tools. 1, First, say I need to change the network from 10.0.0.0/24 to 10.1.0.0/16, I […]

  • MySQL VS. MariaDB 之日期格式差异

    MariaDB Foundation 号称 MariaDB 是 MySQL 的替代品, 但我发现它们之间还是有些不兼容的地方(而且不在已列出的清单里). MariaDB 似乎对日期格式要求更为严格. 以下 SQL 的执行结果就很有不同, 在 MySQL 中 changed_at 是正常的 datetime, 但是在 MariaDB 中 changed_at 完全是 ‘0000-00-00 00:00:00’ 的样子. INSERT INTO … SELECT … GREATEST( COALESCE(`inspection`.`ts`, 0), COALESCE(`inspection_details`.`ts`, 0), COALESCE(`open_time_reason`.`ts`, 0) ) AS `changed_at` ON DUPLICATE KEY UPDATE … `changed_at` = VALUES(`changed_at`) 解决方法是添加一个明确的格式转换: CONVERT(… , DATETIME) CONVERT( […]

  • 小试 MariaDB Galera Cluster

    前些时, 难得老板关注新技术, 哈哈, 我于是有机会尝试一下数据库服务器集群. 什么是 Galera Cluster? 简单的说就是3个或以上的 MariaDB 服务器相互作为镜像. 详细介绍在这里. 我按照 Digital Ocean 的指点, 用 AWS 上 3 个虚拟机做了个最小的集群, 下面是我的一些心得(针对 MariaDB 10.0.17): 首先集群的成员个数一定是奇数, 也就3, 5, 7… 因为有的时候个别成员的数据出现异样的时候, 整个集群会以”少数服从多数”的方式修正数据. 虽然集群是 master – master 方式进行数据同步的, 但如果多个成员同时更新相同的数据集合时, 集群内就很容易出现冲突, 这也就降低了性能以及可靠度. 目前我觉得可行的应用方式, 就是将所有的数据更新(insert, update, delete)都集中到一个成员上, 然后所有的查询(select)可以由所有成员分担. 就好像传统的 master – slave 方式, 嗯, 比较失望, 不过至少倚靠 HAProxy, 当一个成员挂了会自动有另一个来接替, 也不会耽误太多的事情. 估计还有一段距离才能把 Galera […]