The Burnout Effect

Back in October 2015 I got an offer from a big data startup, and after 1 year and 4 months I decided to move on.

There’s a 3D printer and a drone in the office and the team was talking about Fallout 4 in the morning because it was just released. I thought the company and the team were very cool and I still think so now.

My first challenge was to migrate a self-hosted MySQL database to AWS Aurora, because the MySQL server was over stretched and felt like it could collapse anytime soon. I was quite experienced at MySQL so I didn’t think that would be hard. However there were some complications: The MySQL was a huge VM managed by Ganeti cluster and backed by DRBD volumes and the best of all was that the disks were old school magnetic SAS disks.

The DB migration tool recommended by AWS just failed randomly on large tables(~400GB). It wasn’t acceptable to do a full mysqldump(~1TB) and setup replication to Aurora because that will cause huge downtime. And since there’s no access to the Aurora’s system, the option to use LVM snapshot was out too. I thought there must have a way so I created a MySQL replica in the same cluster using LVM snapshot, then setup replication between the replica and the master.

After the replication was done and verified between the master and the replica, I had the opportunity to pause the replication and do mysqldump on the replica and then setup another replication between the replica MySQL and Aurora. After the replica  caught  up with the master we did the DNS switch-over and the apps almost did feel a thing and started to update on the Aurora. This concluded the first success.

I was involved in several big projects such as migration of DNS to route53, migration of core servers(about 40) to AWS and migration of data warehouse from AWS Redshift to Google BigQuery in 2016.

I thought the job should have become more comfortable since a large portion of the infrastructure had been rebuilt. However since a few months ago I started to have poor sleeps, and in daytime poor concentration. I searched for the answer, to my surprise, a lot of people shared the same issue which is called burnout. So rather than being asked to leave for poor performance, I choose to have a break and search for a new job.

On the last day of the job, we played Rocket League together and those were my best hours. I felt super relieved, yet very sad to leave the team. Thanks to the team especially Trist and Adam who I learned a lot from.

🙂

快车道上的2016

我们的2016年仿佛在快进模式中完成,简单回顾一下吧:

终于我们去了传说中的大洋路和12门徒!景色无比美丽,但苍蝇却着实多,估计是托附近众多农场的福。

工作方面,这是我丰收的一年。首先我完成了TB容量的数据库在线迁移,从美国加州某数据中心的服务器搬到AWS Aurora,让同事们基本上告别了数据库过载造成的麻烦,没用“完全”一词因为 Aurora 有时候也会被放倒。接下来的一个大项目可想而知,把私有云上的所有服务器搬家到AWS EC2。这两个项目就花了几个月,而且是我的前任以及前任的前任都没搞定的事情,因此我小小的有点自豪。有趣的是之前我们要求那个数据中心的工程师搭建一条数据中心到AWS的VPN专线,结果迟迟没有完成,这也是老板跟他们翻船的主要原因。后来我用了半天时间连读文档带执行就完成了AWS到公司局域网的VPN,我为自己的学习能力又稍微的自豪了一把。

另外一个大项目就是大数据仓储。公司之前使用 AWS Redshift,但性价比达不到要求。于是我接受了挑战,试用Google BigQuery,并把现有的为Redshift编写的程序和SQL改写一遍。改写还不算难,困难的是让新程序能把数据处理结果上传到BQ并保证两边的数据一致。比较数据也不难,做diff就是了,但30+TB的就不那么随便了,一旦一个月有那么几天的数据对不上,就要一个个环节的去排查。经过几个星期的排错,最后数据终于完全吻合了,100%吻合。BQ的优势相当明显,特别是对于不规律的大数据计算,用多少就付款多少,不需要评估CPU什么的。

其他琐碎的事情就不提了。

两个孩子各有进步。洋洋自打出院以后就很努力的吃饭和睡觉,体重慢慢接近中等水平了。笑笑的NAPLAN 考试成绩意外的好,因为我们俩关注她学习的时间无可奈何的很少。老婆给笑笑找了美术老师,一切都被她计划的很好,我给她点赞 🙂

结婚十年了, 再夸一下老婆

距离上一次夸老婆已经有五年了, 时间太久了不夸恐怕有害健康.

三年前老婆刚开始她在猫本的学业时, 我对于她她的功课没多少了解. 有一次我翻她的一本教科书, 看了一会儿我就困了:通篇都是密密麻麻的英文. 首先要看懂英文, 然后才可以去尝试理解文中的专业知识. 老婆的英文阅读并不是很棒, 她一开始相当吃力, 后来逐渐发力, 最后以优异的成绩毕业了!同时她还要照顾孩子做家务的, 可见老婆对时间的充分利用.

这半年我们对笑笑的成长不是很满意, 因为比起搬家前在 OSPS 上学时, 笑笑变得更贪玩, 难以自律, 而且在求知欲方面似乎也倒退了. 我和老婆谈起这事时, 自然我们会有各种猜测, 但后来我们都想到了搬家后笑笑的小学 SCPS. 查看学校教育水平排名 OSPS 要比 SCPS 高很多, 而且我们去学校的时候也能注意到 SCPS 小朋友在守纪律方面要差很多, 老师也从未邀请家长陪孩子晨读等等.

家附近有一所很不错的天主教小学, 看排名比 OSPS 还好. 我看了一下网站, 叹气的说: “不行吧, 人家只收天主教家庭的孩子, 而且只有5岁的才收.” 老婆并没放弃, 说: “我明天去拜访一下校长.” 结果很意外, 笑笑可以转学过来. 看来学校网站需要更新了呵呵.

这次二宝大病被诊断时, 我基本已呆, 神情恍惚了好一阵子, 还是老婆一直安慰我, 说这里的医生水平都很高的, 肯定能治好的. 这我才回过神来… 二宝住院这一个多月里, 也都是老婆在陪护. 由于心情很糟糕,  睡眠也不足, 老婆的胃口也很差. 可她依旧很努力的给二宝准备母乳. 医院里有专门存放小病号口粮的冰柜, 每天有营养专家给小宝宝们准备配好的奶水, 绿色标签的是母乳+配方奶, 白色标签的是纯配方奶. 二宝的奶瓶标签基本上一直是绿色的, 在众多白色标签里也非常好找.

老爸老妈即将回国了, 老婆不仅给订好了机票, 而且转机的酒店也一并订好了. 老婆做事认真而且周到, 这也是我要向她学习的.

这次由于二宝住院, 老婆的生日也在医院里简单的度过了. 希望二宝尽快出院, 我也好给老婆补一个像样的生日爬梯.