Tag: Python

  • AWS Lambda with Single CloudFormation Template

    This is just a quick snippet of CloudFormation template to deploy a Python 3.7 Lambda function embedded in the template. The source code inside ZipFile can contain up to 4KB. # CloudFormation templateAWSTemplateFormatVersion: 2010-09-09Resources: LambdaCode: Type: AWS::Lambda::Function DependsOn: – LambdaRole – LambdaPolicy Properties: Code: ZipFile: | import boto3 def handler(event, context): … Role: !GetAtt LambdaRole.Arn […]

  • Use Python to Check Difference between Directories Recursively

    I needed to track differences between software release packages, so that if anything changed dramatically, eg. some file missing or much smaller than expected, I can then get a notification to review the new potentially flawed package. I found that filecmp.dircmp class in Python is spot on for this job. Here’s my snippet to compare […]

  • Manage AWS EBS Snapshot Life Cycle with Lambda

    The timing is not so great. The AWS Data Lifecycle Manager has been announced but I can’t wait for its release. So I decided to use AWS Lambda to do some snapshot lifecycle management. First a role for Lambda having full access to snapshots can be created via the console. To create snapshot with Python […]

  • Saving Images in Chrome with Selenium

    Here’s how to save images in Chrome browser using Selenium. The API has element.screenshot_as_png() method but apparently it’s not implemented at the moment. With some minor changes to this answer I can save a image via the browser: 🙂

  • A Simple Python IO Trick

    I thought using Python’s gzip is quite straight forward, however the IO performance almost doubled with an extra BufferedWriter: import io import gzip def export_to_json_gz(schema): json_file = “test.json.gz” with io.BufferedWriter( gzip.open(temp_dir + json_file, ‘wb’) ) as gzfile: for row in stream_table_data(schema):       ujson.dump(row, gzfile, ensure_ascii=False)       gzfile.write(‘\n’) Now the bottle neck […]

  • MySQL/ Aurora to Google BigQuery

    Google BigQuery(BQ) is Google’s column storage, it’s super fast when you need to query over many gigabytes of data. BQ defaults to utf8, so it makes sense the tables in MySQL/ Aurora are also utf8 encoded. I use Python and SQL Alchemy to load data to BQ. It’s a waste for SA’s ORM features but […]

  • 学TurboGears 2顺便还得学习一下Genshi

    Genshi是Python环境下的一个比较出名的HTML/XML/…模板框架. 作为TurboGears2.1缺省的模板组件, Genshi得到了很多好评. 我也正是因为TG粘合了如此优秀的组件而决心学习TG的. TG2.1样本项目里的主模板文件master.html的完整解释: http://www.turbogears.org/2.1/docs/main/master_html.html#master-html Genshi模板基础: http://genshi.edgewall.org/wiki/Documentation/templates.html 首先, 在TG2中通过@expose decorator来为一个View(V in MVC)指定Genshi模板. 如果不需要模板, 就是: @expose() def hello(self): return “Hello world from controller.” 这样在缺省的开发环境下, 访问http://, 将得到简单的 Hello world from controller. 当然, 这除了测试似乎没有什么实际意义. 然后是使用模板的版本: @expose(‘example.templates.index’) def new_hello(self): return dict(hello=”New text from controller + template “, page=”new_hello”) 这样, TG会寻找example项目下的example/templates/index.html作为new_hello的模板. 同时, 将两个key传递给模板: hello和page. 而在index模板内, 可以很随意的调用这两个获得的key. <h1 py:content=”hello”>old Hello, world.</h1> Now […]

  • 学习Python + TurboGears 2

    前一阵子我着实的困惑了几分钟: 下半辈子做什么呢? 当别人努力的stand out时, 我却在不停的试图blend in. 好在依旧自信, 我索性说, who cares, 我要继续学习我想学的东西: 互联网技术+摄影, 不管有没有搞头, 应该不至于挨饿. 废话到此为止. 选择TG而淘汰Django的原因是TG的耦合度较低, 而且学习难度要大些. 你了解我的, 我喜欢做难度大的事情, 例如投胎到中国. TG目前的版本是2.1, 安装方法可以参考: http://www.turbogears.org/2.1/docs/main/DownloadInstall.html 在Ubuntu环境下, 先安装必要基础: $ sudo aptitude install build-essential python-dev python-setuptools python-virtualenv 然后在CLI里如下方法开始第一个TG项目: $ virtualenv –no-site-packages -p python2.6 tg2env $ cd tg2env/ $ source bin/activate (tg2env)$ easy_install -i http://www.turbogears.org/2.1/downloads/current/index tg.devtools (tg2env)$ paster quickstart example […]