Tag: Python

  • A Simple RabbitMQ Consumer with Python

    Here’s a very simple RabbitMQ consumer in Python, deployed in a Kubernetes cluster. I used this to act as a consumer for testing purposes. First the Python code: The above code can be saved as a ConfigMap in the cluster: Then the Kubernetes manifest: At last this can be simply deployed with 🙂

  • How to Easily Enable Webhook Mode for Python Telegram Bots

    I created a Telegram bot(@wordsquad_bot) using python-telegram-bot framework. There are 2 modes to run the bot: Since my bot has got almost all features and run stably for a while, I decided to setup webhook for it. I came across a few articles and did a bit of troubleshooting myself so here’s probably the easiest […]

  • AWS Lambda with Single CloudFormation Template

    This is just a quick snippet of CloudFormation template to deploy a Python 3.7 Lambda function embedded in the template. The source code inside ZipFile can contain up to 4KB. # CloudFormation templateAWSTemplateFormatVersion: 2010-09-09Resources: LambdaCode: Type: AWS::Lambda::Function DependsOn: – LambdaRole – LambdaPolicy Properties: Code: ZipFile: | import boto3 def handler(event, context): … Role: !GetAtt LambdaRole.Arn […]

  • Use Python to Check Difference between Directories Recursively

    I needed to track differences between software release packages, so that if anything changed dramatically, eg. some file missing or much smaller than expected, I can then get a notification to review the new potentially flawed package. I found that filecmp.dircmp class in Python is spot on for this job. Here’s my snippet to compare […]

  • Manage AWS EBS Snapshot Life Cycle with Lambda

    The timing is not so great. The AWS Data Lifecycle Manager has been announced but I can’t wait for its release. So I decided to use AWS Lambda to do some snapshot lifecycle management. First a role for Lambda having full access to snapshots can be created via the console. To create snapshot with Python […]

  • Saving Images in Chrome with Selenium

    Here’s how to save images in Chrome browser using Selenium. The API has element.screenshot_as_png() method but apparently it’s not implemented at the moment. With some minor changes to this answer I can save a image via the browser: 🙂

  • A Simple Python IO Trick

    I thought using Python’s gzip is quite straight forward, however the IO performance almost doubled with an extra BufferedWriter: import io import gzip def export_to_json_gz(schema): json_file = “test.json.gz” with io.BufferedWriter( gzip.open(temp_dir + json_file, ‘wb’) ) as gzfile: for row in stream_table_data(schema):       ujson.dump(row, gzfile, ensure_ascii=False)       gzfile.write(‘\n’) Now the bottle neck […]

  • MySQL/ Aurora to Google BigQuery

    Google BigQuery(BQ) is Google’s column storage, it’s super fast when you need to query over many gigabytes of data. BQ defaults to utf8, so it makes sense the tables in MySQL/ Aurora are also utf8 encoded. I use Python and SQL Alchemy to load data to BQ. It’s a waste for SA’s ORM features but […]