Thanks to the Prometheus – Grafana combo I set up earlier for my Kubernetes cluster I noticed that there was a steep increase of requests to this blog started a few days ago. I checked my Google Analytics dashboard, sadly my blog didn’t become any popular at all. So it must be some sort of bot activity.
Funny though, CloudFlare didn’t think this is brutal enough so it let this attack through.
By the way, stern is a great tool to monitor logs from Kubernetes pods based on selectors. On the other hand the default kubectl logs command only pulls down logs from 1 pod. In this case I used the following command:
stern -l app=wordpress ... wordpress-569bf4bd4-7x8wj nginx 127.0.0.1 - - [18/Oct/2021:23:14:58 +0000] "POST //xmlrpc.php HTTP/1.1" 200 236 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/78.0.3904.108 Safari/537.36" "143.110.187.80,10.246.176.192" wordpress-569bf4bd4-7x8wj nginx 127.0.0.1 - - [18/Oct/2021:23:15:00 +0000] "POST //xmlrpc.php HTTP/1.1" 200 236 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/78.0.3904.108 Safari/537.36" "143.110.187.80,10.246.176.192" wordpress-569bf4bd4-7x8wj nginx 127.0.0.1 - - [18/Oct/2021:23:15:00 +0000] "POST //xmlrpc.php HTTP/1.1" 200 236 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/78.0.3904.108 Safari/537.36" "143.110.187.80,10.246.176.192" wordpress-569bf4bd4-7x8wj nginx 127.0.0.1 - - [18/Oct/2021:23:15:01 +0000] "POST //xmlrpc.php HTTP/1.1" 200 236 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/78.0.3904.108 Safari/537.36" "143.110.187.80,10.246.176.192" ...
This looks quite obvious: Some bot running at 143.110.187.80 was doing brute force attack on my WordPress blog. It’s been going on for days and I didn’t know! Luckily as a good habit of IT professionals, I don’t use any meaningful password, which effectively might cost this bot several dozen centuries to crack my password, at current rate.
But there’s no reason just to let this bot keep wasting my electricity, is there?
From my experience I’ve used nginx to limit request rates, so for now I’ll just brush up my nginx skill and get this done quickly. I use nginx + php-fpm combo to run WordPress, adding rate limiter can be done in 2 lines in nginx configuration:
# in http scope
http {
...
limit_req_zone $http_x_forwarded_for zone=mylimit:20m rate=15r/m;
# note
# 1, I can't use the default $binary_remote_addr as key because the request was passed on by istio so the remote_addr is always 127.0.0.1 as shown in the logs
# 2, with 15r/m this means maximum 15 requests per minute from 1 IP address
# in server scope
server {
...
location ~ \.php$ {
limit_req zone=mylimit burst=60 nodelay;
# note
# this puts a limiter on every .php request. First 60 requests from a distinguishable source are exempted so normal visitors won't get punished.After the rules getting deployed, the logs changed to a new pattern:
wordpress-cf9f6959-l5lhv nginx 2021/10/20 02:31:20 [warn] 20#20: *1166 delaying request, excess: 0.320, by zone "mylimit", client: 127.0.0.1, server: _, request: "POST //xmlrpc.php HTTP/1.1", host: "raynix.info" wordpress-cf9f6959-l5lhv nginx 2021/10/20 02:31:21 [warn] 20#20: *1166 delaying request, excess: 0.840, by zone "mylimit", client: 127.0.0.1, server: _, request: "POST //xmlrpc.php HTTP/1.1", host: "raynix.info" wordpress-cf9f6959-l5lhv nginx 2021/10/20 02:31:25 [warn] 20#20: *1166 delaying request, excess: 0.692, by zone "mylimit", client: 127.0.0.1, server: _, request: "POST //xmlrpc.php HTTP/1.1", host: "raynix.info" wordpress-cf9f6959-l5lhv nginx 2021/10/20 02:31:28 [warn] 20#20: *1166 delaying request, excess: 0.668, by zone "mylimit", client: 127.0.0.1, server: _, request: "POST //xmlrpc.php HTTP/1.1", host: "raynix.info"
Looks like it’s effective. Grafana agreed as well:
I might like to try other ways to harden my blog against attacks. I meant Istio.
🙂