Fixed tracker-store’s CPU hogging issue on Arch Linux

I think it’s since when I updated Arch Linux a while ago the tracker-storeprocess has become CPU-hogging and it can drain the battery pretty quickly and turn my laptop into a heater.

Obviously there are people experiencing this issue already, but most of them that I found were trying to disable the Gnome tracker. I’d hesitate to do that because the tracker’s purpose is to index stuff so when I hit the super key and type, relevant things will come up quickly. Also as I’m a big fan of Gnome Shell, I trust the team wouldn’t just release a buggy program and left it broken for months.

My trouble-shooting 101: if I want to see what the program is complaining, run it from the command line!

tracker daemon -s
Starting miners…
** (tracker daemon:24398): CRITICAL **: 10:08:51.616: Could not create proxy on the D-Bus session bus, Error calling StartServiceByName for org.freedesktop.Tracker1.Miner.RSS: Timeout was reached
** (tracker daemon:24398): CRITICAL **: 10:09:16.640: Could not create proxy on the D-Bus session bus, Error calling StartServiceByName for org.freedesktop.Tracker1.Miner.Files: Timeout was reached
** (tracker daemon:24398): CRITICAL **: 10:09:41.663: Could not create proxy on the D-Bus session bus, Error calling StartServiceByName for org.freedesktop.Tracker1.Miner.Extract: Timeout was reached
** (tracker daemon:24398): CRITICAL **: 10:10:06.687: Could not create proxy on the D-Bus session bus, Error calling StartServiceByName for org.freedesktop.Tracker1.Miner.Applications: Timeout was reached
** (tracker daemon:24398): CRITICAL **: 10:10:06.687: No D-Bus proxy found for miner 'org.freedesktop.Tracker1.Miner.RSS'
✗ RSS/ATOM Feeds (perhaps a disabled plugin?)
** (tracker daemon:24398): CRITICAL **: 10:10:06.687: No D-Bus proxy found for miner 'org.freedesktop.Tracker1.Miner.Files'
✗ File System (perhaps a disabled plugin?)
** (tracker daemon:24398): CRITICAL **: 10:10:06.687: No D-Bus proxy found for miner 'org.freedesktop.Tracker1.Miner.Extract'
✗ Extractor (perhaps a disabled plugin?)
** (tracker daemon:24398): CRITICAL **: 10:10:06.687: No D-Bus proxy found for miner 'org.freedesktop.Tracker1.Miner.Applications'
✗ Applications (perhaps a disabled plugin?)

I don’t think those are the root causes though.

Later I found this issue for Gnome 3.12 and I think it could be my case as well. To my surprise this fixed my issue with tracker-store easily.

First, stop tracker-store gracefully with tracker daemon -tThen I found:

ls ~/.cache/tracker/ -lht
total 908M
-rw-r--r-- 1 ray users 32K Dec 26 10:10 meta.db-shm
-rw-r--r-- 1 ray users 132M Dec 26 10:10 meta.db-wal
-rw-r--r-- 1 ray users 776M Oct 4 09:20 meta.db
-rw-r--r-- 1 ray users 11 Oct 4 09:17 locale-for-miner-apps.txt
-rw-r--r-- 1 ray users 22 Aug 2 13:49 parser-version.txt
-rw-r--r-- 1 ray users 354K Apr 13 2018 ontologies.gvdb
-rw-r--r-- 1 ray users 11 Apr 13 2018 db-locale.txt
-rw-r--r-- 1 ray users 6 Apr 8 2017 first-index.txt
-rw-r--r-- 1 ray users 10 Apr 8 2017 last-crawl.txt
-rw-r--r-- 1 ray users 40 Apr 8 2017 parser-sha1.txt
-rw-r--r-- 1 ray users 2 Apr 8 2017 db-version.txt

ls ~/.local/share/tracker/data -lht
total 77M
-rw-r----- 1 ray users 16M Oct 4 09:17 tracker-store.journal
-rw-r--r-- 1 ray users 8.8M Apr 13 2018 tracker-store.journal.7.gz
-rw-r--r-- 1 ray users 8.6M Apr 13 2018 tracker-store.journal.6.gz
-rw-r--r-- 1 ray users 9.1M Apr 13 2018 tracker-store.journal.5.gz
-rw-r--r-- 1 ray users 8.5M Apr 13 2018 tracker-store.journal.4.gz
-rw-r----- 1 ray users 148K Apr 13 2018 tracker-store.ontology.journal
-rw-r--r-- 1 ray users 8.3M Jun 19 2017 tracker-store.journal.3.gz
-rw-r--r-- 1 ray users 9.1M Jun 18 2017 tracker-store.journal.2.gz
-rw-r--r-- 1 ray users 8.8M Apr 8 2017 tracker-store.journal.1.gz

The 776MB meta.db surely is bloated, as I only have around 200GB of personal files. Since they are only meta-data, deletion won’t hurt.

rm -rf ~/.cache/tracker/
rm -rf ~/.local/share/tracker/

tracker daemon -s
Starting miners…
✓ RSS/ATOM Feeds
✓ File System
✓ Extractor
✓ Applications

All good now 🙂

Playing with Kubernetes Ingress Controller

It’s very very easy to use Kubernetes(K8s) to provision an external service with AWS ELB, there’s one catch though(at least for now in 2018).

AWS ELB is usually used with an auto scaling group and a launch configuration. However with K8s, EC2 instances won’t get spun directly, only pods will, which is call Horizontal Scaling. K8s will issue AWS API calls to update the ELBs so there’s no need for auto scaling groups or launch configurations.

This worked like a charm until when things got busy. There was a brief down time on one of the ELBs managed by K8s, because all instances at the back of the ELB were marked as unhealthy! Of course they were healthy at that moment. With help from AWS Support team, the culprit seems to be similar to this case: https://github.com/kubernetes/kubernetes/issues/47067.

Luckily for me, I had a gut feel that the simple ELB implementation isn’t the best practice and started to adopt the K8s Ingress Controller. And in this case I believe ingress can avoid the down time because the routing is done internally in K8s cluster which doesn’t involving AWS API calls. Nonetheless ingress can use 1 ELB for many apps and that’s good because ELBs are expensive.

Here are steps to deploy an nginx ingress controller as an http(L7) load balancer:

Deploy the mandatory schema, the default replica number for the controller is 2, I changed it to 3 to have 1 in each availability zone:

kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/master/deploy/mandatory.yaml

Some customisation for L7 load balancer on AWS, remember to use your SSL cert if you need https termination:

kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/master/deploy/provider/aws/service-l7.yaml
kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/master/deploy/provider/aws/patch-configmap-l7.yaml

Then an ingress for an app can be deployed:

$ cat .k8s/prod/ingress.yaml 
---
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: my-ingress
  namespace: my-prod
  annotations:
    kubernetes.io/ingress.class: prod
spec:
  rules:
    - host: my.domain.elb
      http:
        paths:
          - path: /
            backend:
              serviceName: my-service
              servicePort: 80
    - host: my.domain.cdn
      http:
        paths:
          - path: /
            backend:
              serviceName: my-service
              servicePort: 80

Notes:

  • my-service is a common NodePort service and has port 80 exposed
  • io/ingress.class is for multiple ingress controllers in same k8s cluster, eg. 1 for dev and the other for prod
  • for now I have to duplicate the host block for each domain, because wildcard or regex are not supported by k8s ingress specification
  • at last, find the ELB this ingress controller created, then point my.domain.elb to it, then the CDN domain can use my.domain.elb as origin.

🙂

Don’t Need Ngrok When I Have SSH

I was trying to create a Slack app. In order to let Slack send REST requests to my dev environment, eg. http://localhost:9000, I searched a bit and saw ngrok. Ngrok is very handy for this kind of setup:

Slack -> xyz.ngrok.io -> localhost

However I just don’t want to install anything so I turned to Google and to my surprise SSH can exactly do this(for who knows how many years). I know I can forward a local port to a remote host to connect to a service behind firewall such as databases, this is my first attempt to forward a remote port to local so Slack API can contact my localhost.

Here’s a better article which explained how to do port forwarding in both directions with SSH.

In short, to forward a remote port to my localhost, I need to

1, update the sshd_config on remote host and have GatewayPorts enabled and then restart SSH service

GatewayPorts yes

2, in a local terminal, run the following command replacing my.remote.host with your server’s domain or IP.

ssh -nNT -R 9800:localhost:9000 my.remote.host

Then test it with

curl -i http://my.remote.host:9800

The request should be forwarded to your localhost:9000.

🙂