It might help to increase how often the cron job runs, assuming that you aren't already maxing out the server.

I've noticed that using the worker in daemon mode is far more responsive than the cron job method. Which way do you have it configured?

Using cron. How did you change to daemon mode?

To me the number isn't the problem it is the rate of growth you seem to be reporting and that your system isn't under heavy load trying to work through it. To me that screams either the workers aren't being created fast enough to keep up with it or you are bottlenecked at the DB. The DB bottleneck would show up as either high CPU load or high disk IOPS. Are either of those curves showing lots of thrashing?

Achtivate these settings in your config file:

        'worker_debug' => true,
        'worker_jpm' => true,
        'worker_jpm_range' => '1',

Then have a look for entries with Load: in it. This tells you about the number of jobs with the different priorities and their processing speed.

Would you mind telling me what config file more exactly? I don't want to ruin anything :D - I suppose it is this https://github.com/friendica/friendica/blob/develop/config/local-sample.config.php - and I have to add those under the "config" section. Correct?