Context
We had a platform running with Symfony where we needed to execute a lot of heavy background jobs daily.
Initially we used a bundle which handles our jobs and their execution. With the growing user base, it had started to let us down. We were getting more and more jobs that are stuck, hung, etc…
Solution
To solve this, i developed a new bundle that offers a batch processing system with the Docker engine.
All jobs are executed separately in docker containers.
This way is more viable as Docker handles the execution of the started containers.
The neat thing with this bundle is, you can pretty much execute anything inside of those containers.
For example, your application is made with PHP / Symfony, but let's say you need a downloader component, rather than doing this with PHP, you could make a quick node.js script taking advantage of js async and package it into a Docker image.
Monitoring UI
The bundle also offers a Monitoring user interface which you may choose to activate.
With this interface, you can:
— get an overview of your jobs
— inspect jobs
— stop jobs
— get usage info for each running job
Job details page
Dashboard page
Check out the bundle, see if it's what you needed.