Wow, what an amazing rehaul. I hope there is no mixed content on this page anymore.
Currently I am working with a library application. This application comes with its own GUI, bash scripts, databases etc. Since the database is complex, the manufacturer delivers bash scripts to grab important data from the database. Unfortunately such a bash script takes around 300-400 seconds to run and more importantly it just redirects the call to the GUI tools.
Now comes my part in all of this. I maintain a set of self-written scripts, running every night per cronjob, and which execute around 180 of those calls. Doing the math, even in the best case of 300 seconds, the whole procedure takes around 15h!
The solution: Naive Parallelization
1 2 3 4 5 6
This was the first idea under the assumption that the system knows how to queue them. Remember these are calls to the GUI and it should be possible for two people to work with the GUI at the same time, right? Well no. This is sadly not the case and when digging deeper it is explicitly stated that this is not intended to be done. Next idea was to create an artificial offset.
1 2 3 4 5 6 7
This works(!), well most of the time. The only thing left was to re-do the cases where it creates problems. Again this could not be done in a direct fashion. Each call_job creates its own log file. To re-do all bad cases, each log file needed to be searched for error messages.
For what it is worth, I think this is a solution beautiful enough. It is easy to maintain and as long as the system does not delete randomly log files, I know it works.