Skip to content

Fix worker count. Add Summon() hook handler. Improve background process count management.

M. Dent requested to merge dentm42/addons:queueworker-improvements into dev

A number of fixes:

  1. Add Summon() hook handler (see core!1692 (merged) )
  2. Fix process management count a) Delay adding a new queueworker process slightly to make sure database is updated (avoid race) b) Add processes dynamically until max process count is reached (previously, the queueworker would prevent new processes if there were too many, but it wouldn't add processes if there was a lot of work to do. Now, as long as there is work to do, the queueworker will kick off new processes until the maximum number of workers is reached)

With these updates, I think we are very much approaching a stable working queueworker. This version has been massively stress tested using a mass-photo import tool that imports photos from a Facebook ZIP archive. With the new Summon() hook and the more robust process count management, things run consistently and reliably and avoid system crashes and fires off the background processes without timing out. Basically, things can be completely DUMPED into the queue rapidly without any need for internal timeouts in front-end processes which would be required otherwise.

This should be helpful for other tools such as content importing and the file import utility which can suffer from web server timeout issues.

Merge request reports