Not all software development projects are treated the same, some have access to modern tech Virtual Private Servers (VPS), Zend Server (http://www.zend.com/products/server/), Memcached, Gearman and all the other goodies I can only dream of. You have a box with LAMP, and you cannot install anything else.
This is an example of how we got around a limitation, using available tools. Problem: I have a list of tasks to execute within my application, however I need to ensure that the tasks are executed and completed, but some are more important than others, and the execution may slow down the performance of the box we are running on. Well in this case we were loading 6 different types of XML files which were FTPed into a location on the box regularly, every 35 minutes and had to be loaded in a specific order. This was further complicated by the fact that we had to reload historical data in case of issues (1 weeks worth of uploads ~ 2100 files) without interrupting the current loading processes.
The approach used the following components:
a) Job Queue – based on the Zend Server Job Queue but simplified for our needs (see data model of tables below)
b) Queue Loader Script – loads the jobs into the job queue by scanning the location containing the files to be loaded and adds the files to the queue (since the queue is a database table, duplicates are discarded without errors) This keeps this file simple and honest
c) Job Executor Script – reads a message from a queue, reads the message body which contains the file name to be processed, could be made more complex
d) Queue Loader Cron Job – calls the Queue Loader Script to add new files to the queue
e) Job Executor Cron Job – calls the job executor script. This job has no effect if a lock file exists, and is not expired which means the script is valid and running. However if the lock file is expired, it means that the process crashed, so the lock file is deleted, a new process is started with a lock file. Basically this keeps the job executor script running indefinitely as long as there are messages to process.
Please feel free to leave a comment on what your experiences are with similar problems.
