There are two main reasons for this: the next job causing an error, or a stale lock.
How to fix a job queue stuck with a job that causes an error
- Check for errors. Go to "Monitor -> Show System Log" and check if there is a message of loglevel "Error". If yes, proceed below; if no, it is more likely that you have the stale lock file issue – see next section.
- Fix the error. Fix the cause for the error and then click "Remove" in the ISPConfig system log view. This will cause ISPConfig to reprocess the stuck job when its cron job runs the next time (which is every minute in a default setup).
- If needed, use Debug log level. If it is not clear what causes the error, enable the Debug log level (in System -> Server Config -> Server -> Loglevel), wait a minute and revisit "Monitor -> Show System Log". Since ISPConfig continuously tries to process the failing job again, now you should at least see more detailed error messages.
- If needed, use direct cron output. If it is still not clear what causes the error, you can look at the direct output of the cron job by executing
/usr/local/ispconfig/server/server.shin a console. If you see "There is already an instance of server.php running.", wait a few seconds and try again – your execution accidentally overlapped with the regular ISPConfig cron job. If you keep getting this however, it is more probable that you have the "stale lock file" issue, see below.
- Or let ISPConfig ignore the failing job. If you don't know how to fix the error, you can let ISPConfig jump over this job and proceed with the next one. For that, increment database column server.updated by one in the ISPConfig database (usually dbispconfig), in the row that belongs to the affected server. If you want to jump over several jobs: make this number correspond to the ID of the last job record which ISPConfig should consider "done". Job records are stored in table sys_datalog, and their IDs in column sys_datalog.datalog_id.
- "Delete ISPconfig job queue" thread
- ISPConfig 3 Manual 1.4, chapter 7.1 "How Do I Find Out What Is Wrong If ISPConfig Does Not Work?"
How to fix when a database creation job causes an error
This is one of the cases from the scenario above, where a job in the job queue causes an error. However, this error is particularly nasty, since it will not show up as an error in the ISPConfig job queue, and can cause teh cron job to occupy all the system's memory, so that even the web server can crash (or fail to restart) in response.
- The ISPConfig job queue is no longer processed, and the next job to process (so, the one it got stuck with) is probably a database creation or deletion job.
- You have the smpyoms of the "stale lock file" issue below, because the ISPConfig cron job does not return, and if it finally crashes, it leaves the stale lock file.
- When deleting the stale lockfile, enabling the debug log level and then starting the cron job from the command line, one gets an infinite loop of error messages like these:
23.02.2014-15:25 - WARNING - DB::query(SELECT count(syslog_id) as number FROM sys_log WHERE datalog_id = 910 AND loglevel = 2) -> mysqli_query Access denied for user 'root'@'localhost' (using password: YES)
How to fix this:
The cause of this was that the MySQL root user password was changed after installing ISPConfig, and without adding this to the ISPConfig configuration. It is needed there for ISPConfig to be able to create and delete MySQL databases. So, edit /usr/local/ispconfig/server/lib/mysql_clientdb.conf and update the MySQL root password there (entering it in plain text). [source]
How to fix a stale lock file
This reason is the most likely cause in cases where you see the following message permanently when enabling the debug log level in System -> Server Config -> Server -> Loglevel and then (repeatedly) trying to start the cron job manually on the console:
user@host:~ $ /usr/local/ispconfig/server/server.sh
user@host:~ $ 15.11.2013-19:56 – DEBUG – There is already an instance of server.php running. Exiting.
It means that the ISPConfig cron job crashed (or you killed it) when it was running the last time, leaving a stale lock file behind.
Fix this by executing:
rm -f /usr/local/ispconfig/server/temp/.ispconfig_lock