Mind of Matt

How to fix that ISPConfig 3 does no longer process its job queue?

There are two main reasons for this: the next job causing an error, or a stale lock.

How to fix a job queue stuck with a job that causes an error

  1. Check for errors. Go to "Monitor -> Show System Log" and check if there is a message of loglevel "Error". If yes, proceed below; if no, it is more likely that you have the stale lock file issue – see next section.
  2. Fix the error. Fix the cause for the error and then click "Remove" in the ISPConfig system log view. This will cause ISPConfig to reprocess the stuck job when its cron job runs the next time (which is every minute in a default setup).
  3. If needed, use Debug log level. If it is not clear what causes the error, enable the Debug log level (in System -> Server Config -> Server -> Loglevel), wait a minute and revisit "Monitor -> Show System Log". Since ISPConfig continuously tries to process the failing job again, now you should at least see more detailed error messages.
  4. If needed, use direct cron output. If it is still not clear what causes the error, you can look at the direct output of the cron job by executing /usr/local/ispconfig/server/server.sh in a console. If you see "There is already an instance of server.php running.", wait a few seconds and try again – your execution accidentally overlapped with the regular ISPConfig cron job. If you keep getting this however, it is more probable that you have the "stale lock file" issue, see below.
  5. Or let ISPConfig ignore the failing job. If you don't know how to fix the error, you can let ISPConfig jump over this job and proceed with the next one. For that, increment database column server.updated by one in the ISPConfig database (usually dbispconfig), in the row that belongs to the affected server. If you want to jump over several jobs: make this number correspond to the ID of the last job record which ISPConfig should consider "done". Job records are stored in table sys_datalog, and their IDs in column sys_datalog.datalog_id.

Sources:

  1. "Delete ISPconfig job queue" thread
  2. ISPConfig 3 Manual 1.4, chapter 7.1 "How Do I Find Out What Is Wrong If ISPConfig Does Not Work?"

How to fix when a database creation job causes an error

This is one of the cases from the scenario above, where a job in the job queue causes an error. However, this error is particularly nasty, since it will not show up as an error in the ISPConfig job queue, and can cause teh cron job to occupy all the system's memory, so that even the web server can crash (or fail to restart) in response.

Symptoms:

  • The ISPConfig job queue is no longer processed, and the next job to process (so, the one it got stuck with) is probably a database creation or deletion job.
  • You have the smpyoms of the "stale lock file" issue below, because the ISPConfig cron job does not return, and if it finally crashes, it leaves the stale lock file.
  • When deleting the stale lockfile, enabling the debug log level and then starting the cron job from the command line, one gets an infinite loop of error messages like these:
    23.02.2014-15:25 - WARNING - DB::query(SELECT count(syslog_id) as number FROM sys_log WHERE datalog_id = 910 AND loglevel = 2) -> mysqli_query Access denied for user 'root'@'localhost' (using password: YES)

How to fix this:

The cause of this was that the MySQL root user password was changed after installing ISPConfig, and without adding this to the ISPConfig configuration. It is needed there for ISPConfig to be able to create and delete MySQL databases. So, edit /usr/local/ispconfig/server/lib/mysql_clientdb.conf and update the MySQL root password there (entering it in plain text). [source]

How to fix a stale lock file

This reason is the most likely cause in cases where you see the following message permanently when enabling the debug log level in System -> Server Config -> Server -> Loglevel and then (repeatedly) trying to start the cron job manually on the console:

user@host:~ $ /usr/local/ispconfig/server/server.sh
user@host:~ $ 15.11.2013-19:56 – DEBUG – There is already an instance of server.php running. Exiting.

It means that the  ISPConfig cron job crashed (or you killed it) when it was running the last time, leaving a stale lock file behind.

Fix this by executing:

rm -f /usr/local/ispconfig/server/temp/.ispconfig_lock

[source]

Posted

in

,

by

matt

Tags:

Comments

8 responses to “How to fix that ISPConfig 3 does no longer process its job queue?”

  1. How to fix “530 Login authentication failed” for ISPConfig FTP logins? « Mind of Matt

    […] Check that the job queue is empty. In ISPConfig, your tasks (like creating an FTP user) get processed deferredly by a cron job that runs every minute. But if it fails, the job queue can get stuck. Check in ISPConfig in "Monitor -> System State -> Show Jobqueue" to make sure it's empty, or if not, fix this as shown here. […]

  2. Stitchy

    Hello Matt!

    I’m having this very issue; I had to restore my ISPConfig on a new server after a crash, but now I am unable to make the queue run; I checked both options you gave in your post, but none of them worked for me. Any more things I could check?

    Thanks in advance!

    Stitchy

  3. matt

    Hey Stitchy, I did not experience any specific other causes for this problem, so here’s just a bunch of generic hints:

    Do you get any error message when enabling the debug log level in System -> Server Config -> Server -> Loglevel and then trying to run the job queue manually on the console, by executing /usr/local/ispconfig/server/server.sh (or wherever your installation is)?

    I would also make sure that you have nothing at all in your ISPConfig job queue. The cron job should at least then run successfully (when starting it manually).

    And the obvious stuff of course: make sure you did not forget to setup cron again on your new server, and that it calls the ISPConfig command at all (just append a command to the cron call that will send you an e-mail or log the event). And in case you copied over the config to the new server but have installed a different version of ISPConfig there, that would seem like a recipe for trouble. I’d probably re-install and re-configure ISPConfig from scratch in that case.

  4. Stitchy

    Thanks for your answer, Matt.

    I have already done everything you are suggesting me, following your post and the ISPConfig 3 ‘official’ manual, but my jobs queue is still stuck. No errors are thrown on the debug mode, I checked versions as well, and new and old where the same. I know I must be forgetting something but I can’t figure what it could be; I can ‘delete’ all the jobs in the queue by changing the ‘updated’ field from the server table, but I am not sure if that’s a safe practise (they have appeared later in my previous installation).

    I setted a new installation, but I am having the same problem again, and this leads me to think that I must be missing something. I know I would find it if I’d understand how ISPConfig jobs work, but I amb not having much luck today.

    Any more hint would be really appreciated; if you don’t have any more, don’t worry, lots of thanks for your time.

    Stitchy

  5. [Solucionado] ISPConfig 3 no funciona: No crea sitios ni sus carpetas – donmik

    […] Si sigues teniendo problemas, te recomiendo este artículo que yo mismo he utilizado y tiene varias respuestas a posibles errores: http://ma.juii.net/blog/fix-ispconfig-job-queue. […]

  6. Daniel Nascimento

    Man you nail it. Thank you soo much – the LOCK even prevent a succesfull update of ispconfig 3 – ispconfig_update.sh was quitting without any errors or log in LogLevel Warn/Debug/Err what so ever… I was clueless till I saw that my jobs were stucked. After solving the LOCK FILE as you mentioned, everything went back to normal.

    What caused the cron job to get stucked it is still a mystery for me. Thats how it was in ps jobs, and how I made it work:

    root@web01:/var/log/ispconfig# ps aux|grep server.php
    root 29683 0.0 0.0 7776 852 pts/0 S+ 21:49 0:00 grep server.php
    root 30518 0.4 2.3 1070352 781892 ? S 11:00 2:46 /usr/bin/php -q /usr/local/ispconfig/server/server.php
    root@web01:/var/log/ispconfig# kill -9 30518
    root@web01:/var/log/ispconfig# rm -f /usr/local/ispconfig/server/temp/.ispconfig_lock
    root@web01:/var/log/ispconfig# /usr/local/ispconfig/server/server.sh
    17.02.2016-21:49 – DEBUG – Found 12 changes, starting update process.

    Thats it! It worked like a charm. All jobs and upgrade script have been processed without any problems.

    Thank you Matt!

  7. Anonymous

    Hi,
    I have configured ISPConfig 3 DNS only(primary and secondary mirror setup) with CentOS 7.3
    From last six month it is working perfectly.But now it showing datalog_status_u_server: 6 in ISPconfig webpanel.
    So i enable debug mode on my secondary serever by following below mentioned thread,
    https://www.howtoforge.com/community/threads/please-read-before-posting.58408/

    Attached is the htf_report.txt file. Below is the output of server.sh script

    21.04.2017-20:26 – DEBUG – Found 2089 changes, starting update process.

    Its just showing this message from last one hour,nothing is happening.

    Kindly help me to resolve this issue because of this issue my new DNS records are not getting updated in my secondary server.

  8. matt

    Hello there … about the “datalog_status_u_server: 6” issue I have no immediate idea, and it does not seem related to the stuck job queue issue. You better ask on the ISPConfig forum, or perhaps also here: https://webmasters.stackexchange.com/ . Good luck 🙂

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.