If you work with Drupal long enough, cron will hang and fail on you. It has failed us all. This may not always be a big deal, but it becomes a real issue when your web application or Drupal site has cron-dependent processes, actions, triggers, etc. We whipped up a clever solution to alert us when there's an issue with cron.
The problem with cron failing is that most of the time we don't realize it has hung up and is no longer running until one of our clients calls and says, "Do you know why this isn't working correctly?". If they've noticed, then users have noticed. When this happens, it is imperative that you figure out what went wrong and get cron back to running like it should. After doing so, you may then have to explain why it hung in the first place and why 200+ notifications will be sent out all at once. Frustrating? Yes. Thankfully, we have a solution.
So Cron has Failed/Hung...
First and foremost, when you have a problem with cron (or anything Drupal-related), it is wise to investigate what others have done to fix the same or similar issue(s) by checking with Drupal.org and/or consulting the Googles. If I were a betting man (and I am), then I would say that you'll find an answer. In the case of Cron issues this page on Drupal.org has some great cron troubleshooting advice.
Clear your cache and try to run cron again
If you're getting the typical "Attempting to re-run cron while it is already running" or "Cron has been running for more than an hour and is most likely stuck" messages, then you'll need to clear the cron_last and cron_semaphore variables from your database. Here are a few ways to do this:
- This sequel query should work:
DELETE FROM variable WHERE name = 'cron_semaphore' OR name = 'cron_last' - Manually delete them using an application like Sequel Pro.
- Using Drush, you can execute the following command:
drush vdel cron_semaphore && drush vdel cron_last
You may need to increase the max_execution_time in your php.ini file if you're getting the "Cron run exceeded the time limit and was aborted" error. You may also be hitting a php memory limit if you are receiving an error like "PHP Fatal error: Allowed memory size of X bytes exhausted (tried to allocate Y bytes) in file on line x". In this case, you will need to increase that limit.
There are countless other ways to troubleshoot cron and the functions that are executed during a cron run. See example here ("Identifying the problematic Task"). Also, consider adding a cron 'enhancement' module such as Ultimate Cron.
Our Solution
Ok, thanks for Googling that for me, but I need to know when cron fails so that I can fix it!
Pondering, Googling, and looking around Drupal.org, I sought an answer to the following question: "How can I know when cron has hung?". Finally, a potential resolution (which seemed to make the most sense) came to me: use a Drush script called by a crontab on the server to monitor cron hangs. Note: if you aren't using Drush, you should be. Do yourself a favor and check out Drush here, here, or here. Once you understand it, keep reading.
I decided that I would check the cron_last system variable and see how long it had been since the last cron run. If cron hadn't been run in a certain amount of time (one hour for now), then I wanted an email to be sent to the site admin. There are many other options: from filing a ticket/issue in your tracking system of choice (Assembla, Mantis, HelpDesk, Basecamp, GitHub, etc.) to clearing the cron_last and cron_semaphore variables automatically (which probably isn't recommended, but possible).
Assumption: You have an up-to-date version of Drush installed and working on your development environment.
Warning: Don't do any of this on a production site until you have tested and performed quality assurance locally or on a development server.
Set up a new Drush command
All of the information about creating a new Drush command can be found within the Drush module, or see this article. Alternatively, you can run drush topic docs-commands for more information about command authoring. Additionally, you can refer to the Drush sandwich.drush.inc example.
I created a file called croncheck.drush.inc and added it to the /sites/all/drush folder. There are a few different places that you can place this file, but I placed it there because I wanted to be able to track it in git and make per-site changes. You could also put it in a custom module.
The level of detail that you wish to incorporate into your Drush command file is completely up to you. Once again, following the help provided from Drush should point you in the right direction. I am only going to include the 'critical' code here and will provide a file for download at the end of this post.
1. Start with hook_drush_command()
/**
* Implementation of hook_drush_command().
*
* @return
* An associative array describing your command(s).
*/
function croncheck_drush_command() {
$items = array();
$items['checkcron'] = array(
'description' => "Check on time since cron run and send alert if needed.",
'aliases' => array('checkc'),
'bootstrap' => DRUSH_BOOTSTRAP_DRUPAL_FULL,
);
return $items;
}2. Add a Drush help
/**
* Implementation of hook_drush_help().
*
* @param
* A string with the help section (prepend with 'drush:')
*
* @return
* A string with the help text for your command.
*/
function croncheck_drush_help($section) {
switch ($section) {
case 'drush:checkcron':
return dt("This command will check to see the last time cron ran and send an alert if needed.");
case 'meta:checkcron:title':
return dt("checkcron commands");
case 'meta:checkcron:summary':
return dt("Keeping cron running helps to keep the site running smoothly.");
}
}3. Create the main Drush command callback
/**
*Main drush command callback. This is where the action takes place.
*/
function drush_croncheck_checkcron() {
// Get the last time cron was run via the cron_last variable.
$last_cron = variable_get('cron_last', '');
// Make the default time to check one hour (3600s). Use the
// drushcroncheck_time_to_check variable to change this.
$time_to_check = variable_get('drushcroncheck_time_to_check', 3600);
// Get the difference in seconds between now and the last time cron ran.
$difference = time() - $last_cron;
// Has it been longer than our time to check since cron ran?
if ($difference >= $time_to_check) {
// Format the date so it is more readable than timestamp seconds.
$cleandate = date('l F d Y - G:i:s A', $last_cron);
// Format a message to be used via the command line OR the email.
$message = t('We have a problem, ' . $cleandate . ' is the last time cron ran! Time since last cron is: ' . drush_timesince($difference));
// Uncomment this line if you want to have the display in the command line.
// drush_print($message);
// Configure and send the email.
drush_mail_send_email('admin@somesite.com', 'Somesite.com: Cron has failed or hung', $message);
}
else {
// Add whatever is needed for successful cron runs, this is unneeded now.
drush_print('All good');
}
}4. Add additional functions as needed
In this case, I'm borrowing some code for sending emails with Drush from here.
/**
* Send an e-mail to a specified e-mail address.
* @see http://drupal.org/files/mail.drush_.inc__0.txt
*/
function drush_mail_send_email($to, $subject, $body) {
// Define $from and headers
if (!$from = drush_get_option('from')) $from = variable_get('site_mail', ini_get('sendmail_from'));
$headers = array();
$headers['From'] = $headers['Sender'] = $headers['Return-Path'] = $headers['Errors-To'] = $from;
$headers['X-Priority'] = 1;
$headers['X-MSMail-Priority'] = 'High';
$headers['Importance'] = 'High';
// If $to is not a valid e-mail address, try to load the account based on the username.
if(!valid_email_address($to)) {
if(function_exists('user_load_by_name')) $account = user_load_by_name($to);
else $account = user_load(array('name' => $to));
if(empty($account)) return drush_set_error('DRUSH_BAD_EMAIL_OR_USERNAME', dt('Bad e-mail address or username.'));
else $to = $account->mail;
}
// D7 implementation of drupal_mail
if(function_exists('drupal_mail_system')) {
// Prepare the message.
$message = drupal_mail('drush', 'key', $to, NULL, array(), $from, FALSE);
$message['subject'] = $subject;
$message['body'] = array();
$message['body'][] = $body;
$message['headers'] = $headers;
// Retrieve the responsible implementation for this message.
$system = drupal_mail_system('drush', 'key');
// Format the message body.
$message = $system->format($message);
// Send e-mail.
$message['result'] = $system->mail($message);
$result = $message['result'];
// D6 implementation of drupal_mail_send
} else {
$message = array(
'to' => $to,
'subject' => $subject,
'body' => $body,
'headers' => $headers,
);
$result = drupal_mail_send($message);
}
// Return result.
if($result) drush_log(dt('E-mail message sent to <!to>', array('!to' => $to)), 'ok');
else drush_set_error('DRUSH_MAIL_ERROR', dt('An error occurred while sending the e-mail message.'));
}Lastly, I added a helper function for formatting the time since cron last ran. The format is Days, Hours, Minutes and Seconds, which with this script running, shouldn't be more than hours. Get the script here.
/**
* Helper function to transfor seconds into a usable time since string
* @see http://www.neowin.net/forum/topic/806866-changing-seconds-into-days-hours-minutes-seconds/page__p__591387898#entry591387898
*/
function drush_timesince($seconds) {
if ($seconds > 3600){
$days = $seconds / 86400;
$day_explode = explode(".", $days);
$day = $day_explode[0];
}
else {
$day = 0;
}
if ($seconds > 3600){
$hours = '.'.@$day_explode[1].'';
$hour = $hours * 24;
$hourr = explode(".", $hour);
$hourrs = $hourr[0];
}
else {
$hours = $seconds / 3600;
$hourr = explode(".", $hours);
$hourrs = $hourr[0];
}
$minute = '.'.@$hourr[1].'';
$minutes = $minute * 60;
$minute = explode(".", $minutes);
$minuter = $minute[0];
$seconds = '.'.@$minute[1].'';
$second = $seconds * 60;
$second = round($second);
return(@$day.' Days '.@$hourrs.' Hours '.@$minuter.' minutes, '.@$second.' seconds');
}Call the new Drush command using crontab
Since this will need to be added to your system's crontab, here is one way to do it so that it gets called once every hour on the 5 minute mark. Check out the Drupal.org reference on Configuring crontab with Drupal if needed.
5 ** ** /usr/bin/drush -l http://www.example.com -r /var/www/site checkcronIt would be wise to check that the Drush command works before adding it to the crontab. To verify that it is working, run the following:
drush checkcron<Feel free to download my croncheck.drush.inc file for use or for tweaking. Please note: CHROMATIC nor myself bear any liability for its usage. I'd love feedback, so feel free to post comments, questions, and suggestions to the comments! Enjoy!
