Drupal Lock Stampede

When a Drupal site begins to be overwhelmed by high volumes of traffic, one of the worst bottlenecks is Drupal core's menu system. A menu rebuild during high traffic can easily causes a stampede on your database, which is exacerbated by the architecture of core's Lock API.

The order of operations that leads to the stampede behavior is as follows:

  • Menu rebuild is triggered by a user action (e.g., change to menu item).
  • Lock API creates a lock in semaphore table in MySQL, temporarily preventing other menu rebuild requests.
  • Before lock is released, a page request is made that calls menu_get_item(). This attempts to rebuild menu, but must wait until current lock is released.
  • Subsequent requests snowball by spinning up additional processes that 1) wait for lock to be released, and 2) rebuild menu again after release.
  • Lock stampede ensues.

In some cases, a single page load can trigger up to 10 write requests to the semaphore table. Imagine the number of writes that would be generated by a high traffic event combined with this stampeding behavior. Your site can jump from 20 idle MySQL processes to 150+ in the blink of an eye--bringing your site to its knees.

How do you know if this is happening to you?

  • Your site sporadically becomes extremely slow
  • You see slow INSERT queries performed on the semaphore table
  • You see unexplained spikes in the number of writes to your MySQL database
  • You see a large number of sleeping MySQL processes

What can you do about it? One solution is to swap out the backend for the Lock API. MySQL just isn't a good backend for a locking system that requires this volume of writes, while also serving a large number of selects. If you're using Memcache with Drupal, there's an out of the box solution for you:

<?php
    $conf
['cache_backends'][] = './sites/all/modules/contrib/memcache/memcache.inc';
   
$conf['cache_default_class'] = 'MemCacheDrupal';
   
$conf['lock_inc'] = './sites/all/modules/contrib/memcache/memcache-lock.inc';
   
$conf['memcache_stampede_protection'] = TRUE;
?>

Note the two lines at the end of that snippet--they change the Lock API to use Memcache as a backend. Usage of Memcache as a backend for the Lock API changes the behavior in the following way:

  • Menu rebuild is triggered by a user action (e.g., change to menu item).
  • Lock API creates a lock in semaphore Memcache, temporarily preventing other menu rebuild requests.
  • Before lock is released, a page request is made that calls menu_get_item(). Since the resource is locked, the menu is served from cache. No additional rebuilds are triggered.

The combination of providing a faster backend with the new anti-stampede logic can make a significant difference in your site's stability.

Drupal Version Compatibility: 

Comments

I GOT A FAST LOAN @ 2% INTEREST RATE with [email protected]

I am a single mother with two kids. i was homeless, stuck in a financial situation and i needed money to pay debt and pay for my husband medical bill and his funeral cost. I tried seeking loans from various loan firms both private and corporate but never with success and most banks declined my credit. Then i was introduced to VICTORIA LAWSON LOAN FIRM by a friend and i applied immediately with { [email protected] } loan sum of $30,000USD and today i could get home, owner business, my kids are doing well at the moment. So dear, if you must contact any firm with reference to securing a loan with low interest rate of 2% and good repayment plans, Contact Victoria Lawson Loan Firm with ( [email protected] ) I am so happy now and i decided to let people know more about her, she offers all kinds of loans. You can contact her company today

*Full Name:_________

*Address:_________

*Tell:_________

*loan amount:_________

*Loan duration:_________

*Country:_________

*Purpose of loan:_________

*Monthly Income:__________

*Occupation__________

*Next of kins :_________

*Email :_________

Add new comment

Plain text

  • No HTML tags allowed.
  • Lines and paragraphs break automatically.