Standup 08/20/2008

edit Posted by Jonathan Barnes on Wednesday August 20, 2008 at 04:36PM

Ask for Help

"We are getting 504 Gateway errors and we thing it is because our mongrels are freezing up do to inability to allocate memory, what to do?"

Without more info on the problem a few possibilities were suggested, such as the OS might be swap thrashing or the OS has no more memory to allocate.

One suggestion is to cut down your swap space to 0 in an attempt to verify that your mongrels are asking for too much, basically remove to OS swapping memory to disk from the equation.

Another suggestion is to boost your swap up to some insane size, also to take it out of the equation, the theory being that we know mongrel can leak memory, we trust the OS to keep the used memory in RAM, and we have plenty of disk space, so why put your OS in the position of not grating a mongrel what it is asking for.

Both solutions above don't seem ideal but, whatever, we are pragmatists, and if we combine those with periodic monitoring of the system using top/ps/vmstat, at least your mongrel can keep running and this may give you time to figure out why mongrel may be so memory hungry

Comments

  1. edbond edbond on August 21, 2008 at 06:05PM

    We had problems with 'freezed' mongrels when used ferret and had wrong permissions on index directory. Mongrels cpu usage was at 100%. strace to mongrel process and see what it doing.

  2. Jeff Dean Jeff Dean on August 22, 2008 at 12:45AM

    I've had success debugging similar issues using FiveRuns - they've got great alerts, too - like "Holy Sh*t the memory is low!". Combining a huge swap size and a solution like FiveRuns/NewRelic might be a good place to start.

Add a Comment (MarkDown available)