I am not talking toilets here – although for a short while, I wish I was – it would have been less frustrating.
I work and play in a LAMP environment, for example, this blog and site lives on a LAMP stack, and have recently needed to look at tweaking some performance handles.
A number of things have made me properly swear this week.
memchached, APC, and suexec primarily, along with the caches that exist in the software “running” the sites – like WordPress and Drupal.
Caches are so vitally important for anything in computing, as the direct path for repetitive data from storage to consumption, either via a computer or ultimately a human is simply, even in this era of super fast computing, just too damn slow.
We have files from disk cached in physical RAM done by the Linux kernel to increase the performance by dropping the latency at which frequently accessed files are actually read off the disk.
We have caching on the storage controllers for much the same reason – disk storage is slow ( SSD isn’t that common – and is still relatively slow when comparing access times of CPU cache )
We have caching of snippets of “compiled” code with APC (or similar) because it is computationally more expensive, which means time expensive, to recompile the code for an interpreter (PHP) than it is to store that compiled code in memory.
We use caches of commonly requested SQL queries because requesting the data from the actual database every time is expensive time wise – this is memcached.
We have the application itself (WordPress/Drupal) caching pages, or most if the pages and some calls it makes to save time when that is required.
All of these caches improve the user experience, even if that user is a computer doing automated queries.
We’ve come a long way.
The downside comes when data in one of these caches can trample over the data in another one of these caches, not from a memory perspective, because the areas of memory that each if these caches uses is very definitely separate (if it wasn’t – then there’d be huge opportunity for data trampling and associated corruption) – but from a a re-use perspective.
A page has been requested by a user, the application has done it’s caching, and thus could serve that page again without having to re-execute some parts (or all) of the request.
Great …..
Until we, the systems engineer wants to tweak more performance by adding another level of caching.
And due to the nature if the beast – the application’s cached results try to get executed at the next level in the chain – and end up having problems with, for example “Cannot redeclare class insert MySQL”
Enter the world of hair-pulling and head-table banging.
Enable APC, from the php/apache configuration perspective, first page off the site works, second, the infamous PHP white screen of death, and a nasty message in the apache log.
All worked perfectly on a test environment, but the only difference in the test environments is that there was one application installed on the server, not two or more.
Different site, same server, works properly.
Eh ? go figure!
Now, this other site, was the one that I had configured with APC/memcached first, so were were talking cached prefix keys trampling over each other ?
This thought ends in this from a google search.
Bottom line, yes, the APC module configuration from the first site could easily have been stamping all over the requests from the second (and third) sites.
So – configure the
as appropriate on both sites, and restart apache to clear the APC cache.
The problem was still apparent !!!!!!
Stop/start apache to clear APC cache, first page, fine, second page, no matter what that page, CRASH.
Bugger.
Now what.
Damn – disable it all, and put to the side for the day.
Next day, start from a clean slate.
And that term, clean slate was what got it for me.
Clean, totally, clear everything.
There is a reason there are a lot of instructions in Drupal for “drush cc all” when deploying new code or functionality, or modules etc, it clears all of Drupal’s internal cache, and starts everthing afresh.
So, clear all caches, Drupal, wordpress, restart memcached, restart apache, – then try accessing the website(s).
All perfect!!
Multiple pages, one ofter another work perfectly now.
We also get some interesting statistics with the use of APC, out of the 512MB we assigned, we are using about 380MB of it at the moment, but, by God the sites are faster.
Therefore, the words of wisdom for today – like you do after visiting the toilet – pull that chain, and flush all caches in a system every time you flush one. Flush ’em all.
