I have a two tier structure, S1 Runs apache, Drupal,APC and memcached, S2 runs mysql.
Moderate traffic about 10K visits/day.
I recently started to use a new monitoring programme and it showing lot of IOwait happening on the server S1. Further analysis shows all that IOwait is due to apache. the disk IO monitor shows most of the IO is write. Which is expected as reads (although several times higher) are happening through memcache and network from mysql. I also depend on CDN to distribute most of the static files.
I have confirmed NO swapping is going on here. Also NO NFS here.
My questions is what apache is writing with such high number? I have uploads enabled for users but the actual files uploaded nowhere seems close to IOwait and disk writes. We use DBlog for watchdog log so it can't be the contributor.
Anyone experienced this? What apache would be writing? I am aware it is writing to apache log but can that be at such high rate? Are there other areas I should look into? Are there any drupal modules that write to disk? Use php as apache module, would that be writing to disk?
Comments
Could be atime
If you don't care about file access times, disable that feature of your FS.
Add "noatime,nodiratime" to the mount options.
Apache logs? PHP notices?
Do you have lots of writes in Apache's logs? Check if you have a high LogLevel in the configuration.
Or do you have lots of PHP notices that are going to Apache's error log for example?
Do you have lots of stuff in watchdog and you are using syslog for logging?
Drupal performance tuning, development, customization and consulting: 2bits.com, Inc..
Personal blog: Baheyeldin.com.
S1? S2? A VPS? What brand? Raid disk alignment?
There are many reasons other than pending IO buffers to flush to disk to cause slow IO, and thus create IOwait conditions.
LVMs can do this. Raid disk alignment can as well. Neither of these can be well detected within the VE. Your VEs might be on different machines, and the web server machine is IO hosed by a 3rd party VE.
Without more information, there is not enough clues to go further.
Peter
LA's Open Source User Group Advocate - Volunteer at DrupalCamp LA and SCALE
There are NOT many errors in
There are NOT many errors in error log. It is set at Warn level but hardly 5-10 errors in a day.
The watchdog is heavily used but it uses DBlog and NOT syslog.
Using Cloudserver from rackspace. Don't know what is the disk type used.
Probably Xen, but ...
Rackspace Cloud is probably Xen based.
But just in case it is not, OpenVZ and Virtuozzo can have really bad disk performance specially if it is using the vzfs filesystem.
Although, if this is the case, you will see sluggish performance, but not high O/I Wait times ...
Try getting iotop and see what it says ...
Drupal performance tuning, development, customization and consulting: 2bits.com, Inc..
Personal blog: Baheyeldin.com.
iotop shows it is httpd that
iotop shows it is httpd that is doing disk writes. But don't know what else to look into or interpret.
is the mod_DEFLATE needs to do lot of disk IO? I thought it must be doing compression on the fly before sending.
Search codebase
Search your codebase for calls to file_put_contents() and fwrite(). Or if you have a way to profile code do that.
lsof
Don't think mod_deflate would use disk at all.
Use lsof to see what files are open by your Apache daemon (httpd or apache2 depending on the distro).
That would give you the names of files open by Apache, and which mode they are open in. Look for writes. Once you identify the files, you can do a tail of it/them to see what is being written.
Could be one of those pesky useless things, like writing bandwidth usage data for every request, or such nineties shared hosting things.
Drupal performance tuning, development, customization and consulting: 2bits.com, Inc..
Personal blog: Baheyeldin.com.
find / -ctime -0
Reduce the records output by lsof to just apache (or httpd)
lsof | grep -i apache
My preference to find out is this command
nice time find / -ctime -0 -ls | tee ~/find.root-ctime-0
which will ls -l all files modified in the last 24 hours.
The file of concern will be the largest file. Unless, apache is overwriting records. Are you using any apache mysql options, forcing apache to write to a database table(s)? Are you hosting many sites, and splitting the log files to different files, instead of a central log file? Is a web site under attack? DOS? Good luck.
Peter
LA's Open Source User Group Advocate - Volunteer at DrupalCamp LA and SCALE