I know, and I can only apologise that everything wasn't fixed last night as I thought. We can now support more users without the site crashing, but are still having problems when 100 people post at once. It doesn't help that due to the way in which phpBB works, that 850 users in physical terms, mean about 300 different 'threads' (or active processes) on the processor. After checking the performance logs, I can see that our CPU hasn't been below 100% usage in over three days (besides the crashes).
Plain and simply, I got it wrong and thought the overriding problem was to do with the amount of page faults and threads that MySQL was producing. Last night, I adjusted some of the variables to compensate and help MySQL's caching system work more efficiently and it doesn't seem to of cured the overall sickness.
We wanted to try and avoid this, but the only way to now find the overall problem is to dive in to the processes that run the server itself, and wade through each time server request bit by bit, to find the root cause. I will either do this tonight or tomorrow night, as I have prearranged to go for a drink with a rag friend tonight. The site will have to be offline for about 2 hours, so I will do it in the early hours. I can only apologise to the regular Aussie/NZ/Asia users we have, who have to suffer the main outages when the UK is in bed. If we hit this root cause, expect a dramatic performance increase. If we don't, we will beaver away at it until we do.
The site will speed up a bit for you if you were to hit the Mark All Forums Read link on the Index page of the forum, as we had to do before.
Thanks,
Damocles
