Astuteboy
Well-Known Member
Damocles said:Firstly, just want to say thanks to those who kept us company on the Twitter feed during the long night. Lots of time spent moving servers is looking at progress bars creep up, and it really broke up the night so to speak.
Anyway, as you may have noticed, we've spent the last 24hours completely down. This was because of a general annoyance with the last server and the hosts; we made repeated requests to them for very specific things and they were entirely unhelpful in every respect. Yesterday, I had a chat with a guy on here who offered to give us some advice, namely Phil Lomax of Dennis Publishing who made a few suggestions on how to improve the performance of the site. After bothering him for a good hour or so, Ric received an email back from the former hosts blaming us for the server problems in the stupidest way possible. They basically said that Ric has a busy site, so we had to just deal with it, or in technical terms ("can you not reduce the queries on your database?").
To try and explain the magnitude of this stupidity, I've just taken a quick screenshot of the current server, albeit without caching with helps this number a bit:
112,000 queries and 7,800 connections in 20 minutes and we aren't busy; you can multiply that by ten when we are busy, and this is after a year of us working on the code to optimise it. So, no, we can't reduce them any further. Dickhead.
This was actually the last in a long line of stupidity, and I suppose that we got sick of it all in the end. I certainly got sick of hearing their bullshit, and I've no doubt that every time Ric had to convey their message on, he did so with a total disillusionment.
Thus, we had a chat and decided to move to a new type of server called a cloud server. It's a system ran by Amazon, called Amazon Web Services (or more to the point, EC2) which allows us to 'create' servers on the fly. Props to Ric here, as despite it costing him quite a bit more, he went for it straight away and this is right before Xmas with a young family to look after. You might want to consider the depth of this commitment when using AdBlock and the like of :-)
Anyway, the last 24 hours have basically been a collection of us trying to get everything back online, but with a huge focus on getting everything exactly right. There's been too many times in the past where we have had to firefight problems that have been caused by us rushing to get back online, at least this time we know that our software stack is exactly what we want.
Speaking of the server, there will be some rockiness for the next few days or so, whilst I get the configuration of it exactly right. New servers mean new equations to balance, memory wise. Weird thing here, we're actually starting from scratch, yet are in front of where we were before the switch already.
We need a big test of users, hopefully we'll get that against Juve, but we're confident that with a bit of patience, we've properly cracked it. It certainly feels like it to us anyway, and the load times will bear this out over the course of the next few weeks.
Anyway, due to all of this, you will get dodgy errors/slowness every now and again; this is just me restarting the server to update the configs (or I'm in bed and something unexpected has happened). Thanks for bearing with us over the last day or so, and we ask for a little more patience over the next few days, and it will be all fixed soon.
We have a few known errors so far. You won't be receiving emails about having a PM yet, Tapatalk isn't currently working exactly as we would like, and there's generally some little bits and bobs to be done. Obviously, there's a bit of login problem for some users, but this will sort itself out in time.
Cheers guys, much appreciated. I think it's Paul Lomax btw rather than Phil (if we're thinking of the same guy).
I'm sure the big user test you're after will come next time we lose!