The story behind the website crash

[Hani AMJ]

The story began two weeks ago when our hosting company had their auditors for an annual audit of accounts. The audits are compiled using their billing system that is linked to all hosting accounts including ours. Unfortunately, their billing system had a glitch and reported that out account and a few others with custom hosting plans are several month over due on payment. The natural reaction for the billing system was to suspend the account. I contacted the host with proof of payments and request they resume the service this was their reply:

Dear Hani,

First of all, let me apologize on behalf of my crew for the delay in responding to your inquiry. It was my intention to delay our response to you as I wanted to make sure we have all the information we need beforehand.

We value your business very much and so I am going to be very honest with you. Our company's auditor recently conducted a routine yearly accounting audit through our billing software. They detected several billing inconsistencies which affects a small group of customer accounts. One of these accounts is yours. These inconsistencies include missing payments, missing invoices and missing hosting accounts.

Further investigation revealed that the inconsistencies were caused by our billing software, rather than human error. So we contacted our software developer at once and together, we managed to uncover the root cause of the problem. A software patch was made and applied to our billing system yesterday to permanently rectify the problem.

Unfortunately, before the patch was applied, the glitch in our billing system carried out an automated deletion of your entire hosting account, including its backup files, without our prior knowledge. Our systems engineers tried every possible option to recover the deleted files from your server and despite a continuous effort of 72 hours, none of the recovered files were usable. They were all corrupted one way or another.

I know that nothing we do or say will be enough to express our regret towards this matter. What matters most right now is to get your website and e-mails back online. Since we failed to do this on our own, we would appreciate very much if you could provide us with your assistance.

I have setup a new account for you on the server. You should receive a Welcome E-mail from us by now so please use the provided log-in credentials to log-in to your new hosting account and setup your e-mail accounts right away. This will let you receive e-mails again.

Next would be for you to provide us with any backup that you may have on your website so that we may restore your website for you. We await your backup and update.

After reading this email I almost fainted, knowing that I have lost all the data of the past 4 years. Our account was so huge because of the gallery therefore there was no way to make backups locally. We had over 80,000 pictures in the gallery totaling 9 GB and the file system of the website was also a few GB’s in size. I completely refused the fact that we have lost all the date and tried finding a solution. I contacted a professional data recovery company to attempt and recover the data however our host refuse to send us the array of hard disks as it violates the privacy agreement they have with their customers. All roads blocked I had no choice but to accept reality and start thinking how I can proceed. I had the Nov back up of our ‘database only’ stored on my computer but only to find out it was corrupted. I sent the file to the host anyways to see what their team could do with it and I also started rebuilding our data base MANUALY by extracting bits and pieces of our old database tables from the corrupted file until I successfully manage to rebuild a whole new sql file with all the data that you see on the website now. Next I was facing a problem of communication between the forum source code and the database due to the changes in version as the current forums we are using is version 3.0.6 against 3.0.4 in our old website. a simple upgrade script and a template of the web resolved the matter and we are back in business. A couple of days after the hosting company contacted me after they have achieved the same result that I had.

In summary, all date including upgrades, posts and ranks have been lost. The gallery is completely wiped out, all avatars, attachments in post and PMs are also lost as well.

As you can see the website is running with basic features only, it should take me a few weeks to rebuild everything all over again and hopefully it will be better than before.

If you lost a star and have a problem with your profile please PM and I will resolve the matter for you.

I appreciate the support that a large number of members have offered, and I would also like to send out a special thx to Iguana and JO who were with me every step of the way.

A special thanks also the team at server fly who were working round the clock to try and get us back in to business.

[mike0967]

What to say except a huge thanks to you and the crew! AD4x4 is not a website, its in our hearts and souls .... think this is what counts most.

Mike

[G-man]

Antrax.... Our Hero!

But I've never heard of everything being deleted "automatically", including Backups of backups... because a customer defaulted on a payment... very weird setup they have there.

[Xof]

wow... what a weid story... looks like a huge american lie to me... anyway, result is that we have to rebuilt!

So more pictures to be taken!!!! More trip to lead, and more post from Watheq!!!

thanks Hani, Islam and Jo... btw, i hope we don't pay for the hosting for the 4 coming years!

[nuclear]

What a story & bad luck !!

thanx Hani

[belbassiouni]

thanks god e are back, and thanks to you for your efforts Hany

[Aidy]

No regrets Hani, you did ur best and are always doing so. Am sure there is a good reason God had this happen, and the club is always in our hearts as well as all the good memories

Thanks again Hani, ur the man!!!!!

[Wasabi]

wow Hani, you are the man!!!!

thank you so much, I was lost without the site.............. and I had to work during that time, I coudn't surf

Wasabi

[Xtreme]

A very Big THANK YOU Hani and all AD4x4 great Team.
I feel it's an American lie too, doesn't make sense, but AD 4x4 is still here.
What is gone is gone, still we all have nice moments and memories to remember with this great family.
Hani, you did it before and we are sure you will do it again and even better.
Would love to help, but not my field, but I'm sure you will find great support from other members.
Good Luck, we are all supporting you.

[Abdoush]

Welcome back guys

[waqmaster]

Great to see the website is back up again!!

How about we send a global request so that anyone with pics of the ad4x4 trips upload them on the ad4x4 facebook page? Anyway the gallery before was open to the public...

Best

Waqar Ali

[Viking]

First of all I would like to say THX to Hani but also Islam and Jo for their support and assistance.

Secondly I am very surprised that something like this can happen and I agree with Xof that it very much sounds like a constructed truth to cover up for something they dont want to tell you.
In my ITIL process profession this is exactly what you need to prepare for and make sure it is not happening. If I was running a business that was depending on my website I would replace them immediately as they very obviously dont have control of their backup-processes and there is no continuity-process in place. For an american company I would say that this is more or less out of the question as their maturity should be far higher than these basic mistakes are showing.

I guess we will never find out what really happened so again thanks for all your effort Hani and I am happy that we are up and running again. Please let me know if there is anything I can help out with. Maybe we should request their continuity-plan in order to make sure that they dont do the same mistake again or at least have a plan for it.

[Abo Omar]

Thank you very much for all yr time, efforts and dedication ... Thanx to Iguana ans Jo for their assistance as well ... The good thing that AD4x4 is up and running again .... BTW i have a backup for most of the avatars that u sent me once before if u would need it ...

[agghazal]

God help you man and wish all the best if u need any help we r here to save ad4x4

[maddog]

Stirling effort guys.
I cant begin to imagine the drama involved in piecing the jigsaw back together again - i have enough trouble keeping my little cyberworld on the rails!

This story is a good lesson for me in fact, as all my flying logbook records are currently stored on an online server. Time for a backup of the backup me thinks!