DreamBot Downtime

[Nuclear Nezz 2093]

Hello everyone, after possibly our longest downtime to date we figured we owed it to you to let you what happened and why it took so long to get the update out. The update ended up taking nearly 89 hours, which doesn't include the ~2.5 hours after the game update when we weren't able to actually start on the update (see https://x.com/oldschoolrs/status/1909935042807148552).

What happened
tldr; I got very sick at a very unfortunate time.

On Apr. 8 I started to come down with an illness, but didn't think a lot of it because I generally bounce back from things pretty fast. Then I slept for roughly 20 hours.

On Apr. 9 at 12:50:00 UTC the game was updated.

Usually updates go pretty smoothly, if anything is broken I can fix it easily or Pandemic is able to take over. Generally our updates take 1-4 hours, in rare rough cases up to 8.

Unfortunately, this one was larger than normal. I realized a large number of our hooks were broken which made the client unusable. I knew this was going to be hours of work, worst case days, but due to my illness and the current time (10:30pm), I was not going to be able to work through it. I hoped another night of sleep would get me at least to a point of being able to work.

Pandemic and I discussed if he'd be able to step in, and decided that by the time he was even able to understand it and start, I would already be well enough to do it.

Each day went by with that same discussion, with the hope tomorrow I would feel well enough to do it.

On Apr. 12, I felt well enough to put in a couple hours of work throughout the day, but I knew it was going to need a lot more.

Finally on Apr. 13, I was able to sit down and give proper focus to the work, and got the client back up and running around 5:48:00 UTC.

I do genuinely appreciate all of the support I received any time I popped in to give an update on discord, and we appreciate the patience you've had during this time.

How we plan on avoiding this in the future
Having a single point of failure, like a single person to be able to fix the updater, obviously isn't great. However, in this specific case, it really seems to have been just unfortunate timing. If any of the events had happened differently (the update came at the normal time, the update didn't break our updater so badly, I didn't get sick) then the outcome likely wouldn't have been so disruptive. There's a reason this hasn't happened before in our entire 11-year history.

Many comments suggested hiring an additional developer, but this wouldn't have really helped in this case unless they specifically worked on the updater. I would likely no longer focus on or be up-to-date with those changes as we're a very small team and try to focus on separate areas with our specific skills and experience. While the outcome could have possibly been better in this scenario, we don't think it'd be prudent considering how rare something like this downtime is.

We think the best way to move forward and minimize the chances of something like this happening is by improving our internal tooling to be more robust so that it's less likely to break during such game changes. If this had been a normal non-breaking or only slightly breaking update, if I'm not immediately available, like this time, Pandemic could have ran the updater and updated the client. With this in mind, though, it's entirely possible that this could happen again with a similar outcome due to the nature of the our business. We have no prior knowledge of the upcoming game updates or how they may break the client or updater. No software is perfect, and we unfortunately can't predict the future.
 
What we're doing for monthly subscriptions that were affected
All affected monthly subscriptions have now had time added to make up for the extended downtime. VIP and Sponsor subscriptions were extended by 7 days, and all other subscriptions were extended by the actual downtime (plus a few extra hours to make up for setup time) as we're unable to extend those further. Any VIP or Sponsor purchases that expired during the downtime have been extended by 7 days, and you should have received the role again. Anyone with a VIP purchase that expired during the downtime (or impacted otherwise) and then bought a separate VIP purchase after the downtime have had their new VIP purchase extended by 7 days.

Thanks for reading,

The Dream Team

Edited Monday at 03:57 AM by Hashtag
Fixed link to X

[Meteorite 72]

I hope you and your wife are feeling better. Thank you for everything. 💜

[Calvin808 2]

We love you, always take good care of your health. Health > everything else. Good to hear you are doing a lot better than before. Praying for everything to be good!

[comtruise123 1]

All the best mate.

[kramnik 19]

Glad you are feeling better now  Just wanted to ask, maybe you would consider throwing a bone to people who mainly hold ton of Lifetime instances and Lifetime sponsor? I have only couple monthly subscriptions at the moment so extentions wouldn't do a lot, but still lost a lot of money due proxy, server costs and having few hundred bonded accounts that were losing days. Personally I would like some discount code for limited time or purchases. In the end that would be Win-Win situation since that would boost up your sales.

[discowiz 10]

Hey Nezz, glad to hear you're feeling better - hope it's in the rear view and you're back to feeling your best.

Following a 4 day outage, the below language isn't what people want to hear;

"However, in this specific case, it really seems to have been just unfortunate timing."

"we unfortunately can't predict the future."

"There's a reason this hasn't happened before in our entire 11-year history." 

I'm fairly new to this community, but having spent a sizable amount here in the last 2-3 months - including 16 lifetime scripts, this update hasn't filled me with confidence to continue spending additional $$ here. It doesn't really sound like the single point of failure is being addressed. 

Additionally, would like to express that the communication from your team whilst you were out was poor. There were haphazard updates, screenshots from randoms in a thread, and a banner that came up late (and was out of date for days). People searching through forums to find an update, with a bunch of imbeciles arguing about who's gonna sue who is not ideal.

Anyway, just my 2 cents, hopefully some meaningful changes will be made to ensure that an extended outage is avoided in future. 

Edited Monday at 07:18 AM by discowiz

[boomslap 7]

Very happy to hear that you are doing better! I have to say that this down period produced some great comedy content in the forums and on Discord. It was pretty entertaining to watch and participate.

Jokes aside though, I don't think that a one man show can support such a project where a lot of people are investing their money in. I understand that everyone has their expertise, but I think that It is important to have a fallback mechanism. Since you are two main developers, I believe that Pandemic should also know how to patch the updater in case of need. Sure it might not be his expertise but he should have the knowledge or at least an overview of what he is supposed to do so that he can accomplish it in case of need. Even if it would take him two or three times the amount of time that you would need to update it yourself. 

That would be my suggestion and I think that most people would be satisfied and pleased to know that if something were to happen to you, then there is already a fallback plan of someone who could continue this project.

I wish great health to everyone! ✌🏻

[Sokeripala 1]

Thank you for fixing it. Hope you stay healthy

[RudyShields 5]

Thanks for the extra VIP days. Glad you got better.

 

[YUNGTHUGFATGP 127]

This is why I just lifetime everything

 

edit - glad you got better buddy

Edited Monday at 04:05 PM by YUNGTHUGFATGP