Dev Deletes Entire Production Database, Chaos Ensues

Bike race music was the best part of the video.

and that is the "onosecond"

"accidentally deleted the database, gg"

gg

Never `rm -rf`. Always `mv dir dir.sav`. I am surprised they decided to `rm -rf` at all in the first place. Just move the data, deal with removal later. This is highly irresponsive.

Those who have felt the dread after screwing prod data up can sympathize.

They accidentally delete data without knowing for sure if there is a backup available, and they get rewarded by DRI, damn.

I'm surprised that this happened on a Tuesday. This is the kind of thing that happens on a Friday at 6PM

Does this get us fired if we were in his shoes?

I know how he felt after deleting db1😂😂

Isn't DMARC verification a process from the receiver side to decide if an email is received from the legitimate sender? Why would they set anything regarding DMARC verification on the server that sends the email? I guess you mean they didn't set the DMARC DNS record to tell the receiver what to do with the emails they receive from the server. But that still doesn't make sense since the lack of a DMARC DNS record would not make the receiver reject the emails or make the server not send them...

"linux is better"
yeah right

damn that was a horror movie

I am still waiting for the video “how gitlab deleted their production database… again”

This is why you save in multiple slots.never know when you might softlock your progress

and that is why you change foldernames instead of outright deleting them

Cant believe they forgot about the emergency blood sacrifice

I once accidently dragged an important file i to the trash bin. I learned to keep my trash bin far from my important files.

Checklists like they use in aircrafts are very handy when it comes to mission critical operations.

I like how it's being narrated in Chornobyl nuclear accident story manner.

There is a magical command called mv that does everything rm does and is completely rollback-able.

What kind of psycho doesn't just rename the file?

Ask chatGPT first before implementing things

are u sure it's not out of malicious intent?

You can’t prevent mistakes but you can sure prepare for them!

Oof moment

Bruh

Been there, done that, only in my case it was taking down the main network interface on a Solaris YP server used by an entire site of Solaris servers and workstations. The entire site ground to a halt in an instant. I didn't have access to the DC to get local access, either, so I had to make a red-faced confession to my boss for him to make the 2 mile drive to the secure DC.

When I was just starting in a company, I accidentally deleted all the ticket intervals from the database. Causing all the tickets to close immediately and make some massive spam to the admins. I was really terrified of the situation and didn't know what to do, we didn't have any backup as well. I apologized as much as I can and didn't make another mistake like this again in years, sometimes mistakes make you work harder and be more careful in life.

Bruh

The problem with postgres backward-version incompatibelity is serious. It should get tackled in future versions.

Yeah, at approx 2 commands per second I let someone else check every of my commands before I execute them. In what kinda Willy Wonka Wonderland are you living?

Me who understands less than 50% of the words: hmnn yes interesting wow

linux does have an undo
just not when you use the rm command

Had a friend who his co-worker deleted sql because their app's endpoint was pointing to redis, so he assumed sql was useless xD

The best action in such cases is no action until root cause is determined.

GG gitlabs, may you prosper <3

I deleted the main site from our backend in my first month as a full stack developer. Fortunately i figured out how to rebuild the apache server and clone the repository but i definitely worked well past my hours that day and the stress was crazy

Further proof that computer engineers don't know enough about the systems they are using, they just assume pasting code they find online will fix their problem and if not they can search a forum for help. Gross.

Proper documentation helps a lot while handling crucial data.

This is why you use managed DB services like AWS RDS.

how did these people get to be dbas in gitlab?

its 3:00 am here rn and that slack notification sound at the start scared the shit outta me

explained it like it was the chernobyl nuclear disaster

How on the h would this person have the access necessary to delete the entire production database?

Is it crazy to have an offline backup of your directory? Even if it's not fully update to date, isn't that better than nothing bc you know, mistakes happen? ...EDIT: oh

Lol that's why you don't test in production

When you are deliting anything, check, recheck and rererecheck before hitting enter.

(*×*)

Microsoft Embrace, Extend, EXTINGUISH