Inspired by the very similar thread about school incidents.
The big incident at my previous job was the (extremely incompetent) HR person accidentally sharing a spreadsheet of every employee’s salary. I was an hourly worker at the time and thought it was funny, but some of the senior engineers were pissed to find out how much more other engineers made. HR was not fired, instead she was put on a temporary paid leave.
Other incidents include:
- One of the owners’ vape exploded, forcing everyone to evacuate.
- VP of the company got caught buying escort services on the company credit card.
- Aforementioned owner (who was no longer an owner after the company was bought out) got fired shortly after a town hall zoom meeting, where he used a Trump/Pence background.
- The head of the NOC team had a stroke at work.
- COO “accidentally” revealed in front of everyone that one of the Project Managers had cancer. She sued, and they settled.
- COO verbally abused one of the senior managers over the phone during a big meeting. He sued, and they settled.
It was a pretty shitty company.
So my team lead ordered a bottle of a particularly nasty chemical (don’t remember what, this was long ago) Thing is, he went on holiday immediately after and didn’t tell anyone about it. The bottle had to be shipped and kept at -20℃ otherwise it would decompose into a deadly gas; one of those lovely CMR types. So next thing that happens is, I get called on my day off by the boss saying that there is a scary box in the lab and if I could check it out. I was reluctant but I figure I’ll check it out. I though to myself that it was a little unusual that they would ship it in a styrofoam box without any dry ice (it had evaporated) so I take out a slightly bloated bottle which seems to be filled with some liquid. I then tear of the package label and read the MSDS. At this point I read all the scary labels and realize that this thing has been out here for a while, all the dry ice has evaporated, the bottle is bloated and filled with gas which I am sitting right next to. So I turn on ventilation and GTFO. I inform the boss who immeditally got his home freezer to cool it down. I meanwhile started to notice a stinging in my eyes (which was one of the effects) We wash out my eyes, stinging goes away, lungs are fine and everything ends without injury.
TLDR
Coworker orders deadly chemicals, goes on holiday, doesn’t tell anyone, almost kills me.
Well that’s rude.
I was supervising filling in a pit we had dug on the edge of a forest. We had dump trucks coming in dumping gravel. One particular driver wasn’t great at his job and there had been issues with him in the past.
That driver came in and dumped his gravel, but then he drove off with his bed still raised and almost immediately smashed into electric lines that ran off into the forest. One telephone pole even snapped at the base and fell over.
Within 30 seconds multiple cops came speeding onto the job site. It turns out those electric lines ran to a radio tower in the woods that ran the police radio. The idiot in the dump truck had taken out the police comms for the whole town.
Note: if you’re planning a crime in that town, you only have to cut one wire to disable all police communication.
That’s some lacking infrastructure
A productive team member was heard giving their daily stand-up report during their team’s daily stand-up…
To another company. Oops! Don’t forget to mute your mic if you’re working two jobs at the same time!
INC-224, never forget.
I am an infra engineer at a fairly large scale (not like Amazon, but we have some BIG customers) SaaS company; despite our scale, we are only like 250 people and of them only about 90 engineers. We store a bunch of data in MySQL.
15:30:00, I get a page “MySQL table is full.” I immediately know my day is ruined, since I’ve never heard of this error before, but know it ain’t great.
15:30:10, every Pagerduty escalation policy in the entire company gets bombarded with pages.
I look at the database instance. The table size is “only” 16TiB, so it’s a bit confusing.
We are hard down for several hours as we scramble to delete data or somehow free up space. Turns out, google backs ClpudSQL MySQL instances with ext4 disks instead of zfs, and the max file size on ext4 is… you guessed it, 16TiB.
We learned a LOT of lessons from this, and are now offloading a shitload of json into either MongoDB or gcs, depending on the requirements. The largest table is down to 3TiB now :D
Worked at a place where our CIO was completely unqualified to be a leader, much less a leader in IT. She was a micromanager who took the position of “telling stakeholders” instead of “working with stakeholders” so any project she was on was really her pushing through whatever agenda she had at the time. Meanwhile her deputy CIO was stealing computer equipment from the server room but I digress…
April fools one year and I decide to prank it up. I moved the hinges (not the door handles) of the freezer/fridge in the breakroom so that the handle and hinges were on the same side. It’s a fifteen minute job to move everything so I did it the night before the 1st.
The next morning our hungover CIO stumbles into the breakroom and cannot get the fridge to open. After a few seconds of futile tugging on the handle, she gave up and took her lunch to her office.
Others in the office figured it out pretty quickly and had a good chuckle.
Later on that day CIO sends out a nastygram about pranks being unprofessional, property damage, someone was going to be in huge trouble, yadda yadda…
But she’s not the director. The director tells her to basically fuck off, it was a funny prank, and perhaps she needed to lighten up.
She never found out it was me.
The overnight IT guy was caught watching porn while working (this was over a decade ago, he was in the office every night and not a remote worker). How was he caught? He was saving the pornographic photos on a shared network drive…
When confronted, he didn’t try to deny anything, his explanation was simply, “That’s just my thing.”
Boss: “Were you looking at porn in the office?”
IT guy:
One of my very first tech jobs, there was a guy who watched porn in his office, as far as I could tell just continuously. I saw the DNS logs so I was aware. I didn’t care as long as it wasn’t interfering with my ability to get things done.
One day I had to go in his office and talk to him about some of his code that was breaking, and he went OH and tried to hide all the blatant porn on his screen. Like dude I know. I don’t care. I am here to talk with you about your shitty code not your personal failings and issues; those are purely your own problem IMO.
An IT contractor at my government job was one of the people that tried to kidnap Michigan governor Gretchen Whitmer.
Software company before git. The source server corrupted and the product code was lost. 5 guys had to get together and figure out the latest version between them (everybody had different changesets) and produce a new “current” version. At the end we lost all history prior and ever since all changes prior to 2008 have been attributed to 1 guy.
Gotta respect that save. Reminds me of the Toy Story 2 assets being lost from a server failure and they were saved by one employee having a copy on their personal computer at home.
I used to work at an accounting/consulting firm who were dead set on writing business applications in VBA within Excel. The code was embedded in the notebook, and to distribute the software was sending the latest version of the Excel file. This made version control virtually impossible, and we would instead combine our work manually.
I cannot recommend having tech-illiterate people lead software projects.
The amount of times I hear people telling me that “I should just do it in Excel”. Excel. Is not. A database.
Excel is a whole OS unto itself. Like Emacs except you can get out of it.
Had an executive assistant at my company who did very little if anything. Nobody knew why she was kept around and paid so much. Everyone pressured the CEO to fire her, but he strongly resisted. Eventually she was fired, but immediately threatened to sue for sexual harassment. CEO threw her a lovely settlement check despite claiming that nothing ever happened. Mmhmm.
The CEO sent out the email that COVID vaccination was required for all employees or the people that refused would be terminated, like most workplaces. It was fully expected seeing as it is a hospital, and all but a handful of employees compiled. This moron decided to reply all (the reply all to CEO emails has thus been disabled since) with a lecture full of antivax nonsense about how COVID vaccines were experimental and contain fetal cells, and the revelation that they had been reading patient charts that they had no business reading for COVID test results and the consultation or ER notes, and wrote about the “proof” that they had that nobody had died or gotten really sick from COVID, and how the CEO was extremely misinformed on the subject of COVID. That resulted in immediate termination and it was pretty hilarious to read their nonsense and the fact they admitted to every employee they had been violating patient confidentiality. You want to be that dumb, have at it.
Years ago I worked for a large-ish post production company. They had recently moved into a swanky new location and everything there was tailored to spec, including the server room. In norwegian we sometimes call a server room a ‘machine room’, this is relevant.
As a part of the server room spec, a dry fire suppression system was among the requirements.
The summer of the incident was particularly hot, and we experienced some trouble with our cooling, so a cooling technician was called to have a look. While he was working on the unit inside the server room, he made a mistake that caused all the cooling gas to dump into the room, triggering the fire extinguishers.
A dry fire system works by releasing an inert gas into a space to displace any oxygen, effectively choking any fire. I imagine this is usually done by some solenoids opening some canisters of gas and the room quickly, but gradually becomes oxygen free. Luckily, my boss at the time was present and he quickly got both himself and the tech to safety.
All good right? No. The contractor who constructed the new location had ordered and installed a system meant for maritime machine rooms, not the computer ‘machine room’ we had. In an environment filled with fuel and grease, you optimize towards filling the room with an inert gas as quickly as possible, and it turns out they use explosives to complete the task. In this room there were three canisters in the ceiling with fire shooting out of them, burning pellets to generate the inert gas. The gas and smoke from the canisters combined with the leaked cooling gas, and started condensing.
Into hydrochloric acid.
While all this was going on, all of the servers and workstations were happily humming along, sucking the now extremely corrosive atmosphere into themselves, making sure that every nook and cranny inside and outside got covered in a thin greasy film of acid.
The aftermath: Mine and two colleagues’s summer break was cut short, as we were called in to do damage control. Ripping out and wiping hard drives clean was what we did all summer. With external help we managed to recover all of the data. One feature film was delayed a few weeks. The insurance payout actually made the company a bit ahead financially. As far as I know there’s still burn marks in the floor of the server room, from when flames shot out of the fire extinguishers. Everyone involved now knows what a proper dry fire suppression system for a server room looks like.
The kicker is, the cooling was messed up because a fabric awning on the building had fallen down and was covering the air intake. If anyone had thought to check the roof this whole thing would have been avoided, and that server room would probably still have bombs attached to its ceiling.
One day a coworker of mine was walking into our huge office building and thought he saw a mitten on the ground of the lobby. When he picked it up it was actually a pair of lacy women’s underwear. Ostensibly it fell out of someone’s gym bag or got caught in their pant leg in the laundry and dislodged there. He drops it immediately and comes into the office. He doesn’t mention this to anyone.
Two hours later the main receptionist comes in with the underwear in front of our whole group and says she saw him drop these this morning and she wants to return them. He’s denying the whole thing and at this point none of us have the previous context and all locked in to the conversation and silent laughing. She says, “We just want to give these back in case they have sentimental value!” and the the whole group is dying laughing now. He eventually convinces her he isn’t interested in a stranger’s underwear (which she bare handing) to which she says she’ll keep them in case he changes his mind (???).
It’s been 5 years and it gets brought up nearly daily
This was actually after I left, although I have a million crazy stories about this place- but the guy who I used to work directly under was an alcoholic, and one day Monday he didn’t show up for work, wasn’t answering his phone, etc. This was pre-social media, so they couldn’t ask around or anything.
He comes in a week later and it turned out that over the previous weekend, he had gotten drunk, driven from where we lived in Indiana down to Georgia for some reason (he had no connections in Georgia), went into a bar in some small town, got into a fight, and wound up in the slammer for a week.
He was no longer employed after that. And this is a small business where every employee was so vital to the owner that I once got mad at him, screamed, “GO FUCK YOURSELF, [his name]!” and stormed out and went home and he called me up the next day, apologized and begged me to come back in with various compensation promises which I can’t remember. It took a lot to get fired from there, but that was enough.
Previous HR was well beyond retirement age essentially working to have something to do and one day emailed all of management a spreadsheet asking us to verify our information. That sheet contained each of our full names, addresses, phone numbers, birth dates, social security number, etc.
To my knowledge nothing of significance happened. I have my credit frozen.
I worked for a company that handled a ton of personal data. Pretty much every person in Germany, including addresses, bank account details, etc.
On my first day there (fresh from university) I was given literally full read access to the entire database. And as I later found out by accident: they did not track any data exfiltration at all. I copied several gigabytes of data without anyone noticing.
Your data is only as secure as the least motivated data broker sees fit. And that’s not very fit.
