Written warning from employer for following manager's instruction to deploy bugfix directly to production

manpreet Best Answer 2 years ago

I'm wondering whether you wonderful people here could offer me some advice on how to deal with a written warning that I received today.

Background

I've been working for a software company for the last 8 years who deals with broadcast email. Clients use our service to upload their customer lists and send them promotional emails, track email responses etc. Some of these emails can be sent out to hundreds of thousands of people at a time.

There was a problem with one of the applications involved in sending the email. I was assigned to look at the problem by my immediate manager. I had not really seen this part of the application before, so its process was completely new to me; I had to learn as I went.

After some investigation I found the issue and reported back. My manager then asked me to make a change in order to fix the bug. However, I was asked to do make this change in the live version of the application, not the development version. For the non-software developers out there, this is not best practice and is potentially dangerous.

I didn't question my manager as I assumed (wrongly) that if he's willing to allow me to make a change like this to the live application, it couldn't really be that catastrophic if something went wrong, so I made the change. I asked about how we could test this and he replied:

"Client X have scheduled an email to go out to 5k clients in 10 minutes; that'll be a good test".

He then left for his lunch, and so did I. Upon returning, I was informed by the support staff that the process had failed, meaning an incorrect email had been sent to their entire database rather than a subset of their clients.

The fallout

This morning, our managing director arrived at the office and was extremely cross at what happened. He hauled both my manager and myself in to a meeting immediately and asked what happened. My manager stayed mostly silent throughout the entire meeting, leaving me to do all the talking. At the time, I didn't want to throw my manager under the bus as I wanted to allow him to admit his mistake in instructing me to do the job on a live system. However, this time did not come. The managing director then stopped me talking and issued us both with a written disciplinary notice as there is a potential to lose this client who is worth about £200k per year.

My question

I've been given the right to appeal the written notice, however I must do so within 7 days. I feel as if this written notice I have been issued with is unfair as I was only acting under direct instruction from my manager. I spoke to my colleague about it today, and she told me that she heard the entire conversation between my manager and I about testing on live data and thinks that my notice is unfair.

Do you think I have a point for appealing this decision? and if so, how would I approach it? We're a small company (10 employees), and I don't really want to make things awkward between me and my manager in the office.

manpreet 2 years ago

There are numerous comments and answers criticizing what the standard practices not being followed etc but I know from personal experience that such incidents are not too uncommon in small mom-n-pop type of non-tech companies. There is usually no change management, no standardization, inadequate test/staging environments, no version control...and everyone (sometimes even the non-IT guys) have administrative access to production databases/servers. In such companies, most small enough changes go directly to production/live systems because the existing developers know the systems inside out (such systems are not too complex either). But when someone new joins, he/she is very likely to make such mistake initially when not properly supervised.

What happened is clearly that your manager took your word for it when you said you have fixed it but as it turned out, you had not fixed it. It is not your fault either because you were new to the system and your manager should have known better.

That being said, I think this could actually be a great opportunity for you to either get promoted or gain sufficient respect in the eyes of your managing director for the next appraisal. Below is what I would recommend you can do -

1. Setup a meeting with your MD (and anyone else higher up except your immediate manager) and explain to them what transpired. Don't blame your manager and don't take the blame all on yourself but do explain the facts of the case. Don't forget to make it clear that it was not your own idea to mess with the production system and that were following your manager's instructions throughout (but again without directly blaming him, a tricky conversation). If you can't do this conversation without making it look like you're backstabbing your manager then don't do it.

2. Then emphasize on the fact that the real problem here is the lack of industry best practices like change management, release management, TDD, continuous integration etc in the company. Explain how they were followed in your previous organizations and that such issues were nonexistent there. Tell him confidently that you are well versed in those practices and if he likes, you can implement them here as well. This step is is going to launch your career in this company because this will change you from the-new-guy-who-messed-up to the-new-guy-who-will-make-our-problems-go-away. Do make sure to thoroughly understand the best practices before this meeting so that you know what you're talking about, people in senior positions are usually smart enough to see through lies.

Remember that the key is to use all of the tact at your disposal to shift the focus from who did it to what the real problem is and how you're going to implement the industry best practices to make such issues go away forever (and doing it while giving the impression of being completely honest and sincere, believe in yourself and your ability to do it).

The above steps will most likely eliminate the need to worry about the written warning and stuff and you'd have placed yourself in a very favorable position with your MD. In addition, you'll have the opportunity to implement the best practices that will not only help strengthen your position but also improve things in your company across the board. It's a small 10 people shop, if you're good with your MD, you can stay there forever.

Good luck.