team management

Support In; Complain Out

The #watercooler used to be a physical space. Not a slack channel. (Yes, kids, we didn’t always have LaCroix and fancy bottled water.)

And I believe the watercooler can be a healthy place for colleagues to gather and discuss ideas. Some companies even promote cross-functional teams meeting to discuss things wholly unrelated to work because those discussions can spur ideas. And the relationships formed in those moments can benefit the team — and the company — in a real way.

Great Minds Discuss Ideas; Average Minds Discuss Events; Small Minds Discuss People.

People work well with people they trust. People work well when they feel safe.

But the watercooler (which can come in many forms) can become malignant. The watercooler can turn from a place where people discuss ideas to a place where people discuss people. And that is one of my canaries for a toxic work environment.

I’ve long loved the quote, “Great Minds Discuss Ideas; Average Minds Discuss Events; Small Minds Discuss People.“ It’s often attributed to Eleanor Roosevelt but that credit is perhaps unfounded.

I’m the first to say I don’t always live up to this ideal. That’s the thing with goals and principles — you don’t always achieve them. But they do serve as a guide to redirect you when you become lost.

Humans Are Pack Animals

I’m human. I don’t like some people. And not for any good reason. I just don’t like them. And there are people with whom I have no interest in forming a longterm personal relationship. This isn’t always a reflection of them as a person. We just don’t jell. And that’s OK. The beautiful part of this diverse and thriving world is that we all have (or should have) our people.

There are also people with whom I have personal disagreements due to conflicts of principle. And this is where humans tend to get into trouble.

I have strong feelings about healthy conflict. And my natural personality (along with my childhood in DC) lends me the gift of being extraordinarily comfortable with direct conflict. I used to have a saying, “I don’t talk shit. If I say shit behind your back, it’s because I already said it right to your face.” And I lived up to that. Because the culture I was in allowed for it.

I’ve always preferred that approach — perhaps mostly because that’s simply the culture I grew up in — because I always knew where I stood with people. I knew X didn’t like me. And, well, fuck them. But more than that I was secure in the fact that no one was running around talking behind my back.

Although public confrontation is uncomfortable (painful?), it comes with a gift. You always have the chance to defend yourself. To meet the accusation. And that style of conflict, while more intense, is finite. It ends.

The Insidious Nature of Gossip

The cousin of direct conflict is indirect conflict. This typically comes in the form of gossip. And social ostracization. If you can’t (or won’t) have direct conflict with someone, the next option most people fall into is turning others against them.

And this behavior isn’t always malicious. Sometimes it’s just habit. And there are a lot of reasons those habits and patterns of behavior exist.

I can no longer always live by the rules of conflict I once did. For three reasons:

  1. Not everyone is comfortable with it. And solving conflict means meeting people where they’re at. I have learned to appreciate that not everyone is as comfortable with direct conflict as me. And I try (and sometimes succeed!) at resolving those disagreements in different ways.

  2. Some people are much better at managing public perception than me. I used to naïvely think that direct conflict and gossip were mutually exclusive. But, turns out, that’s not actually true. Some people will gossip about you for years after a direct conflict is “resolved.” Go figure.

  3. I’m a public figure. For me, public conflict has changed from my local community of people who know me and morphed into thousands of people on the internet that see a fraction of who I am. In addition, I have a platform and, because of that, my words come with extra weight. it’s easy for me to actually be punching down. I try to be cognizant of this and overcorrect for it.

I’ve talked about the danger of gossip and tribalism in tech as a community. And I ruminate on this problem a lot. Because it is a cancer on the community I want to see all of us cultivate. A community where everyone — everyone — feels safe and welcome. And perhaps that utopia is, well, just that. But you know how I feel about unattainable goals…

Gossip Is Contagious

I think the most dangerous part of gossip is it is malignant, in every sense of the word. It grows quickly and indiscriminately. And it’s incredibly difficult to destroy without also destroying healthy cells.

But more than that, it can quickly become the norm. And people at the edges of gossip can mimic the behavior because they’re under the (sometimes unconscious) perception that to belong — to be liked — they must gossip. Or choose sides.

Venting

And I also want to be careful to acknowledge that being able to vent is healthy and important. I vent. Mostly to Jessica West. She knows things. But she also knows that when I whine, I’m mostly just moaning in the moment and after a bath and a sleep — and probably too much champagne — I’ll have a plan to work through it. As my confidant, I trust that she will keep my words confidential. I trust that she knows I don’t want her to run around sharing my heated thoughts with the entire community. She consistently earns that trust and that type of relationship is absolutely priceless. I’m forever grateful for her friendship. And the friendship of so many others.

It’s OK to vent. But what’s the difference between venting and gossip? Well, that’s a tough question. And like all things in tech, it depends. But I do think there are two guardrails to help keep you (mostly) within bounds.

  1. To whom you complain.

  2. How you complain.

People Matter

I think the second guardrail is more nuanced. I want to address it. Perhaps at another time. For now, I want to focus on the people to whom you complain, vent, gossip, moan, whine.

I first heard the concept of “support in; complain out” from one of my early mentors, Karen Flood. And it always stayed with me. I’ve done my best to visually illustrate the concept below.

You more than likely exist in multiple communities or groups. Work is one of the easiest scenarios to use to consider and evaluate this concept.

I think the healthiest organizations have space and room and trust for solving interpersonal conflict directly. That conflict will be contained and resolved. But achieving that trust is a long road and most of us don’t work in those extraordinarily healthy companies.

So what I’m suggesting to you is that instead of gossiping — or accidentally falling into the habit of gossip — you choose to complain out and support in.

chart take two.png

Complain to managers, mentors and personal friends. Ask their advice. Seek their guidance. Vent. And trust that those words will stay with them. Support your colleagues, employees and community members. Speak positively. Recognize the good things. Celebrate successes. Build trust and rapport.

If you’re not comfortable talking to your boss (or your boss is the problem), seek out your boss’s boss. And if you don’t feel safe doing any of those, it may be time to consider another workplace. There are many of us who will gladly help connect you and make introductions.

And finally, talk about ideas. Not people. You exist in a community of some of the most brilliant humans on the planet. We are better than gossip. We are better than division. We spend a lot of energy cutting others’ down. Winning others to “our” side. What if we stopped? What if we spent that energy another way? What if we solved the hard problems. Together.

complain out.png

Humpty Dumpty + DevOps

I’m convinced Humpty Dumpty is a story of DevOps gone wrong. 

Humpty Dumpty sat on a wall,
Humpty Dumpty had a great fall.
All the king's horses and all the king's men
Couldn't put Humpty together again.

First, who asks a horse to do surgery? Hoofs can’t hold scalpels. Second, either the king’s men are inept or they’re not communicating. Two kindergarteners with some Elmer’s could have done the job.

You see, Humpty is a deploy. He was fine in staging but shit the bed in production. Now the site’s down and your boss is threatening everyone’s jobs. IT is saying the code is broken. The developers are saying it’s a server issue. 

Meanwhile, Humpty is bleeding out. And your customers are complaining on Twitter. Which means a customer service rep has entered the #incident channel to tell you the site’s down. Yea, no shit, Tom. 

Sound familiar? 

We’ve all been there. A deploy goes awry and the entire department is up in arms, defending themselves and blaming each other.

All The King’s Horses and All The King’s Men

So there we are. In chaos. The site’s down. The boss is pissed. 

I don’t know about you, but I always think best when my boss has morphed into Al Capone and I’m staring down the barrel of a metaphorical Tommy Gun.

Not.

At first glance, you might think we’ve assembled the best team to handle the crisis. There are engineers who know the code inside and out and an ops team that can handle any systems fire. 

Yet, it doesn’t work out that way. It always devolves into a blame game. You know the scene.

Developers say…

  • “It’s a server issue.” 
  • “My code worked in staging.”
  • “A configuration must have changed.”

Operations people say…

  • “Has to be a code change from the last deploy.”
  • “What was just deployed?...”
  • “Why are we the only ones that know what is broken?...”

All the king’s horses and all the king’s men aren’t working together.  

Let’s Stop Fighting

Ok, first, I’m not going to address all-out fist fights. Because honestly, if your department hosts a weekly version of Fight Club, you should really change jobs. 

I’m talking more about what could be described as friction, attitude, or a general inability to tolerate each other without eye rolls and audible sighs. What I like to call good ’ol Southern-style passive aggressiveness.

I’ll give you an example. 

I recently had some mild conflict with one of our DevOps guys. A day after a deploy, a feature on one of our sites — the ability for admins to upload new photos — wasn’t working. The user didn’t receive an error message after uploading. The new photo just didn’t show up. 

A project manager (PM) messaged me and an ops guy.

I had 20 minutes before another meeting — why are there so many meetings?! — so I felt a little pressure to locate the issue quickly. I should have recognized that I was on edge and ill prepared to deal with the situation at that moment.

But I didn’t. I’m human. 

To make matters worse, an eerily similar issue had come up during testing in QA. During that code hunt, the ops team lovingly implied that it was my problem. And after an hour or two of log reading and double-checking my work, I discovered it was in fact an ops issue. 

Not that it isn’t my fault sometimes. I make plenty of mistakes. And then I obsess about them for months...

Fast-forward two weeks and here we are again. So I’m primed and have plenty of attitude. My bad. 

I holler across the room to see what the logs say. 

“I don’t know.”

Um, wanna go look it up?! 

OK, I didn’t actually say that. But I’m 80% sure I got the message across with my eyes. 

So I track down the production logs while coordinating with the PM so I didn’t have to test the issue in production. 

Minutes pass and now I’m in my meeting trying to do both. Because multi-tasking is proven to be so effective. 

The production logs have nothing but 200s and everything looks good. 

Finally, the ops guy checks the S3 logs. Surprise, surprise. The image is there. Pff! Not my fault. (My inner dialogue may or may not be an eight-year-old.)

Yep, you guessed it. Another ops issue. 

Now it’s not that I think operations issues are easy. They scare the shit out of me. But I get a little huffy puffy when I’m constantly met with “it must be the code.” And I’m sure it’s beyond irritating that ops teams constantly get “it must be a server issue.” 

Which brings me to my core point: we need to work together, guys.

“Dev”.concat(“Ops”)

Change is hard. I dislike it as much as the next person. But I think this cultural shift is worth the struggle. 

By now you’ve probably heard something about DevOps. It’s all the rage these days. 

But if you aren’t an expert in what exactly the term DevOps means, here’s a quick history. 

The term was coined by Patrick Debois and Andrew Clay Shafer while attending a conference in 2008. Hilariously, Patrick had planned to speak about DevOps at the event, but received such negative feedback that he decided to skip his own session. (TIL don’t give up on ideas just because you get a poor response.)

John Allspaw and Paul Hammond joined the #devops conversation with a talk called 10+ Deploys per Day: Dev and Ops Cooperation at Flickr. The talk is 40 minutes but very much worth your time. Just play it during dinner tonight. Your kids are gonna love it. Promise.

Since then, DevOps has become a term that encompasses a company culture where developers and operations people work together. 

Traditional Thinking

Before we continue toward DevOps nirvana, it’s important to recognize your development team has a fundamentally different priority than your operations team. 

Like it not, developers are measured by the number of features they release. No CEO has ever cracked open code to review your thorough test suite or pondered at the glorious variable name you picked out. (I appreciate it, though. So you have that going for you.) 

If all of us decided to tackle our growing mound of tech debt this month instead of working on the latest and greatest idea your sales team came up with, you better believe we’d be hauled into someone’s office and chided. 

But operations people are measured on an entirely different aspect of the business: site reliability and uptime. And you better believe keeping a site up 99.999% of the time is no easy feat. 

I’ll spare you the math. That’s a little over 5 minutes downtime per year. FIVE. MINUTES. PER. YEAR.  

So, to break this down, developers must deploy new code to release new features. But deploys are the most frequent cause of downtime. 

No wonder we’re natural enemies. 

Be The Change

What we need is operations teams that think like developers and developers that think like operations people. 

It’s not easy. But it is simple. 

Operations: Empower Your Developers

Trust your team

You’re on the same side. If a developer says the code works, trust them. They’re not lying to you. And they don’t want to make your life a living hell. They honestly believe the code works. Which brings me to...

Give read-only access to all developers

To what? To EVERYTHING. I’m not saying to hand out root access like candy. But you are not the gatekeeper of information. Do you like being interrupted every 5 minutes so you can copy and paste an error message? I didn’t think so.

Developers are writing the code that runs on your systems. It’s not a reach to think they should be able to get some feedback about whether it works. After all, don’t expect developers to jump in and help when they don’t have access to your machines. 

Create consistent platforms

Integrated platforms are easier to develop and support. Pay attention to the parity between environments. Staging and production should be identical. That means the same allocated resources and the same data. Otherwise deploying will always be a roll of the dice. 

Share source control

Keep your configuration tools on GitHub with the rest of your company’s code. Code is code. It’ll be much easier for operations and developers to solve problems together if everyone knows how to locate the affected code.

Add your devs to the on-call rotation

My friend likes to say, “You build it, you support it.” No one likes to be woken up at 2:00 a.m. And if you’re tired of stumbling through the dark toward your computer in the middle of the night, share the pain. There’s no reason developers shouldn’t be on rotation. Remember, they can access logs and view your configuration tools now. Awesome!

Simplify deploys

Pushing code to production should not be a production. Unnecessary steps increase the opportunity for error and decrease the number of people who can deploy.

Oh, one more thing. Stop preventing developers from deploying their code to the QA and staging environments. Seriously. If I have to ask permission to test my shit anywhere other than dev, you deserve to put out the fire. 

Developers: Stop Being Assholes

Make operations part of the planning process

Thinking about a feature? Include operations. Talk about what will change before you write a single line of code. Discuss why this feature is important, who will need to be involved and what the risks are. You can’t deploy mystery code and then get irritated with your operations team when they start asking 100 questions. 

Make small changes. Deploy. Repeat.

If your feature requires you to change 30% of your app’s spaghetti code, break the feature into smaller pieces. Not sure if your feature is too big? Apply the same rule you use for method naming. If the method needs an “and” it’s doing too much. Small deploys make it MUCH easier to determine what went wrong in case of failure.

Communicate

You know how you already included operations in your feature planning? Notify the operations team when you deploy too. Whether you use Slack or HipChat, make sure all developers and operations people have a single place to communicate. Lots of companies use an #incident channel. Find what works for you and then use it. 

Yes, and…

There’s a rule in improve that forces participants to say “yes, and…” rather than “yes, but…” Try this next time you’re in a meeting and the results will likely surprise you. That simple language change will make everyone feel heard, validated and a part of the team. 

Be open to other options

If someone on operations says there’s going to be a problem, listen to them. That means shutting your mouth and really hearing what they have to say. You’re an engineer, not God. The core competency of operations is site reliability. Let them help you. The solution you come to together will be much better than the one you thought of on your own.

Have some humility

If someone was woken up in the middle of the night because of something you released, say sorry. Buy some coffee. Help ’em out. Own your shit. When you take responsibility for a mistake, your colleagues are much less likely to make a voodoo doll of you and keep it by their bed. 

Practice Failing Together

Failure is never a question of if, but when. 

You will fail. A deploy will bring down the site. A typo in your configuration will bring users to Twitter fisticuffs. 

It happens. We’re human. And until Skynet, we’re all stuck dealing with our occasional mistakes. 

Have a healthy attitude around failure

You need to 80/20 your failure preparedness procedures. It’s okay to spend 80% of your time trying to prevent failure, but devote at least 20% to practicing how you will handle failure when it happens. We all half ignore the safety talk given at the start of every flight, but I appreciate that oxygen falls from the ceiling in the event Bane decides to crash my plane.  

Stop pointing fingers

Avoid blame. It never feels good to make a mistake. And when 20 people are required to rectify it, it feels even worse. When I screw up, I’m embarrassed. And if I feel attacked, I become defensive. I think most of you would probably say the same. Let’s give everyone a little slack. It could have been your typo. 

Leave your egos at the door

When I started powerlifting seriously, I joined a small team of intimidating lifters. The head of the group — a 60-year-old Juggernaut-like man whose traps rose to just under his ears — had one rule: leave your ego at the door. It didn’t matter that we had to strip off 400 pounds every time it was my turn to squat. All that mattered was that I listened, learned and respected the team. We could all learn a lot from that. 

Recommended Reading

Here’s a short list of books you may be interested in:

And no, I haven’t read all those books. I’m convinced people lie about how many books they’ve read. Or I watch too much Netflix. Don’t judge me.