Q&A with Leah Melvoin, Creator of Amazon’s “Voice of the Developer” Initiative

In 1996, self-proclaimed “alternative kid” Leah Melvoin started at Amazon as a Special Orders Books Buyer — with no computer experience and no college degree. Long before automation, she and her team members worked near large filing cabinets, updating orders and order values into a manual database. Leah eventually learned to write software on the job and worked her way into the role of a Software Support Engineer for the Supply Chain organization. The on-call rotation was “awful. We routinely experienced up to 100 pages per week”. This prompted Leah to petition for the managerial role of the team. Her unique background led to what turned out to be effective out-of-the-box thinking (“because I had never been in the box!”). She re-designed the team to grow support engineers into SDEs, a model that was then replicated across the entire Consumer/Operations organization. She also managed a software development team. Over the next fifteen years, she would create and grow the transformative Voice of the Developer initiative and an annual engineer-focused Tech Survey that still guides Amazon engineering today.

Now retired from Amazon, Leah now offers her expertise as a consultant to share what she learned. She recently sat with Uplevel to talk about engineering effectiveness and her two-sided relationship with data.

What has been one of your greatest successes or something you’re most proud of as an engineering leader?

When I became the manager of the software support team, I found junior software developers that didn’t meet the Amazon SDE hiring bar and structured their work to get them there. I’d give them opportunities to work on below-the-line features, arranged mentorships, and emphasized root cause resolution — in essence I guess I created a sort of a “dev ops bootcamp.” We created a tool very similar to PagerDuty and a cascading monitoring tool called “Smart Alarms” that alerted the right dependencies causing outages (vs. spamming everyone in the pipeline). My support model was very successful. Our share of problem resolution increased from 45% to 75% of incoming tickets and on-call pages decreased. As for the SEs themselves, over a four-year period, I promoted 21 Support Engineers to Software Developer and all of them did well. Many achieved senior SDE levels, others senior management levels. One is now a Senior VP!

The other thing I’m very proud of is the creation of the Amazon Tech Survey [“Voice of the Developer”]. I didn’t have a lot of help — I had to beg, borrow, and steal resources. In the time that I ran the survey, we had a 90-95% response rate. It was basically a “census”.  I started out surveying 1,500 engineers, and by the time I left Amazon I had expanded the survey to include all Amazon technical roles (over 65,000 individual contributors and management). It became such an important mechanism that it still exists today.

What has been one of your greatest challenges as engineering leader over your career?

I was a disaster as a software developer manager. I had never learned how to manage a significant software project prior to becoming a manager of a software team.

I fell into all the traps common to inexperienced managers: I allowed too much scope creep, I randomized the team, I didn’t prioritize my engineers as well as I should have.

We delivered the right automation, but I didn’t understand the implications of design decisions we were making. I was pushing myself, but the role was beyond my capability at that time.

In my opinion, to be a solid software development manager, you really do need experience as a software developer or a technical program manager FIRST.

When I hit ten years with the company I thought, “I’m leaving” — I struggled with Imposter Syndrome and I was losing confidence in myself. I was fortunate that a leader I respected offered me this interesting opportunity: to create a “Voice of the Developer” program focused on understanding developer requirements and prioritizing the best features to improve engineer productivity.

Prior to becoming the leader of that program, they had two other people in role, who were both formerly engineers. They approached it systematically, driving one feature at a time and implementing. To me, this was too slow. I knew there was more that we can do. In retrospect my approach was pretty irreverent. I said to the SDEs, “Tell me everything that’s bothering you.” That opened the floodgates. I received thousands of emails complaining about everything wrong, everything slowing them down, everything that affected them — tiny things, big things, you name it. This began a 13-year period (2006-2019) focused on the resolution of hundreds of pain points. It was overwhelming, but fun. The program was really successful.

How would you define engineering effectiveness?

To be productive and engaged, engineers need a frustration-free product management environment.

Give them the opportunity to be a part of solving problems — don’t just give them solutions. Help them see the customer or business need. Allow them to be part of the design process.

When a company is small, getting features out quickly is the key to survival. But over time, the shortcuts made start to bog engineers down. Code that takes a long time to build or merge is the beginning of a death by a thousand cuts. Each problem layers on top of the other, increasing their frustration factor. It’s the same when engineers are randomized by management or pulled into meetings or surprised by work needed.

Engineers need to develop their own culture of pride. They need to agree on coding conventions, make sure all code is reviewed (thoughtfully) and tested before releasing. They need to decide which best practices are expected (and should be culturally reinforced). This is how engineers demonstrate ownership and are self-acting, not relying on management to guide them. To have an effective engineering culture, leaders need to encourage this type of ownership. If the product management cycle is entirely about meeting deadlines…you’ll end up with a lot of problems.

What are some signals you use to know that your team is effective?

It comes down to two questions:

(1) Is the customer satisfied?

(2) How hard was it for the engineers to deliver those experiences?

What’s your best advice for an engineer being promoted into their first management role?

First, read Steve McConnell, especially Software Project Survival Guide. Next, engineers that become managers need to take the time to understand job levels and what engineering capabilities map to those levels. Think about people development before delivering product. Think about managing.

For many engineers, the #1 issue is lack of career growth, but it’s also important for managers need to know what talent they have to work with. Get to know each engineer: past experience, how they feel about the work they are doing, and what their goals are. Determine if they need or want coaching (or a mentor). Evaluate whether they are meeting or exceeding the expectations of their level. Managers that skip this step, risk being unprepared during performance season. Managers also need to be cognizant of job levels to make sure the work assigned is appropriate for the engineer. We always want to set folks up for success. This means appropriately stretching engineers to keep them challenged and promoting them when they demonstrate they are operating at the next level.

Reflecting on your past experiences, what data have you found most valuable to help you manage an engineering organization?

Companies always want to measure happiness, but that’s a fairly ephemeral emotion. Instead, ask employees what they want their leadership to know about their work experience. Listen to them and respond appropriately. Leaders spend a lot of time thinking about what they want to say to their teams, but not a lot about what their teams might want to say to them.

What early warning signs do you look for to know that your deliverables aren’t on track?

Every company, no matter who you are, starts off scrappy, not necessarily thinking about how things will scale as they grow. You end up with a lot of software in various forms, some great and some horrifyingly difficult to work on. No comments, no explanations of functions, vague business logic, short-term workarounds, etc. Projects start taking longer and longer to complete.

Not all code “debt” is a problem, but you’ll know if it is when complaints about why it’s taking so long for deliverables to ship start to become common. “Accrued code and quality issues” slow engineers down. Systems get complex. The mechanism to mitigate this risk is allocating engineer time on code-related issues, as part of regular housekeeping. So often the tendency is to put even more pressure on engineers to deliver features – giving them arbitrary deadlines – which only results in even more rushed code (vicious cycle).

Have you set goals or used data to manage your organization that turned out to be ineffective? Why?

We all want quantifiable data to measure developer effectiveness. Unfortunately, there are some types that just aren’t that useful. For example, how often an engineer checks in code or how many lines of code they write code may not directly determine that engineer’s performance. It’s a style thing. One can assume that checking in more often or writing a lot of code is a good thing, but it may not be. A junior software engineer will often write a lot of code. A senior engineer might write very little code, spending more of their time thinking through a very complex problem or on system design.

You need the context to understand whether quantifiable metrics are indicating a problem (or not). An engineering manager should be connected enough to their team to have this context, even in a remote environment. It’s much more challenging for senior leaders, especially as companies get bigger and move faster.

Data should give you the right information to review what’s happening, but where engineering is concerned you need to think about the engineer, their level, their style, and the complexity of the work expected. Most of this can’t be digested for you and labeled as ‘bad’ and ‘good.’

Managers need to determine if it’s bad or good for their team.

Leah Melvoin works with companies to share her expertise in engineering and technical talent management. For more information check out Voice of the Developer LLC.