Development Process | Hrakaroo

Start with On-Call

Mon, 22 Apr 2024 00:00:00 +0000

This is my second post on running a standup, you can read my first post about walking the board here.

Start with On-Call: A Healthier Way to Begin the Day

On-call work can be isolating, exhausting, and easy to overlook if you’re not the one holding the pager. I’ve had nights where pages hit back-to-back, leaving me exhausted and irritable the next morning. When that happens, the idea of opening a standup with project updates feels disconnected from reality. What I really want to talk about is why I was paged—and how the team can help make sure it doesn’t happen again.

Rethinking Standup: Begin with On-Call

Standup is usually the first meeting of the day and exists to align the team. But before diving into project updates and task boards, start with one question:

Did the on-call engineer get paged overnight, and if so, why?

This isn’t a courtesy check-in—it’s a signal that the well-being of the team matters, and that operational issues are real work. If the overnight experience was rough, it might be appropriate to pause project tasks and focus on the underlying problem.

Classify Every Page

To make these conversations productive, each page should land in one of three categories:

1. Known Issue Already Being Worked On

If the root cause is already under investigation—underprovisioned services, a bug in progress—these pages can be deprioritized. Ideally, you’d mute the alerts, but that risks hiding new issues. Just be honest and precise: don’t group unrelated pages together.

2. Rare, One-Off Events

A “rare” page should truly be rare. Over time, the definition should stretch from once a month to once a quarter to once a year. If you’re getting paged weekly, it’s not a one-off.

3. New, Uninvestigated Issues

Anything that isn’t known or rare belongs here. These deserve investigation tickets. Failing to follow up leads to the same interruptions night after night—and often much bigger problems down the road.

Prevent Escalation: Keep Small Problems Small

Investigation work should be treated as a top priority for the on-call engineer and ideally the secondary as well. Small problems are easy to ignore when you’re busy, but they’re even easier to fix before they escalate. As the backpacking adage goes: keep small problems small.

Protect Sleep: Rotate When Needed

Being paged at 2 AM isn’t just inconvenient—it’s damaging if it happens repeatedly. If someone was paged between 9 PM and 9 AM, consider swapping them with the secondary for the next night. Guaranteeing a solid night’s rest can make all the difference—and reinforces shared responsibility. In some cases, you may want to also let the on-call person log off early to recover.

Why This Matters

In many teams, managers don’t participate in on-call rotations. When page volume is high, that disconnect can breed resentment, especially if nighttime interruptions feel invisible. Starting the day by focusing on on-call sends a different message: the team’s experience matters, and operational pain is a shared concern.

A small shift—like opening standup with one intentional question—can change how teams operate, collaborate, and care for each other. And for the person who held the pager last night, that shift can mean everything.

Creating Simple and Effective Guidelines for Code Reviews

Tue, 09 Oct 2018 00:00:00 +0000

The following is a blog post that was written from a speech I gave about how to set the ground rules for code reviews.

Following New Relic’s Project Upscale—an innovative reorganization intended to make our development teams more autonomous—the engineering organization formed several new teams, one of which was the New Relic Database (NRDB) team. As the name implies, the NRDB team is responsible for the development of our events database, which powers the New Relic Insights tool as well as several other products.

When we formed the NRDB team, it included several senior-level software engineers. This was a highly skilled and very passionate group of developers reviewing one another’s pull requests.

When passion turns toxic

Being passionate about your work is one of New Relic’s core values. In this case, however, we may have experienced too much of a good thing: our code reviews soon became collision points, and we increasingly turned to passive-aggressive communications to settle our differences.

In the example on the left, the reviewer left the PR in an in-between state. They didn’t explicitly reject it, but they didn’t approve it either. In the example on the right, the reviewer made a highly subjective request, and the author just made the change, but from their tone you can kind of guess that they didn’t appreciate the feedback.

As a result, the NRDB team’s developers grew increasingly frustrated, team trust eroded, and several members (myself included) contemplated switching to other teams. We were in trouble.

Refining our process—and saving the team

We decided as a team to take a step back; we resolved to figure out what was going on, why it was happening, and what we could do to fix it. Since most of our frustration was tied to our code reviews, we started by asking a simple question: how could we give one another more effective and constructive feedback?

We answered the question by developing four basic guidelines for code reviews. We think you’ll find them useful, too, but before we spell them out, we want to share the full story behind what happened to divide our team and what was really as stake for us.

A flawed approach to the code review process

Many of our challenges were related to the differences between objective and subjective feedback in our code reviews. Being able to differentiate clearly between these two types of feedback can be critical to the success of a code review, and to the effectiveness of a development team. In too many cases, we weren’t handling subjective feedback in a constructive manner—in fact, just the opposite was true.

We probably aren’t the only ones who struggle with this issue. Many developers are trained from the start to downplay differences between the two types of feedback. In fact, students in academic software engineering programs rarely learn how to give or receive critical feedback of any sort.

When I went to school, this certainly was the case. The computer science curriculum focused on algorithm analysis, data modeling, and problem solving. Our instructors treated code review as a functional quality-assurance task; they rarely presented it as a creative process. Code review feedback tended to be straightforward: The code either worked, or it didn’t. Because of this kind of training—or rather, lack of training— many software engineers still treat all aspects of code reviews as completely objective activities.

It’s useful to contrast this approach with the one employed in an academic creative writing program. There, instructors conduct workshops that include training on how to give critical feedback. Creative writing instructors understand that giving and receiving critical feedback is an essential part of the creative process. They also understand, however, that critical feedback can be harmful and create resentment unless it is handled properly. The goal is to provide feedback in a positive and constructive way that helps to hone a writer’s ideas, enhance their creativity, and leave both parties enriched by the process.

The struggle over subjectivity

Many facets of a code review, however, are not straightforward. In particular, there are issues that demand subjective assessments for which there are no “correct” answers. This is where the rigid emphasis on code review as a totally objective activity, and the failure to consider the creative nature of software development, can become a problem.

Many elements of a modern code review process are now fully automated. Editors and IDEs will find syntax errors, evaluate Boolean logic, and warn about infinite loops. As a result, the bugs that survive are much harder to find, especially when you’re at the end of the process and are just looking at a code snippet with limited context.

Editors and IDEs, however, can’t detect—or prevent developers from focusing on—subjective issues such as confusing method names, questionable style preferences, and bad variable formatting. And when we dislike and disagree with what we find in such cases, we often forget that these “flaws” are subjective matters of opinion—not objective matters of fact.

This approach also makes it easy to forget that a debate over subjective issues during a code review can get emotional and heated very quickly.

Some teams try to regulate this problem out of existence by creating style guides that make objective rules out of subjective preferences. This approach rarely succeeds: software development is full of subjective choices, and there is no way to cover every subjective choice that developers may face in the course of project.

When a team lacks a clear communication channel for subjective feedback, the problem gets even worse. Reviewers may mix their subjective and objective comments without acknowledging the differences; here too, the process can end in resentment, frustration, and a breakdown in team communication.

Our four guidelines for code reviews

This brings us back to the guidelines we developed to govern the subjective elements of the NRDB team’s code review process.

First, as a preliminary to our four guidelines, we agreed to define who is ultimately responsible for the correct execution of any code changes. This was important to us because in a subjective debate, the opinions of the person who has the ultimate responsibility—in other words, verifying code execution— should carry the most weight. As a result, we decided that “The author of the code change is responsible for the correct execution of the change.”

This may seem obvious, but not all teams work that way. Some teams, for example, treat the review process as a QA process where the reviewer is ultimately responsible for verifying correct execution.

We found that subjective comments were most often presented as objective feedback at the pull request stage of the process. As a result, this is where we focused our code review guidelines.

In creating these rules, we laid a foundation for team members to clearly identify what a code reviewer should look for, and how to give both subjective and objective feedback. Here are the guidelines:

The reviewer should identify errors that will cause an issue in production. It’s a code review, after all, so the reviewer should identify missing semicolons, unending loops, or missing error handling. Reviewers aren’t responsible for finding all such errors (that’s still the responsibility of the author), but they should be on the lookout for obvious issues that will break the system if they’re are deployed into production. Such issues are a valid reason to block the pull request.
The reviewer should verify that the stated goal of the code change aligns with the changes being made. If the author submits a pull request that says they’re making changes to the networking code of a service, reviewers should expect that all of the changes are in and around the service’s networking code. This seems obvious, but it’s no secret that developers have a tendency to try to pack in multiple changes in such cases. This isn’t even necessarily a wrong practice, as long as the changes are mostly co-located. When you align a code change to its stated goal, however, you make it easier to determine if the pull request potentially submits any new bugs. Here, too, we agreed that failing to align the code change with its stated goal would justify blocking the pull request.
The reviewer should verify that any changes align with the team’s coding standards. I’ll cover this more in a bit, but as an example, if the team has decided that all variables must use camel case, and the reviewer finds a variable that does not use camel case, they should block the pull request.
The reviewer should look for anything they personally disagree with. This guideline addresses any comment which the first three rules do not cover. We want reviewers to give feedback, even if it’s not covered by the first three rules. We didn’t want our guidelines to suppress feedback, which is essential for how we learn from one another. Because these comments are clearly subjective, however, we agreed that they do not justify blocking the pull request.

To remove all confusion, we ask that reviewers specifically call out their comments as either blocking or non-blocking; and to add those comments as tags in their reviews. For example:

Objective comments

“Blocking: You are missing a semicolon.”
“Blocking: This loop never ends.”
“Blocking: You are missing some error handling here”

Subjective comments

“Non-blocking: Your method name is not clear enough.”
“Non-blocking: You should put the open curly brace on the line above.”
“Non-blocking: You should use camel case for your variable here and not snake case.”

Working within our code review guidelines

As we adopted these guidelines, the team had the most difficulty with the fourth one. Adopting this meant we had to accept two conditions:

The code our team produced did not need to be uniform. This meant overcoming a trend in our industry that says you should strive to remove all fingerprints from your code that identifies who wrote what part. We found the ROI on following this trend was pretty low, and trying to do so just led us back into the same subjective debate: If a developer writes code in a manner slightly different than their peer would, does that mean the code is incorrect? Clearly, we decided, that wasn’t the objective case.
If a reviewer adds non-blocking feedback, the author should take the time to consider it. Early on, some team members were concerned that authors would simply ignore all non-blocking comments, as their code was no longer blocked by subjective feedback. Our solution, then, was to reiterate that “we trust our teammates.” If, as reviewers, we took the time to enter a comment, we trusted that the author would take the time to read and consider it.

These both were contentious points, and the team spent a long time debating them. But ultimately, we found that the only way to work through these issues successfully is to live with the guidelines and give them a chance.

Sponsoring a coding standard

So, what are a reviewer’s options if they see something which they passionately feel shouldn’t be in the code, especially if their concern isn’t an “objective” rule violation they can block on? For such concerns, we agreed that a reviewer could choose to sponsor an addition to the team’s coding standards.

Every two weeks, we hold a retrospective meeting where team members are welcome to suggest changes or additions to our coding standards. There are two restrictions to this activity:

We cannot describe coding standards in subjective language. For example, a sponsor can’t say, “variables must not be ambiguous,” as ambiguity is subjective. But, the sponsor could add a standard that states, “variables must use Hungarian notation,” as this is objective and easily enforceable.
If you sponsor a coding standard, you must support it. The sponsor must provide documentation and training as needed. If there was a plugin or other tool the team needs to installed, the sponsor is responsible for supporting it. This restriction ensures the sponsor is passionate about anything they want to add to the team’s coding standards.

Finding respect and compromise in code reviews

After agreeing to these guidelines, we cleared all our existing coding standards and started over. For the first few weeks it was hard to break old habits, and we had to remind several team members to add the blocking and non-blocking tags during their pull request reviews. But once we got rolling with the new guidelines, we saw a number of successes.

First, by forcing reviewers to clearly identify those comments that were subjective, we noticed a change in how reviewers phrased their comments.Reviewers can no longer demand changes that meet their preferences; instead, they must request changes politely, and explain why they’re requesting the change. When we provide more explanation and context in this manner we create an environment that makes it easier for teammates to learn from one another. Plus, asking for changes, rather than demanding them, shows respect and acknowledges that the code’s author has valid feelings about their work, as well.

We also noticed that when a reviewer did write a non-blocking comment asking for a change, the author typically made the requested change or came up with a compromise—even though the author had the option of ignoring the comment. This demonstrates why asking for changes, rather than demanding them, builds stronger teams: the author feels less resentful, and the reviewer feels that the author genuinely appreciated their feedback.

We’ve identified a few other terrific benefits from this process. By limiting the scope of what qualifies as a blocking comment, for example, we reduced the time it took us to approve and merge changes, which resulted in greater overall project velocity. We have also reduced the time required to onboard new new team members and to get them up to speed with our code review process.

We have also updated our training materials to reflect our new code review process: We distribute one page that documents our guidelines, and another page that documents our coding standards. New team members now know exactly what they should be looking for and how best to communicate their suggestions.

We also expected the number of coding standards to increase greatly as reviewers sponsored new standards for items they could no longer block on. At the beginning, we did adopt several new coding standards, but after an initial burst, the number of new agreements fell off significantly. We concluded that since reviewers felt that authors were taking into consideration their subjective feedback, they did not feel as motivated to “convert” them to objective constraints based on their point of view.

These guidelines aid in team autonomy

The most important thing about these guidelines is that they support team autonomy; in no way do these guidelines dictate which coding standards teams should adopt. Teams are free to choose their own style guides, and they decide how strict they want to to be. These guidelines simply explain how to define coding standards and how reviewers should look for and give feedback.

We have come to appreciate the role that a strong and effective feedback process can have on building team morale, increasing team trust and communication, and improving development velocity. We implemented guidelines to strengthen the feedback process and to address issues that put the process at risk—and so far, I think we’re getting exactly what we hoped to get from these improvements.

Walking The Board

Sun, 09 Jul 2017 00:00:00 +0000

In the traditional standup format everyone stands in a circle and each person, in order, gives an update. The updates follow the agile format:

What they did yesterday
What their plans are for today
Are there any blockers

I have a number of problems with this format.

Context Switching

People don’t usually stand in any particular order so the updates from person to person often jump between the project(s) the team is working on and support tasks. Additionally, if we’ve sized our tasks correctly, then a single person could potentially be working on a new task each day. This makes sense to the person giving the update, but for everyone else, trying to track actual progress becomes increasingly difficult.

Project Status

Related to context switching, it’s nearly impossible tell the overall project status from this type of update. Even if you could figure out how many tickets were closed, without looking at the kanban board it’s nearly impossible to get even a sense of what is left todo on the project. Plus, this format does nothing to encourage people to actually update the kanban board, compounding the issue.

Public Speaking

Public speaking does not come naturally to everyone, even when presenting just to the team. I’m much more relaxed with it now, but there was a time when this type of setting would cause me enough stress and I would ignore whatever was being said and just rehearse what I was going to say. Then, after I gave my update, I would spend rest of the standup obsessing over the things I had messed up or forgotten to say.

Competition for Busiest

Without focus, the standup can become a passive competition for who is the busiest person.

I’ve got a meeting with a customer today to talk over a feature they want added.
Well I’ve got two meetings today to talk over features.
I’ve got meetings pretty much all day long so I’m not likely going to get much else done.

It sounds silly, but it’s something I’ve seen happen at multiple companies, especially if people are feeling insecure (imposter syndrome). It can take the form of lamenting (but really bragging) about the number of meetings, or making the tasks you completed sound like herculean efforts. It’s pointless and unhealthy for the team.

Forced Update

Finally, it forces everyone to talk when they may not want to. Maybe they just got back from vacation and are still working at getting caught up. There have also been times in my career where my personal life has taken over and I’ve needed to take some time off, be it medical or family related. At the update I don’t want to go into details about what I’m dealing with. In fact, I may be at work specifically to take a break from the other stresses going on. The last thing I want to do is be forced to talk, or worse, explain why I don’t want to talk.

As an individual contributor I find this standup format stressful and as a project lead and manager I find it unhelpful. Instead I prefer using the standup time to walk the kanban board.

Walking the board

With most ticket tracking systems you can use swim lanes to split up your kanban board into project tasks and support tasks. Then, starting with the top swim lane, go down the Done column and each each owner can give an update for their ticket. (This also avoids that initial awkward moment of trying to figure out who should be the first to speak.)

Once the Done column is finished, move to column to the left (likely In Progress or Blocked) and repeat the process. Continue until you are done with all assigned tickets for that swim lane. Rinse and repeat for the next swim lane until all updates are done. The context switching is much less and the team can focus on what is being said instead of trying to memorize what they are going to say.

(Since Done is often an end state and you don’t necessarily want to re-walk this entire list every time, some teams will create a Closed column and use the standup as an opportunity to move tasks from Closed to Done.)

Closing comments

One critique of this format is that it is possible for a team member not working on a task to feel left out. There are a couple of ways to handle this: First although standup is suppose to be project focused, so I’ve always stressed that if you don’t have any updates for the project it is okay to not talk. (This assumes your team is a safe space where individuals won’t feel like they are being judged if they don’t talk.) But, if an individual really wants to explain to the team why they haven’t picked up any tasks, they should save that for the post-standup discussion. If, in the rare case, an individual is checking out and not picking up any tasks, that is something the manager should bring up during their 1:1 and not in front of the team.

As a project lead and manager I find walking the board to be significantly more useful as I can tell exactly the progress of the project (as I’m looking at the tickets) and I can also be sure that tickets for the project are up to date. As an individual contributor I don’t feel like I have to prepare as much for this standup and I can just attend and focus on what everyone else is saying.