Many of us work for prolonged periods at departments where the performance engineers group together to share their knowledge. These departments have a variety of names: performance competence centers, performance center of expertise, load and performance testing center, or even (God forbid) load testing factory.
Besides allowing for a rapid exchange of information and ideas that these performance competence centers, as I will call them now, offer for the performance engineers themselves, these centers also provide a single point of contact for the organization they work in for performance-related expertise. However, the firms we work in, or work for, are far from static, undergoing dramatic changes in the way they are organized, the tools they use, and the demands that are being imposed upon them: faster delivery of software into production, with better quality, and with lower costs, promoting continuous learning and continuous improvements in the organization.
Some performance engineers (not you, of course) might think that these changes will have little impact on their performance competence center, and that their position will be unchanged, and definitely not threatened. They might say: “Haven’t we provided enormous added value in the past? Haven’t we shown again and again that we can prevent performance problems from happening in production? And that we have helped to solve issues in production? We have scarce knowledge and expensive tooling: surely, the organization will appreciate us for our work! How can they operate without us?”
While all of these statements may be true, these ideas might not be in the heads of decision-makers. I have seen organizations where Scrum, Agile, and DevOps were introduced in a dramatic way, cutting off all overhead and making teams responsible for all components of software development, including non-functional and performance. In fact, the performance competence center was seen as a threat to the introduction of DevOps. I was at one of these companies, where management made this very explicit: if they kept this central point of contact, the teams and the delivery of software would still depend on this center, with “lazy” teams and blockers for the Agile processes as a consequence. So that’s what they did: all competence centers and central groups, including monitoring, including security, and, yes, including the performance competence center, were dissolved, and the engineers were given two choices: join a DevOps team and do your work there, or leave. And many left.
For a while, what the managers intended came true. Software was delivered faster into production, at lower cost. Teams felt they were not blocked, could decide for themselves what is production-ready software or not. A wave of enthusiasm spread through the organization: Agile worked!
But not for long. The issues began. Weird and inexplicable behavior was heard of in production. Customers complained of slowness. And of short periods of unavailability. With the lack of an end-to-end monitoring solution, these issues were not internally confirmed, could not be researched, and, in the beginning, also not taken seriously. How could software that was produced using these super-modern processes and with super-modern deployment tooling, be anything but flawless?
The problems persisted for more than a year. The company was being heavily criticized for the lack of stability and availability in their products. Then someone in management took the brave decision: reintroduce the performance competence center. To save a lot of embarrassment, it was now called a performance testing guild, but had a similar function as the old performance competence center: performance testing and monitoring over the chain, carrying a team-overlapping responsibility for the performance of the end product. The stability of the software in production was then slowly improved.
Still, possibly due to the shame of the decision to dissolve the performance competence center, a sense of dissatisfaction with the performance guild remained in the company. Was it really not possible to incorporate these activities into the teams? Even now, after the performance guild has proven to be very useful for their end-to-end view on the chain, the team`s existence is far from assured: time and time again, they not only need to prove their added value to the company, but also explain why their activities cannot be taken over by the DevOps teams.
Yes, this story is an embarrassing one, with only a limited positive outcome, for everyone involved: the performance engineers, management, as well as the organization. How can
we learn from this?
In my talk I gave for the NeotysPAC, I presented 10 reasons why performance competence centers need to change. And how they could adapt. It is about the need to learn new skills. The need to learn to communicate. The challenge to learn to teach. The coolness about being able to automate performance analysis and performance testing. And about working together with the teams. That requires courage and effort; however, it builds the performance engineers of the future. The future you.
As Charles Darwin put it: “It is not the strongest or the most intelligent who will survive but those who can best manage change.”
If you are determined to change your competence center, the one vital step you could take right now is to create a curriculum of courses for the teams. To teach them about what performance risks are. To help them think about their own performance requirements. And to help them automate performance analysis and performance testing in their pipelines.
See the video for tips on these courses, and other best practices that you could undertake in the transformation of the performance competence center, to convince your management that the existence of your competence center does not conflict with Agile and DevOps. That it still offers immense value. And that it still rocks.
0 Comments