We're usually expensive, so this is a fair question :) .
I've addressed this elsewhere, where I've humbly submitted the following:
I measure how we are doing as Agile Coaches, in part, by focusing on the conducting of product experiments by Product Owners, and process experiments by Scrum Masters.
Later, I added specificity to the thought as it applied to Scrum Masters, proposing:
“new process experiments over time” as the meta-metric uniquely aligned with the Scrum Master role, which can be a leading indicator for the other metrics
If the above does not jive with you, that's great. Seriously. This means you have an idea in your head of what an Agile Coach should be doing, and how their presence should be adding value to your organization. Let's hold on to your idea.
Because Agile Coaching is still a relatively new profession, there is some variability as to what this person does.
I've been the flavour of Agile Coach where I've interacted concurrently with 4 teams as effectively their Scrum Master.
I've been the flavour of Agile Coach where I've interacted concurrently with 2 teams as a Scrum Master, while also evolving those teams from sticking to the Scrum Guide to a more Kanban way of life, or Mob Programming.
I've been the flavour of Agile Coach where I've interacted concurrently with 6 teams primarily through advising their 6 Product Owners and mentoring their 6 Scrum Masters.
Because Agile is still a relatively new way of working, albeit less new in software development circles, this usually means there is a way of working that is already in place... the 'incumbent' model: a way of working more akin to the "Command & Control" feel of traditional project management. Yet you're looking for a change, so you're interested in the Agile, and we're back to your idea of what an Agile Coach does.
Want to do Agile? Read some stuff online. Watch a few videos. See what competitors are doing. Get teams to start doing stand-ups every morning. Maybe get 'em to take some training. Maybe even hire an Agile Coach. Or a few. Better yet, have all this be part of an Agile Transformation. That'll show you're serious. You're transforming. These are very outwardly visible steps, and as you make these organizational changes, you might even feel good about yourself.
What to be Agile? First. Ask. Yourself. Why. Yes, what's the hope for your organization's new way of working, but also what's the pain you are addressing? What's the urgency?
If you are not bought in to the problem you are trying to solve, then you are not bought in to bringing in Agility.
Going Agile is change management. Put the change process aside; I'm talking about change stickiness. The change ain't gonna stick if people do not see the utility in the change. Agility ain't gonna stick if people do not see that going through the trouble will actually make life better.
Better for whom?
And better than what, exactly?
What aspects of that previous life are now better?
If you're reading this, you've likely gone through, or know of, an Agile Transformation where not everybody involved was included equally, and not everybody was bought in to the problem you are trying to solve, let alone how to solve it.
An Agile Coach can help you with your Agility goals. If those goals are not clear, then your idea of what an Agile Coach does is by default not aligned.
First ask: What clear problem are we trying to solve with Agility? Once this is clear, goals for Agility and their success metrics can be tackled next. Now you can judge if Agility is effective. Now you can ask if your Agile Coach is effective.
Running a company is expensive, so this is a fair question :) .
Oh, and if you're a Scrum Master / Agile Coach, ask your boss / Office of the Merry Band of Agile Coaches what organizational problem Agility is trying to solve. And how improvements are being measured. If there are no Agility goals or metrics, then make some up and share 'em loudly - try to get buy-in.
If you don't already have these, others (the leaders) already do, at least informally in the back of their heads.
We will be faster. (We will have more releases, and more frequently.)
We will be better. (We will get fewer complains of bugs from our customers.)
We will be cheaper. (We will have less waste as we get stuff done.)
Be proactive here. Then you have something to work with when you answer for yourself if you are effective.
Again, we're usually expensive, so this is a fair question :) .