Instrument the channel that pays.

We had daily reports on a channel that produced almost nothing, and no report at all on the channel carrying thousands of users. Nobody chose that arrangement. It happened because one channel came with a dashboard and the other required building one — and effort follows instrumentation, wherever the customers happen to be.

How the inversion happens

Analytics for your own website is a solved problem: sign up, paste a snippet, and charts arrive forever. Analytics for a marketplace catalog, a platform listing, or a distributor's storefront means reading someone else's API, reconciling their definitions, and building the report yourself. So the website gets measured on day one and the platform channel gets measured someday.

Then the feedback loop does the rest. Measured channels produce numbers; numbers produce tasks; tasks produce more numbers. A quarter later the team is expert in the channel it can see — which pages, which queries, which tweaks — while the channel where the users actually are runs unexamined. The dashboard did not lie. It just answered so promptly that nobody asked whether it was the right question.

The tell is a ratio nobody computes

The diagnostic is one division: users or revenue per channel, over the same window, side by side. It is rarely computed because the numbers live in different systems with different owners, and one of them may not exist until someone builds it. When we finally ran it over the same 90 days, the instrumented channel showed 2 clicks and the uninstrumented one showed 3,947 distinct users — three orders of magnitude, pointing the wrong way relative to where the effort had gone.

A ratio like that does not call for rebalancing. It calls for admitting that the measured channel had been standing in for the business because it was legible, the way the drunk searches under the streetlight because the light is better there.

What the paying channel wanted instead

The uncomfortable part is that the productive channel had been asking for work the whole time — visible only in its own numbers. Its analog of a bounce rate is a failed run; its analog of a ranking factor is a success percentage; its analog of a bad landing page is a product whose users arrive, try, and quietly leave a two-star review. Every one of those is actionable engineering, none of it appears in website analytics, and all of it was waiting behind an API we had simply never read.

One afternoon of building the missing report — 154 products, one API call each — produced a sharper work queue than a month of the instrumented channel's dashboards: which products strangers actually use, where their 59,000 monthly runs fail, which of the 147 ratings diverge from reliability. That last cut is the most valuable signal in the set, since a well-rated crash and a poorly-rated success point to opposite defects.

The rule we took from it

Instrument channels in order of where money moves, not in order of how easy they are to instrument — and treat any channel that lacks a report as unexamined rather than unimportant. The absence of a dashboard is itself a finding. Ours was hiding the majority of the business behind one API call we had never made, and the only real cost of making it was admitting the charts we already had were answering a smaller question than we thought.

← All notes