Collecting the Right Data Without Drowning in Metrics

You can measure everything. That's the problem.

Modern infrastructure makes it trivially easy to collect metrics on every button click, every API call, every page view, every server heartbeat. So teams collect everything, build dashboards with 40 charts, and then make decisions based on gut feeling anyway — because when everything is measured, nothing stands out.

The hard part of data collection isn't the collection. It's the curation. Deciding what matters before you start measuring is the difference between data that drives decisions and data that just fills dashboards.

Start With the Decision, Not the Data

Most teams approach metrics backward. They ask "what can we measure?" and then build dashboards around whatever's available. This produces comprehensive dashboards that answer every question except the ones that matter.

Reverse it. Start with the decisions you actually make, then figure out what data would make those decisions better.

"Should we invest more in onboarding?" → Measure new user activation rate, time to first value, drop-off points in the onboarding flow.

"Is this feature worth maintaining?" → Measure feature adoption rate, usage frequency among active users, support ticket volume related to the feature.

"Are we shipping fast enough?" → Measure deployment frequency, lead time from commit to production, change failure rate.

Each of these is a specific question with a specific set of metrics that answer it. Notice what's missing: pageviews, total registered users, average session duration. Those metrics feel informative but don't drive any specific decision. They're vanity metrics — numbers that go up over time and make you feel good without telling you what to do.

The Metric Hierarchy

Organize your metrics in layers so you know which ones to check daily, which to review weekly, and which to investigate only when something's wrong.

North Star Metric (1 metric)

The single number that best represents the value your product delivers to users. For a project management tool, it might be "tasks completed per active user per week." For a content platform, it might be "articles read to completion." For a SaaS product, it might be "weekly active teams."

This isn't a revenue metric — it's a value metric. Revenue follows value, but measuring revenue directly doesn't tell you how to create more of it.

Health Metrics (3-5 metrics)

The vital signs of your product. If any of these change significantly, something important happened and you need to understand why.

For most software products:

• Retention rate — Are users coming back?

• Activation rate — Are new users reaching the moment where they get value?

• Performance — Is the application responsive? (P95 latency, error rate)

• Revenue — Is the business sustainable? (MRR, churn rate)

These get a weekly review. You're looking for trends and anomalies, not daily fluctuations.

Diagnostic Metrics (many metrics)

Everything else — detailed breakdowns, feature-level analytics, funnel steps, technical metrics. These exist to answer "why" when a health metric changes. You don't review them regularly. You investigate them when something in the health layer triggers a question.

If your activation rate drops, you look at the diagnostic metrics: which onboarding step saw increased drop-off? Was it a specific user segment? Did it correlate with a deployment?

Diagnostic metrics are valuable precisely because you don't look at them all the time. They're the detail you drill into when the higher-level signal says something changed.

What Not to Measure

Just because you can measure something doesn't mean you should. Every metric you collect has costs: storage, cognitive load, dashboard clutter, and the risk that someone makes a decision based on a misleading number.

Don't measure what you won't act on. If you'd see the number and shrug, don't track it. Metrics exist to drive decisions. If no decision depends on a metric, it's noise.

Don't count things when you should measure outcomes. "Number of deployments" tells you activity happened. "Change failure rate" tells you whether the activity produced good results. The second is actionable. The first just measures busyness.

Don't track cumulative totals. "Total registered users" only goes up. It tells you nothing about current health. "Weekly active users" can go up or down, and the direction tells you something real.

Don't measure proxy metrics when you can measure the real thing. "Time on page" is a proxy for engagement. It could mean the user is reading carefully, or it could mean they left the tab open. If you can measure actual engagement (scroll depth, interaction events, completion rate), measure that instead.

Instrumentation Principles

When you do decide to measure something, instrument it well.

Name Things for Future You

event_click_btn_3 means nothing in six months. checkout_payment_method_selected is self-documenting. Name events in a consistent format — noun_verb or context_action — and document what each event means, what properties it includes, and when it fires.

Capture Context, Not Just Events

An event that says "user clicked buy" is less useful than an event that says "user clicked buy on the pricing page, for the annual plan, on mobile, during their first session, 3 minutes after signing up." Context turns events into stories.

Standard context to attach to every event: user ID (or anonymous ID), timestamp, device type, page/screen, session ID, and any relevant feature flags.

Validate Your Data

Broken instrumentation is worse than no instrumentation. A tracking event that fires twice per click, or doesn't fire at all on mobile, or captures the wrong value in a property — these create false signals that are worse than having no data.

Write tests for your critical tracking events. Validate the data pipeline end-to-end. If your analytics say something surprising, verify the instrumentation before you act on it.

The Dashboard That Gets Used

Most dashboards fail because they show too much. A dashboard with 30 charts requires 30 minutes to read and yields no clear takeaway.

The dashboard that actually gets used:

• Has fewer than 10 charts on the main view

• Shows the North Star and health metrics prominently

• Uses clear time comparisons (this week vs. last week, not just current values)

• Highlights anomalies rather than requiring you to spot them

• Links to deeper views for diagnostic investigation

The goal isn't comprehensive visibility. It's at-a-glance understanding. If you can't interpret the dashboard in under 60 seconds, it has too much on it.

Key Takeaway

Start with the decisions you need to make, not the data you can collect. Organize metrics into a North Star, health metrics, and diagnostic metrics. Don't measure what you won't act on. Name events clearly, capture context, and validate your instrumentation. Build dashboards that get used, not dashboards that impress. Less data, better curated, is worth more than all the data, poorly organized.

---

Next in the Data-Driven Decisions path: Experimentation for small teams — A/B testing and validated learning when you don't have millions of users.