AskData

Fast answers or trusted answers: pick one

After FinSight gave StreamCo one governed source of truth, a new bottleneck appeared: people still needed an analyst to get answers out of it.

• Simple questions ("What was churn in International last quarter?") waited 2–3 days in the analyst queue.
• Analysts spent much of their week on repeat lookups instead of deeper analysis.
• Teams had started pasting questions into generic AI chatbots, which gave confident numbers that didn't match finance.

Key insights (simulated interviews)

• Trust needs proof: users won't trust an answer they can't check, so showing the SQL and assumptions became a core requirement.
• Questions are patterns: most questions combine a metric, a filter, a period and a breakdown, which made a structured approach possible.
• "I don't know" beats a guess: every persona preferred a refusal to a plausible wrong number.

Approaches I compared

Key decision: the LLM never calculates a number. It only translates the question into a structured request. The limitation ("only answers defined metrics") is a feature: it's what makes the answers trustworthy.

Which AI use cases to build first (RICE)

Reach = users per quarter; Impact on a 1–3 scale; Effort in person-months. RICE = Reach × Impact × Confidence ÷ Effort.

PRD summary

• Goal: trusted answers in seconds; numbers match FinSight exactly.
• Launch gate: ≥ 90% held-out accuracy and 100% guardrail pass rate.
• Must-haves: plain-English Q&A, SQL and assumptions on every answer, breakdowns with charts, guardrails.
• AI requirements: strict JSON output, temperature 0, no raw data in the prompt, p95 latency under 3 seconds, under $0.05 per question.
• Out of scope: forecasts, customer-level data, free-form SQL, write actions.

How a question becomes a trusted answer

Prompt design

# System prompt (excerpt)
You are AskData, an analytics assistant for StreamCo.
Map the user's question to the JSON schema below. Do NOT calculate numbers.
- Use only metrics, dimensions and values listed in SEMANTIC_LAYER.
- If no metric fits, return {"type": "unsupported", "term": "..."}.
- If the question asks for individual customers or personal data, return {"type": "refuse_pii"}.
- If the question is about the future or outside 2025, return {"type": "out_of_range"}.
- If the question is too vague, return {"type": "clarify", "options": [...]}.
- Record every default you apply in "assumptions".
Output valid JSON only. Temperature: 0.

Confidence labels: High when every part of the question maps directly; Medium when a default was applied (e.g., no period given) or the wording was broad; Low when nothing maps, which triggers a refusal or a clarifying question.

Metrics defined once, as code

The semantic layer is what makes AskData trustworthy. It holds each metric's SQL, owner, synonyms, type and "good direction", the same definitions FinSight uses.

# semantic_layer.yml (excerpt): metrics are defined once, as code
metrics:
  - name: churn_rate
    label: Monthly churn rate
    owner: Product
    sql: 1.0 * SUM(churned_subs) / SUM(start_subs)
    type: ratio            # aggregate first, then divide
    good_direction: down    # "worst churn" = highest
    synonyms: [churn, attrition, cancellation rate, retention]
  - name: subscribers
    sql: SUM(end_subs)
    type: point_in_time    # Q3 subscribers = September value
    synonyms: [customers, accounts, users, subscriber base]
  - name: variance
    owner: FP&A
    sql: SUM(revenue) - SUM(budget_revenue)
    good_direction: up      # "missed budget most" = most negative
dimensions: [tier, region, month]
valid_values:
  tier: [Basic, Pro, Enterprise]
  region: [West, Central, East, International]
time_range: 2025-01 to 2025-12
unsupported: [nps, cac, ltv, ebitda, marketing_spend]   # refuse, never guess

From question to SQL: real examples from the demo

{
  "metric": "arpu",
  "group_by": "tier",
  "filters": {
    "tier": [
      "Pro",
      "Enterprise"
    ],
    "region": []
  },
  "period": {
    "start": "2025-10",
    "end": "2025-12"
  },
  "rank": null
}

SELECT
    tier,
    1.0 * SUM(revenue) / SUM(end_subs) AS arpu
FROM fct_revenue_monthly
WHERE month_start BETWEEN '2025-10-01' AND '2025-12-01'
  AND tier IN ('Pro', 'Enterprise')
GROUP BY tier
ORDER BY arpu DESC;

{
  "metric": "churn_rate",
  "group_by": "region",
  "filters": {
    "tier": [],
    "region": []
  },
  "period": {
    "start": "2025-01",
    "end": "2025-12"
  },
  "rank": "max"
}

SELECT
    region,
    1.0 * SUM(churned_subs) / SUM(start_subs) AS churn_rate
FROM fct_revenue_monthly
WHERE month_start BETWEEN '2025-01-01' AND '2025-12-01'
GROUP BY region
ORDER BY churn_rate DESC;

{
  "metric": "churned",
  "group_by": null,
  "filters": {
    "tier": [],
    "region": [
      "Central"
    ]
  },
  "period": {
    "start": "2025-04",
    "end": "2025-06"
  },
  "rank": null
}

SELECT
    SUM(churned_subs) AS churned
FROM fct_revenue_monthly
WHERE month_start BETWEEN '2025-04-01' AND '2025-06-01'
  AND region IN ('Central');

{
  "metric": "variance",
  "group_by": "month",
  "filters": {
    "tier": [],
    "region": []
  },
  "period": {
    "start": "2025-01",
    "end": "2025-12"
  },
  "rank": null
}

SELECT
    STRFTIME('%Y-%m', month_start) AS month,
    SUM(revenue) - SUM(budget_revenue) AS variance,
    100.0 * (SUM(revenue) - SUM(budget_revenue)) / SUM(budget_revenue) AS variance_pct
FROM fct_revenue_monthly
WHERE month_start BETWEEN '2025-01-01' AND '2025-12-01'
GROUP BY month
ORDER BY month;

How I know it works

I wrote a golden set of 45 questions across 8 categories. A question is scored correct only if the metric, filters, breakdown, period and ranking direction all match. After tuning, I wrote 15 new held-out questions and tested once, without changing the parser, to get an honest accuracy number.

What the evaluation caught

Limitation: I wrote the golden set myself. In production, it would be built from real user questions and thumbs-down feedback, which is why the demo includes a feedback button.

Business case (illustrative)

Base case: 150 users, 6 questions each per month, 60% coverage, 70% adoption. That's about 3,400 analyst hours and $238K saved a year against $42K in run costs, a $196K net annual benefit and 80% year-1 ROI after a $90K build.

• v3 parser: fix the open held-out failure and grow the golden set to 200+ questions from real usage.
• Model cost optimization: test a smaller model once the golden set is large enough to compare fairly.
• Add governed CAC and NPS, the most-requested unsupported metrics.
• Anomaly alerts: proactive "something changed" messages, with precision targets before launch.
• Route low-confidence answers to an analyst automatically.

Next project: RetailPulse → (finance and retail analytics)