Summary

Sarah Friar (OpenAI CFO) proposes a practical AI scorecard to measure real ROI by tracking outcomes engineers can act on: useful work produced, cost per successful task, dependability, and return on compute. The approach focuses on business-aligned success definitions, instrumenting pipelines for observable success signals, and computing per-task economics so teams can optimize models, infra, and workflows.

Key Points

• Define "useful work" as the fraction of model outputs that meet a clear success criterion (business or product metric).
• Quantify cost per successful task = total AI spend / number_of_successful_tasks and break down by model, endpoint, and feature.
• Measure dependability via SLIs: success rate, error rate, latency p50/p90/p99, and degradation windows; set SLOs and alert thresholds.
• Compute return on compute as value generated (revenue or savings) divided by compute units (GPU-hours, FLOPs) to compare models and optimizations.
• Instrument requests end-to-end: tag requests, capture outcome labels, attribute costs, and store metrics in time-series stores for dashboards and A/B analysis.
• Use A/B tests and canary rollouts to validate that model/infra changes improve scorecard metrics before wide release.

Recommended next steps for engineering teams

• Agree on success labels and sampling strategy for manual/automated labeling.
• Add cost attribution to inference logs and aggregate cost per successful task daily.
• Create dashboards for the four scorecard metrics and add SLO-based alerts.
• Run experiments to optimize for the highest return on compute (model size, batching, quantization, routing).

This scorecard turns abstract AI value into measurable engineering levers so teams can prioritize changes that improve real-world ROI.