Understanding the Hidden Costs of AI: A Shift in Measurement
In recent articles by The New York Times and The Wall Street Journal, a pressing issue has emerged within companies: the unexpected costs driven by AI agents generating tokens. Here’s a breakdown of the problem:
- Tokens vs. Compute: Traditionally, tokens represented usage, but this model is breaking down as AI becomes more complex.
- Execution Depth: Agentic AI can trigger numerous steps (e.g., planning, retrieval, tool use, and validation) in a single request, leading to increased compute that isn’t reflected in token counts.
- Cost Control Issues: Companies can monitor token usage, but without proper metrics for compute, they struggle with accurate cost control.
This misalignment is not just an accounting oversight; it signifies a crucial shift in how we must approach AI pricing and budgeting.
Let’s discuss how your organization manages AI costs! Share your thoughts and insights below.
