AI Inference Metrics

Time to First Token (TTFT) How long it takes before the AI starts responding — the "thinking" delay before the first word appears. Lower is better. High TTFT feels like the AI is "hanging" even if the full response arrives quickly.

Invocations How many AI requests were made in each time bucket. Use this to see usage patterns, spot traffic spikes, or understand which model is getting the most calls.

Token Throughput The volume of text processed by the AI — input tokens are the prompts and context you send, output tokens are the AI's responses. Tokens ≈ words (1 token ≈ ¾ of a word). Higher output volume = more AI-generated content delivered.

Prompt Cache Bedrock can cache repeated context (system prompts, docs) so you don't re-process it on every call. Cache reads are cheap — you're reusing saved work. Cache writes are when new context gets saved. High read-to-write ratio means strong savings.

Errors Client errors (4xx) are usually bad requests from your app — malformed input, invalid parameters, or rate limit hits. Server errors (5xx) are problems on AWS's side. Either type degrades AI availability for your users.

Quota Utilization How much of your token-per-minute (TPM) quota you're using. 100% means you're at the limit — sustained highs here lead to throttling and failed requests. Each model has its own quota, managed via AWS Service Quotas.

Estimated Cost Approximate spend per time bucket based on token counts and Bedrock on-demand pricing. These are estimates — actual billing may differ due to batch discounts, reserved capacity, or pricing changes.