Summary

This PR adds comprehensive examples for counting tokens with structured responses to the existing "How to count tokens with tiktoken" notebook, addressing issue #2102. The addition includes a new function for calculating token overhead from structured response schemas, practical examples comparing different response formats, and API verification to ensure accuracy.

Motivation

The existing notebook provided excellent coverage for counting tokens in regular messages and tool calls, but lacked examples for structured responses - a key feature that adds token overhead due to schema definitions and validation instructions. This gap left developers without guidance on how to estimate token consumption when using structured outputs, which is increasingly important as more applications adopt structured response formats for better data consistency and parsing.

The new examples fill this gap by:

• Providing a dedicated function to calculate structured response token overhead
• Demonstrating both simple JSON object format and complex structured schemas
• Showing how schema complexity directly impacts token consumption
• Including API verification to validate the token counting estimates
• Following the same educational pattern as existing sections for consistency

This addition makes the cookbook more complete and helps developers make informed decisions about when and how to use structured responses based on their token budget considerations.

For new content

When contributing new content, read through our contribution guidelines, and mark the following action items as completed:

• I have added a new entry in registry.yaml (and, optionally, in authors.yaml) so that my content renders on the cookbook website.
• I have conducted a self-review of my content based on the contribution guidelines:
  • Relevance: This content is related to building with OpenAI technologies and is useful to others.
  • Uniqueness: I have searched for related examples in the OpenAI Cookbook, and verified that my content offers new insights or unique information compared to existing documentation.
  • Spelling and Grammar: I have checked for spelling or grammatical mistakes.
  • Clarity: I have done a final read-through and verified that my submission is well-organized and easy to understand.
  • Correctness: The information I include is correct and all of my code executes successfully.
  • Completeness: I have explained everything fully, including all necessary references and citations.

We will rate each of these areas on a scale from 1 to 4, and will only accept contributions that score 3 or higher on all areas. Refer to our contribution guidelines for more details.