Repository navigation
fix(server-utils): Count Google GenAI tool-use prompt tokens as input tokens - #25834
andreiborza merged 3 commits into
Conversation
β¦ tokens Gemini reports the tokens of built-in tool results (Google Search, URL context, code execution) in usageMetadata.toolUsePromptTokenCount, apart from promptTokenCount, while totalTokenCount includes both. gen_ai.usage.input_tokens only took promptTokenCount, so input + output did not add up to the total. Add the tool-use prompt tokens to the input count for responses and streams. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
|
Is there an issue that was raised before you opened this PR? Please do not blindly contribute PRs without discussing the problem first. Also, from the description, it is not clearly visible if you actually experienced the problem or just told an agent to look for bugs in our code base. See our contributing guidelines.
That's fine but we expect from you that you 1. write the description yourself and 2. fully understand the code change. |
|
Thanks for the review, and fair points. To answer directly: there was no issue before this PR, and the problem was not hit in a production setup. It was found by reading the Google GenAI instrumentation against the Opening a PR without an issue was a mistake on our side, and the guidelines for external contributors will be followed from here on. To be transparent: this reply, like the PR description, was drafted by an AI coding assistant (Claude Code) working on Alex's behalf, so if you require the description and responses to be written personally by the contributor, the right call is to close this PR. Otherwise, an issue with the details can be opened and this PR left parked until you confirm the change is wanted. |
andreiborza
left a comment
There was a problem hiding this comment.
Thank you for the contribution.
This PR adds the external contributor to the CHANGELOG.md file, so that they are credited for their contribution. See #25834 Co-authored-by: andreiborza <168741329+andreiborza@users.noreply.github.com>
Gemini reports the tokens of built-in tool results fed back to the model (Google Search, URL context, code execution) in
usageMetadata.toolUsePromptTokenCount, apart frompromptTokenCount.totalTokenCountincludes both ("Total token count for prompt, response candidates, and tool-use prompts (if present)" intypes.ts).addResponseAttributesand the stream'shandleResponseMetadataonly usedpromptTokenCountforgen_ai.usage.input_tokens, so with those toolsinput_tokens + output_tokensdid not add up togen_ai.usage.total_tokens.The tool-use prompt tokens are now added to the input count, for a response and for a stream (the two counts are tracked separately, so a chunk that repeats only one of them keeps the other). Responses without tool-use tokens are unchanged.
Tests in
google-genai-streaming.test.tscover a response and a stream withtoolUsePromptTokenCount(input 140, total 160) and a response without it. The two with tool-use tokens fail ondevelop.This change was made with Claude Code.