[Fix][Generator] Use custom_chat_template in each step retokenization#233
Conversation
CharlieFRuan
changed the title
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request correctly applies the custom_chat_template in the re-tokenization step for consistency. While the change is good in isolation, it highlights a critical bug in the agent_loop logic. The initial prompt length is calculated using the default chat template, but the token IDs from which the prompt is later extracted are generated using the custom chat template. This mismatch will lead to incorrect prompt data. I've added a critical review comment with a detailed explanation of the issue and a suggested fix.
|
/gemini review |
There was a problem hiding this comment.
Code Review
This pull request addresses an inconsistency by ensuring custom_chat_template is used during the re-tokenization of chat history. The changes in agent_loop and _get_next_input_ids_by_retokenizing_chat_history correctly apply the custom template when needed. The fix is sound and improves code consistency. I have provided one suggestion to enhance the code style in agent_loop.
…#233) Prior to this PR, in the re-tokenize chat history codepath, we did not use `self.custom_chat_template` in some places (only used it in the final tokenization). Though currently only Qwen3 enters this codepath, which would not be affected, it is still good to apply the `custom_chat_template` for consistency.
2 participants
Prior to this PR, in the re-tokenize chat history codepath, we did not use
self.custom_chat_templatein some places (only used it in the final tokenization). Though currently only Qwen3 enters this codepath, which would not be affected, it is still good to apply thecustom_chat_templatefor consistency.