[FR] Improve Ask SB's resilience to tool call input hallucinations

[komapa]

Describe the bug

Error occurred
Invalid input for tool searchCode: Type validation failed: Value: {"queryRegexp":"Dockerfile|requirements\\.txt|pip install","fileNamesFilterRegexp":"[\"Dockerfile.*\", \"requirements.*\\.txt\", \".*\\.dockerfile\"]","limit":10}. Error message: [ { "code": "invalid_type", "expected": "array", "received": "string", "path": [ "fileNamesFilterRegexp" ], "message": "Expected array, received string" } ]

To reproduce

Just asked How do we install the ddtrace-run package? with claude-sonnet-4. Ran 3 steps and errored on the 4th.

Sourcebot deployment information

Sourcebot version (e.g. v3.0.1): Using :latest docker container tag.

Additional information

No response

[brendan-kellam]

This looks like a hallucination. The LLM inputed "fileNamesFilterRegexp":"[\"Dockerfile.*\", \"requirements.*\\.txt\", \".*\\.dockerfile\"]" (notice that the array is wrapped in "", so it was interpreted as a string).

Unfortunately this is one of the downsides with working with a non deterministic black box 🤠 Jokes aside, I think we can improve our prompts though to minimize instances where this occurs. The sdk we use also has the concept of tool call repair where we could provide type validation error message back to the LLM and have it fix the error.