[Gen AI] Since version 26.3.0 Tock doesn't work with Small or Medium llm models #2090

The core issue is that, since the "breaking change" to the RAG configuration and the new prompt system that forces the LLM to respond in JSON, it’s now impossible to run the RAG with a model that can operate locally via Ollama on a developer machine (even if it a MAC with enough memory). The workload requested by the prompt is too heavy to yield an LLM response within a reasonable timeframe, even with a very small model. I’ve already set the timeouts to 2 minutes...

The prompt requires the AI to perform numerous checks. For example, when 10 chunks are retrieved for a given question, it’s asked to generate 10 JSON objects and explain why each was unused... doing this for a straightforward question with small chunks feels quite disproportionate to me.

To improve this, we might need to reduce the overhead of context_usage in the responses. For instance, adding an option to return only the used chunks in a simpler format (when enabled):
"context_usage": ["123", "456"]

The other parameters appear to only serve debugging purposes for the responses provided to end users.

We could leverage the ability to pass a json_schema during LLM calls (e.g., with_structured_output in LangChain) to enforce the target JSON format directly. This would eliminate the need to handle it within the prompt and shoud reduce response time . Additionally, since the code validates this schema (which remains consistent across all calls), it would prevent errors and avoid poorly defined JSON structures in the prompt.