Why LiteLLM sits in the middle
I got tired of every app speaking a slightly different model API. LiteLLM is the boring piece in the middle. The chat window talks to it. Later, if I add a faster model or a slower one, the apps do not have to care. Today there is still mostly one local model, so the gateway is half lock and half future hook. I installed it before I fully understood why. That turned out to be fine.
RAG is just search with extra steps
I kept hearing “memory”. It is not. I take markdown I already maintain, cut it into chunks, embed those chunks on the same machine, and on a question I pull a few of them back into the prompt. The model then has my notes in front of it. If I forget to re-ingest after an edit, it will confidently quote last week’s version. There is no magic refresh.
The index stays in the lab. Nothing in that path is a cloud API.
The prompt was doing the damage
Giving the model read-only tools helped until it didn’t. Every question shipped the full tool list, even when I asked something tiny. Most of that text was never used. I only noticed because I finally measured tokens instead of guessing.
The router is keywords and string splits. No model classifies the question. That would spend an inference call to save an inference call. Simple questions get a smaller toolbox. Some two-part questions get asked in two goes, same conversation, so “them” still points at the first answer. If it cannot tell, it fails open and offers everything.
The score is mixed, which is useful
On the small model, single questions used about a third fewer prompt tokens and still found the right tool. Two-part questions used more tokens — two calls — but the second half got answered more often. I ran the same rules on a larger model and they made it worse. So this is not a universal upgrade. It is a patch for a model that answers the first half and sits down.
I still need the routing to be per-model, and the knowledge index to follow the docs. Usable. Not finished.