Notes
Notes on machine learning, language models, and the practical side of using them.
-
What I check before trusting a benchmark
A leaderboard number is a starting point. A few questions decide whether it means anything for your task.
-
Prompts are interfaces
Treating a prompt like an API contract makes it easier to change without breaking things.
-
Retrieval first, fine-tuning later
When a model needs to know your documents, the simplest approach is usually to look them up.
-
A longer context window is not a memory
Fitting more text into a prompt helps, but it does not mean the model will use all of it well.
-
Evaluating outputs without fooling yourself
Reading ten good answers in a row proves less than it feels like it does.
-
Tokens are not words
Why a model can write a sonnet but stumble when asked to count letters.
-
Small models deserve a second look
Bigger is not always the right default. A few cases where a small model is the better tool.