LLM prompting vs RAG vs fine‑tuning — weights or input?

How do you choose between prompting, RAG, and fine-tuning?

If the model’s weights need to change, fine-tune; if not, change the input. Of the ways to change the input, fetching outside documents and inserting them is RAG, and everything else is prompting. Why that rule works becomes clear once you see what training and inference each change.

I felt I had to understand the internals when I hit a concrete question. Build anything with AI and you reach the point of choosing: will a prompt do, do I need RAG, or do I have to fine-tune? I could choose by feel, but I couldn’t say why.

Tokens are compressed pieces of the corpus

It starts with a corpus: picture a huge pile of text from blogs, websites, social media, and code repositories, all joined into one long line. To a computer, this text is numbers from the start.

package main

import "fmt"

// To a computer, text is numbers from the start. A Korean syllable is 3 bytes in UTF-8.
func main() {
  for _, s := range []string{"점", "점심", "lunch"} {
    b := []byte(s)
    fmt.Printf("%-6q %d bytes → %v\n", s, len(b), b)
  }
}

$ go run ./utf8
"점"    3 bytes → [236 160 144]
"점심"   6 bytes → [236 160 144 236 139 172]
"lunch" 5 bytes → [108 117 110 99 104]

In UTF-8 a Korean syllable is three numbers between 0 and 255, and an English letter is one. The two-syllable Korean word for lunch is longer than the five-letter “lunch”.

Used as-is, these number sequences are too long, so frequent combinations get merged into a vocabulary. BPE (Byte Pair Encoding), which many LLMs use, repeatedly merges the pair that appears together most often, a fixed number of times. The resulting pieces are tokens, and the list of pieces is each model’s own vocabulary.

package main

import (
  "fmt"
  "strings"
)

// Split words into letters, then merge the pair that appears together most often.
func main() {
  corpus := map[string]int{"low": 5, "lower": 2, "newest": 6, "widest": 3}
  words := map[string][]string{}
  for w := range corpus {
    words[w] = strings.Split(w, "")
  }

  for step := 1; step <= 4; step++ {
    pairs := map[string]int{}
    for w, toks := range words {
      for i := 0; i+1 < len(toks); i++ {
        pairs[toks[i]+" "+toks[i+1]] += corpus[w]
      }
    }
    best, cnt := "", 0
    for p, c := range pairs {
      if c > cnt || (c == cnt && p < best) {
        best, cnt = p, c
      }
    }
    a, b, _ := strings.Cut(best, " ")
    for w, toks := range words {
      var out []string
      for i := 0; i < len(toks); i++ {
        if i+1 < len(toks) && toks[i] == a && toks[i+1] == b {
          out = append(out, a+b)
          i++
        } else {
          out = append(out, toks[i])
        }
      }
      words[w] = out
    }
    fmt.Printf("merge %d: %q + %q (%d times) → newest = %v\n", step, a, b, cnt, words["newest"])
  }
}

$ go run ./bpe
merge 1: "e" + "s" (9 times) → newest = [n e w es t]
merge 2: "es" + "t" (9 times) → newest = [n e w est]
merge 3: "l" + "o" (7 times) → newest = [n e w est]
merge 4: "lo" + "w" (7 times) → newest = [n e w est]

Two things follow. Tokens don’t map one-to-one to words: a three-word phrase might be five tokens or ten. And since every model has its own vocabulary, the same sentence has a different token count per model. Paste a sentence into a tool like tiktokenizer and you can watch the split points and counts change as you switch models. API pricing and context length are both counted in these units.

Training raises the probability of the right token

The model’s output is a probability distribution

The goal of training fits in one sentence: build a machine that, given a token sequence, predicts the next token. The model gives every token in the vocabulary a score (a logit), and softmax turns those scores into probabilities that sum to 1. Let’s check with five candidates after “The weather today is”.

package main

import (
  "fmt"
  "math"
)

// Turn the model’s final output (logits) into probabilities.
func softmax(logits []float64) []float64 {
  maxv := logits[0]
  for _, v := range logits {
    maxv = math.Max(maxv, v)
  }
  sum := 0.0
  probs := make([]float64, len(logits))
  for i, v := range logits {
    probs[i] = math.Exp(v - maxv) // avoid overflow
    sum += probs[i]
  }
  for i := range probs {
    probs[i] /= sum
  }
  return probs
}

func main() {
  vocab := []string{"nice", "clear", "cloudy", "cold", "cat"}
  logits := []float64{3.2, 2.9, 1.5, 1.1, -2.0}
  for i, p := range softmax(logits) {
    fmt.Printf("%-7s logit %5.1f → %5.1f%%\n", vocab[i], logits[i], p*100)
  }
}

$ go run ./softmax
nice    logit   3.2 →  48.7%
clear   logit   2.9 →  36.1%
cloudy  logit   1.5 →   8.9%
cold    logit   1.1 →   6.0%
cat     logit  -2.0 →   0.3%

Logits aren’t probabilities. They can be negative, and they don’t sum to 1. Softmax uses the exponential function to turn them into probabilities and, in doing so, widens the gaps: “cat”, 5.2 logits behind the top, all but disappears at 0.3%. What the model produces is this whole table, not the single word “nice”, and the sampling step later decides what actually gets used.

Pre-training

Cut a token sequence from the corpus, hide the last token, and have the model guess it. The answer is already in the text, so nobody has to label anything, and moving the cut point gives an essentially endless supply of examples. At first nothing has been learned, so the ranking is a mess. The score is the probability given to the correct token, and the cross-entropy loss is -ln of that probability.

package main

import (
  "fmt"
  "math"
)

// The probability p given to the right token is the score. loss = -ln(p)
func main() {
  for _, p := range []float64{0.9, 0.5, 0.1, 0.01} {
    fmt.Printf("P(correct) %4.0f%% → loss %.3f\n", p*100, -math.Log(p))
  }
}

$ go run ./loss
P(correct)   90% → loss 0.105
P(correct)   50% → loss 0.693
P(correct)   10% → loss 2.303
P(correct)    1% → loss 4.605

At 90% the loss is 0.105; at 1% it’s 4.605. The lower the probability on the right answer, the more steeply the loss climbs.

The model traces the error backward (backpropagation) and nudges the weights. Think of the knobs on a DJ mixer: the settings decide the sound, and every miss means turning the knobs a little. The knobs are the parameters, and each knob’s value is a weight. The 7B or 70B in a model’s name means 7 billion or 70 billion knobs.

package main

import (
  "fmt"
  "math"
)

func softmax(z []float64) []float64 {
  out, sum := make([]float64, len(z)), 0.0
  for i, v := range z {
    out[i] = math.Exp(v)
    sum += out[i]
  }
  for i := range out {
    out[i] /= sum
  }
  return out
}

// The answer is "clear" (index 1). Treat the logits as weights and push them by gradient descent.
// The gradient of softmax + cross-entropy is (predicted probs - one-hot answer).
func main() {
  logits := []float64{3.2, 2.9, 1.5, 1.1, -2.0}
  answer, lr := 1, 0.5
  for step := 0; step <= 8; step++ {
    p := softmax(logits)
    if step%2 == 0 {
      fmt.Printf("step %d: P(clear)=%.3f loss=%.3f\n", step, p[answer], -math.Log(p[answer]))
    }
    for i := range logits {
      grad := p[i]
      if i == answer {
        grad -= 1
      }
      logits[i] -= lr * grad
    }
  }
}

$ go run ./train
step 0: P(clear)=0.361 loss=1.019
step 2: P(clear)=0.584 loss=0.538
step 4: P(clear)=0.713 loss=0.338
step 6: P(clear)=0.786 loss=0.240
step 8: P(clear)=0.831 loss=0.185

Running this over hundreds of billions of tokens and more, with enormous amounts of GPU time and power, is pre-training, and the result is a base model. The weights only change during training like this; inference never touches a single knob.

Post-training

A base model can continue text naturally, but it doesn’t behave appropriately for the situation. You need more training so that a coding question gets a coding answer and a report request gets a report. All of that is called post-training, and fine-tuning is part of it.

Fine-tuning itself comes in two kinds. You can retrain all the weights, or leave them untouched and train only a small set of added parameters (adapter methods such as LoRA). The second is far cheaper and widely used. Keep in mind that even a few hundred thousand fine-tuning examples are tiny next to pre-training. It’s a tool for adjusting behavior and format more than for loading in large amounts of knowledge.

Few organizations can pre-train from scratch, so why are there so many models? Because post-training is a matter of recipe. Which data you mix, in what order and proportion, becomes the model’s personality, which is also why each model’s strengths on benchmarks are so uneven.

Inference appends one token at a time until a stop token

Now we use the finished model. Inference aims for the same thing training did, the one token that follows a sequence. The differences are that there’s no answer key and the weights are fixed.

When does it stop? The model doesn’t decide to stop on its own. Stopping is a token too: when a special token like <eos> (a stop token) is sampled, the loop ends. Teaching the model to emit it at the right time is also part of post-training. Let’s check the shape of the loop with a toy model.

package main

import (
  "fmt"
  "strings"
)

// A toy "model": looks at the last token and returns next-token candidates with probabilities.
// A real model looks at the whole context, but the loop has the same shape.
var model = map[string]map[string]float64{
  "is":    {"sushi": 0.46, "ramen": 0.31, "what": 0.23},
  "sushi": {"today": 0.72, "and": 0.28},
  "today": {".": 0.9, "!": 0.1},
  ".":     {"<eos>": 0.95, "And": 0.05},
}

func next(last string) string {
  best, bp := "<eos>", 0.0
  for tok, p := range model[last] {
    if p > bp {
      best, bp = tok, p // greedy: always take the top one
    }
  }
  return best
}

func main() {
  ctx := []string{"Our", "lunch", "menu", "is"}
  for step := 1; ; step++ {
    tok := next(ctx[len(ctx)-1])
    fmt.Printf("step %d: %-8s → %s\n", step, ctx[len(ctx)-1], tok)
    if tok == "<eos>" { // stopping is a token too
      break
    }
    ctx = append(ctx, tok)
  }
  fmt.Println("result:", strings.Join(ctx, " "))
}

$ go run ./loop
step 1: is       → sushi
step 2: sushi    → today
step 3: today    → .
step 4: .        → <eos>
result: Our lunch menu is sushi today .

The loop ended when <eos> was picked at step 4. This toy looks only at the last token, but a real model looks at the whole context.

And it’s not only what you typed in the chat box. An invisible system prompt, the conversation so far, and various settings are all prepended into one long token sequence.

The transformer

Token IDs first become embedding vectors, and position information goes in.¹ Then they pass through N layers with the same structure. The idea is to do the same work several times to get a better next token.

Each layer has two parts. Self-attention is where tokens measure their relationships and exchange information; the MLP (feed-forward) then processes each token on its own. Each layer hands its result, the hidden state, to the next layer, which repeats the same work. The vector at the last position of the final layer becomes logits over the whole vocabulary, which gives the probability table we saw earlier.

Here’s a common misconception.

The hidden state flowing forward is backpropagation run in reverse.

Backpropagation goes backward to find the cause of an error and fix the weights. The forward pass at inference goes forward to repeat a computation with fixed weights. The number of layers, the number of attention heads, and the maximum context length are all set by the model’s designers before training.

Attention measures the relationship to every earlier token

Before transformers, language models (the RNN family) read a sentence from the start and carried their memory forward, so the longer the text, the more they forgot. In “Our lunch menu is fried chicken. … (ten sentences later) Anyway, it was delicious,” they’d lose track of what “it” refers to. Attention was first used in 2014 machine translation research to ease this, and Google’s 2017 paper “Attention Is All You Need” dropped recurrence and built the transformer from attention alone. Every token measures its relationship to every earlier token directly.

Each token produces three vectors: what it’s looking for (Query), a label for what it holds (Key), and the content it passes along (Value). The dot product of a Query with earlier tokens’ Keys gives scores, softmax turns them into weights, and the weights mix the Values. Here’s what it looks like up to the weights.

package main

import (
  "fmt"
  "math"
)

// Toy attention: 4 tokens, 2 dimensions. Future tokens are masked.
func main() {
  tokens := []string{"cat", "saw", "fish", "it"}
  Q := [][]float64{{1, 0}, {0.5, 0.5}, {0, 1}, {0.2, 1.4}}
  K := [][]float64{{0.3, 1.2}, {0.8, 0.2}, {0.1, 1.5}, {0.4, 0.4}}
  d := float64(len(Q[0]))

  fmt.Printf("%-8s", "")
  for _, t := range tokens {
    fmt.Printf("%9s", t)
  }
  fmt.Println()
  for i := range tokens {
    scores := make([]float64, len(tokens))
    sum := 0.0
    for j := range tokens {
      if j > i { // causal mask: tokens that haven’t arrived yet
        continue
      }
      s := (Q[i][0]*K[j][0] + Q[i][1]*K[j][1]) / math.Sqrt(d)
      scores[j] = math.Exp(s)
      sum += scores[j]
    }
    fmt.Printf("%-8s", tokens[i])
    for j := range tokens {
      if j > i {
        fmt.Printf("%9s", "·")
      } else {
        fmt.Printf("%9.2f", scores[j]/sum)
      }
    }
    fmt.Println()
  }
}

$ go run ./attention
              cat      saw     fish       it
cat          1.00        ·        ·        ·
saw          0.54     0.46        ·        ·
fish         0.37     0.18     0.45        ·
it           0.32     0.13     0.41     0.15

Above the diagonal are tokens that haven’t arrived yet, hidden by a causal mask. In this toy I set Q and K by hand so that “it” puts its largest weight on “fish”. In a real model these values come from training, and several attention heads each look at different relationships. With a loop over heads inside the loop over layers, the computation multiplies.

This also hints at why a long conversation can feel like it loses track of earlier points. Attention weights sum to 1, so as the context grows, each token’s share can get thinner. There are measurements showing models make poor use of information in the middle of long inputs (“Lost in the Middle”). Still, the sum-to-1 property alone doesn’t fully explain that effect.

Prefill and the KV cache

A large part of the wait before the first character appears is prefill: the whole input, system prompt and conversation history included, passes through at once, computing attention between all of its tokens. The “thinking” indicator of a reasoning model is a different thing. It’s mostly time spent generating reasoning tokens you don’t see, and those are billed as output tokens.

Why is everything after the first token fast? By the loop above, adding each token would mean recomputing the whole, longer context. But the Keys and Values of past tokens don’t change when a new token arrives, because the causal mask keeps past tokens from seeing future ones. Compute them once, store them, and compute only the new token’s; that store is the KV cache.

Let’s grow the prompt to 1,000 tokens and count K·V computations.

package main

import "fmt"

// Generating N tokens after a P-token prompt: how many per-token K·V projections do we compute?
func main() {
  P := 1000
  for _, N := range []int{10, 100, 500} {
    noCache, withCache := 0, 0
    for step := 0; step < N; step++ {
      ctx := P + step
      noCache += ctx // rebuild K and V for the whole context every time
      if step == 0 {
        withCache += ctx // everything once in the first step (prefill)
      } else {
        withCache += 1 // after that, only the new token
      }
    }
    fmt.Printf("generate %3d: no cache %7d / with cache %5d\n", N, noCache, withCache)
  }
}

$ go run ./kvcache
generate  10: no cache   10045 / with cache  1009
generate 100: no cache  104950 / with cache  1099
generate 500: no cache  624750 / with cache  1499

Without a cache, every generated token recomputes the whole context, so even 100 tokens cost over 100,000 computations. With a cache, after 1,000 in prefill it’s one more per token. The cache exists for every layer, so the longer the context, the more memory it takes. A web cache stores responses; a KV cache stores intermediate material for computation.

Temperature shapes the distribution; top-k and top-p pick the candidates

Choosing one token from the probability table is sampling. Always taking the top token gives a fixed result but flat text. So usually a few candidates are kept and one is drawn from them by probability. Three values you pass when calling a model’s API control this step.

Temperature is often described as where to cut off, but cutting is what top-k and top-p do. Temperature makes the distribution sharper or flatter before anything is cut. Here are the same five logits converted at three temperatures.

package main

import (
  "fmt"
  "math"
  "math/rand"
  "sort"
)

var vocab = []string{"nice", "clear", "cloudy", "cold", "cat"}
var logits = []float64{3.2, 2.9, 1.5, 1.1, -2.0}

func probs(t float64) []float64 {
  out, sum := make([]float64, len(logits)), 0.0
  for i, v := range logits {
    out[i] = math.Exp(v / t) // divide by the temperature
    sum += out[i]
  }
  for i := range out {
    out[i] /= sum
  }
  return out
}

// top-p: accumulate from the most likely and keep candidates until the total reaches p.
func topP(p []float64, limit float64) []int {
  idx := []int{0, 1, 2, 3, 4}
  sort.Slice(idx, func(a, b int) bool { return p[idx[a]] > p[idx[b]] })
  acc, keep := 0.0, []int{}
  for _, i := range idx {
    keep = append(keep, i)
    if acc += p[i]; acc >= limit {
      break
    }
  }
  return keep
}

// top-k: keep only the k most likely.
func topK(p []float64, k int) []int {
  idx := []int{0, 1, 2, 3, 4}
  sort.Slice(idx, func(a, b int) bool { return p[idx[a]] > p[idx[b]] })
  return idx[:k]
}

func main() {
  for _, t := range []float64{0.5, 1.0, 1.5} {
    p := probs(t)
    r := rand.New(rand.NewSource(42))
    count := make([]int, len(vocab))
    for n := 0; n < 1000; n++ {
      x, acc := r.Float64(), 0.0
      for i, v := range p {
        if acc += v; x < acc {
          count[i]++
          break
        }
      }
    }
    fmt.Printf("T=%.1f  of 1000: %v  top-p 0.9 keeps %d, top-k 3 keeps %v\n",
      t, count, len(topP(p, 0.9)), names(topK(p, 3)))
  }
}

func names(idx []int) []string {
  out := []string{}
  for _, i := range idx {
    out = append(out, vocab[i])
  }
  return out
}

$ go run ./sampling
T=0.5  of 1000: [615 350 23 12 0]  top-p 0.9 keeps 2, top-k 3 keeps [nice clear cloudy]
T=1.0  of 1000: [506 332 95 63 4]  top-p 0.9 keeps 3, top-k 3 keeps [nice clear cloudy]
T=1.5  of 1000: [433 319 119 113 16]  top-p 0.9 keeps 4, top-k 3 keeps [nice clear cloudy]

This draw is a big reason the same question gets a different answer each time. The weights stay the same throughout.

So which one do you pick?

At inference time, the only thing you can change is the input token sequence. Following it this far, the answer to the opening question becomes an order of steps.

Fine-tuning comes down to repeated formats and cost

If you want a particular response format, you put instructions and examples in the system prompt, and that’s usually enough. The trouble is that those long instructions ride along on every call.

Say a manufacturing or pharmaceutical site produces a flood of data, and a simple task that rules can’t decide, so it needs an LLM, runs a million times a day. The response must be JSON in a fixed order. However small the per-call cost, it adds up over a month.

package main

import "fmt"

// Total input tokens for the same task called 30 million times a month (token counts are assumptions).
func main() {
  const calls = 1_000_000 * 30 // 1M calls a day × 30 days
  input := 150                 // the actual data, different each call

  cases := []struct {
    name   string
    system int // instructions, examples, output format sent with every call
  }{
    {"format via prompt", 1200},
    {"short prompt after FT", 50},
  }
  for _, c := range cases {
    per := c.system + input
    fmt.Printf("%-22s %4d tokens/call → %5.1fB tokens/month\n",
      c.name, per, float64(per*calls)/1e9)
  }
}

$ go run ./cost
format via prompt      1350 tokens/call →  40.5B tokens/month
short prompt after FT   200 tokens/call →   6.0B tokens/month

The token counts are assumptions, but the structure is clear. Baking the format into the weights lets you drop the instructions sent each time, and under these assumptions monthly input fell from 40.5B to 6.0B tokens.

This calculation leaves out prompt caching, though. Today’s APIs cache the identical prefix sent with every call and charge less for it. The Anthropic API bills cached input reads at 10% of the base input price, and less for some models.² Charging the 1,200 instruction tokens at 10% gives 270 tokens’ worth per call, or 8.1B tokens’ worth a month, which leaves much less of a gap to fine-tuning’s 6.0B. Once you add fine-tuning’s cost and upkeep, it’s worth working out the cached number first.

So fine-tuning is worth considering when:

• the same response format repeats endlessly,
• the task is a narrow, well-defined domain job, and
• the call volume is large enough that shrinking the input saves more than training costs.

Put the other way, without volume there’s little reason to fine-tune. As models and prompting improve, prompts handle more, and the space where fine-tuning is needed keeps shrinking.

RAG puts only the needed evidence into the input

Internal policies and yesterday’s documents aren’t anywhere in the weights. Put all the policies into the prompt and attention spreads thin; if the documents exceed the context window, they won’t fit at all. RAG (Retrieval-Augmented Generation) finds only the parts relevant to the question and inserts them into the token sequence.

Here’s how the finding works. Split documents into chunks, turn them into embedding vectors and store them; turn the question into a vector too, and fetch the chunks closest in meaning. The toy below scores by overlapping words instead of vectors, but the flow is the same: search, pick the top chunks, assemble the prompt. The model is called after that.

package main

import (
  "fmt"
  "sort"
  "strings"
)

// Search → pick the top documents → insert them into the prompt. The model is called after that.
func main() {
  docs := []string{
    "Leave policy: 15 days of annual leave after one year of service.",
    "Deploy rule: no production deploys on Friday afternoons.",
    "Leave requests must be approved by your team lead three days ahead.",
  }
  question := "When must leave requests be filed?"

  type hit struct {
    score int
    doc   string
  }
  var hits []hit
  for _, d := range docs {
    s := 0
    for _, w := range strings.Fields(question) {
      if strings.Contains(d, strings.Trim(w, "?")) {
        s++
      }
    }
    hits = append(hits, hit{s, d})
  }
  sort.Slice(hits, func(i, j int) bool { return hits[i].score > hits[j].score })

  fmt.Println("[Reference documents]")
  for _, h := range hits[:2] {
    fmt.Printf("- (%d) %s\n", h.score, h.doc)
  }
  fmt.Println("[Question]\n" + question)
  fmt.Println("[Instruction]\nAnswer only from the reference documents; if they don’t cover it, say you don’t know.")
}

$ go run ./rag
[Reference documents]
- (3) Leave requests must be approved by your team lead three days ahead.
- (1) Leave policy: 15 days of annual leave after one year of service.
[Question]
When must leave requests be filed?
[Instruction]
Answer only from the reference documents; if they don’t cover it, say you don’t know.

The leave-request rule, sharing three words with the question, came first, the leave policy with one came next, and the deploy rule was left out. To the model, the context simply got longer. The weights don’t change, and attention does the work of finding the evidence in the inserted documents.

That’s why RAG fits especially well when documents change often. Re-running fine-tuning for every document change doesn’t pay off, but with RAG you just update the store. There’s a cost: if retrieval brings back the wrong chunks, the answer is wrong too.

Prompting handles the rest

Outside those two cases, most things are solved with prompts and instructions. Giving an agent skills or writing instruction files belongs here too, broadly speaking, because it’s all a question of what goes in front of the token sequence. In practice, prompting plus RAG is the most common combination.

Choosing a model follows the same line. Check constraints like security requirements or on-premises deployment first; within them, it’s mostly a question of cost.

Wrap-up

• Tokens are pieces made by compressing the corpus at the byte level. Vocabularies differ by model, so the same sentence has different token counts.
• Training adjusts the weights so the right token gets a higher probability. Pre-training produces a base model; post-training makes it usable.
• Inference runs fixed weights through N layers to get a probability table, samples one token, and repeats until a stop token.
• Attention re-measures relationships to earlier tokens each time, and the KV cache reuses Keys and Values already computed.
• If the input can solve it, prompt; if the model needs facts it lacks, use RAG; if one format runs at large scale, consider fine-tuning.

“Should I use RAG or fine-tune?” looked like a question comparing tool names. After following training and inference, it narrowed to one question: change the weights, or change the input? Keeping up with every new technique used to wear me out; now I start by asking which part of this loop it touches. Most of what I’ve looked at so far turned out to be about how to fill the input token sequence.

References

• Attention Is All You Need (Vaswani et al., 2017): the transformer
• Neural Machine Translation by Jointly Learning to Align and Translate (Bahdanau et al., 2014): attention, first used in machine translation
• Neural Machine Translation of Rare Words with Subword Units (Sennrich et al., 2015): subword tokens with BPE
• RoFormer: Enhanced Transformer with Rotary Position Embedding (Su et al., 2021): rotary position embedding (RoPE)
• Training language models to follow instructions with human feedback (Ouyang et al., 2022): SFT and RLHF
• Direct Preference Optimization (Rafailov et al., 2023): DPO
• LoRA: Low-Rank Adaptation of Large Language Models (Hu et al., 2021): adapter-based fine-tuning
• The Curious Case of Neural Text Degeneration (Holtzman et al., 2019): top-p (nucleus) sampling
• Lost in the Middle: How Language Models Use Long Contexts (Liu et al., 2023): weaker use of information in the middle of long inputs
• Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks (Lewis et al., 2020): RAG
• Claude API · Prompt caching: cache read pricing and lifetime
• Claude API · Extended thinking: reasoning tokens are billed as output tokens
• tiktokenizer: see how each model splits text into tokens