AI Engineering Interview Preparation

RAG is a pattern where, instead of relying on what a model memorised during training, you retrieve relevant documents at query time and put them into the prompt so the model answers from evidence you control.

There are two pipelines. Offline (indexing): documents are loaded, cleaned, split into chunks, passed through an embedding model that turns each chunk into a dense vector, and written to a vector store alongside the original text and metadata.

Online (query time):

1. The user's query is embedded with the same model used at index time.
2. The vector store runs an approximate nearest-neighbour search and returns the top-k most similar chunks.
3. Optionally a reranker — usually a cross-encoder — rescores those candidates and keeps the best few.
4. The surviving chunks are pasted into a prompt template alongside the question and an instruction like “answer only from the context; if the answer isn't there, say so.”
5. The LLM generates the answer, ideally citing the chunk IDs it used.

The key insight to say out loud: RAG does not change the model's weights. It changes what is in the context window at inference time. That is why it is cheap to update — reindex a document and the system's knowledge changes instantly.

A pre-trained LLM knows only what was in its training data, has no idea which parts it actually knows, and cannot cite anything. RAG fixes all three by grounding answers in retrievable, attributable source text.

Concretely it solves four problems:

Knowledge cutoff

The model cannot know about your product launched last week, or last night's incident report. Retrieval supplies it.

Private data

Your contracts, tickets and wiki were never in any public training set — and you don't want them there.

Hallucination

Grounding in retrieved text plus an instruction to abstain measurably reduces fabricated answers, and gives you something to check the answer against.

Attribution

You can show the user the source paragraph. In regulated domains this is often a hard requirement, not a nice-to-have.

Compared with the alternatives: fine-tuning teaches style, format and task behaviour but is a poor and expensive way to inject facts — and you must retrain to update them. Long-context stuffing (dumping the whole corpus in the prompt) works for a few hundred pages but scales badly on cost, latency and attention quality. RAG is the option where knowledge updates are a database write.

Seven components: ingestion, chunker, embedding model, vector store, retriever, reranker, and generator — plus an evaluation loop wrapped around the whole thing.

The interaction that matters most: quality is bounded by the weakest upstream stage. A frontier LLM cannot rescue a retriever that returned the wrong paragraph, and a great retriever cannot rescue a chunker that cut a table in half. That is why debugging starts at ingestion, not at the prompt.

Keyword search matches strings; vector search matches meaning. BM25 asks “which documents contain these tokens?”, dense retrieval asks “which chunks sit closest to this query in embedding space?”

Two things follow. First, dense retrieval returns something for every query — there's no "no results". A low-relevance answer still comes back with a plausible-looking score, so you need a similarity threshold or a reranker to decide when to say "I don't know."

Second, and this is the answer interviewers want: in production you rarely pick one. Hybrid search runs both and fuses the rankings, because the two fail in opposite directions.

Two distinct failures. Irrelevant context distracts the model into confidently wrong answers; contradictory context creates a conflict between parametric memory and retrieved evidence, and which one wins is not something you should leave to chance.

Irrelevant context. Models are strongly biased toward using what they're given — a plausible-looking but wrong passage often produces a worse answer than no context at all. Mitigations:

• Set a similarity floor; if nothing clears it, don't call the LLM with junk — return "no supporting information found."
• Add a reranker and keep only the top 3–5 after rescoring.
• Add a lightweight relevance grader (a cheap model that labels each chunk keep/discard) — this is the core idea behind Corrective RAG.
• Prompt explicitly: “If the context does not contain the answer, reply exactly: Not found in the provided sources.”

Contradiction with training data. Decide the precedence rule deliberately and state it in the system prompt: for enterprise knowledge, retrieved documents should win, because they're current and authoritative. Then reinforce it structurally — put source, date and authority level in the context block, and instruct the model to prefer the most recent authoritative source and to surface the disagreement rather than silently pick a side.

They're three generations of the same idea: naive RAG is retrieve-then-read in a straight line; advanced RAG adds pre- and post-retrieval optimisation around that line; modular RAG breaks the line into reconfigurable components with routing and loops.

The progression is driven by failure, not fashion. You move to advanced RAG when you discover retrieval is missing obviously relevant documents. You move to modular RAG when a single retrieval pass structurally cannot answer the question — “compare our 2024 and 2025 refund policy and tell me what changed” needs two retrievals and a comparison step.

Evaluate retrieval and generation separately, because they fail separately — then add end-to-end and operational metrics on top. If you only measure the final answer you cannot tell whether a bad answer came from bad retrieval or bad synthesis.

Retrieval metrics (need a labelled set of query → relevant-chunk pairs):

• Recall@k — did the right chunk appear in the top k? Usually the single most important number; if recall is low, nothing downstream can save you.
• Precision@k — how much of what you retrieved was noise.
• MRR / nDCG@k — rank-aware: was the right chunk near the top, not just present.

Generation metrics (typically LLM-as-judge, e.g. RAGAS-style):

• Faithfulness / groundedness — is every claim supported by the retrieved context? This is your hallucination detector.
• Answer relevance — does it actually address the question asked.
• Context precision / recall — bridges the two halves: was the useful context ranked highly and was all needed context present.
• Citation accuracy — do the cited sources really contain the claim.

Operational: p50/p95 latency by stage, cost per query, abstention rate, and — the one that matters commercially — human thumbs-up rate on real traffic.

Almost every RAG failure lands in one of six buckets, and you localise it by walking the pipeline backwards from the answer: check what was retrieved before you touch the prompt.

The debugging procedure: log every query with its retrieved chunk IDs, scores, final prompt, and answer — this is non-negotiable and is the difference between debugging and guessing. Then for a failing query: (a) is the answer in the corpus at all? (b) if yes, does it appear at any k up to 100? (c) if yes, is it in the final prompt after reranking and truncation? (d) if yes, it's a generation problem. Each "no" points at exactly one stage.

First establish whether it's a real contradiction or just two documents from different points in time — most "conflicts" in enterprise corpora are versioning problems, and metadata solves them before the LLM ever sees the text.

1. Prevent avoidable conflicts at index time. Every chunk carries source, effective_date, version, authority_tier and status. Deprecated policies get filtered out or down-weighted at retrieval, so the model never has to arbitrate between the 2023 and 2025 handbook.

2. Deduplicate and cluster. Near-identical chunks waste context and amplify whichever version happens to be duplicated more. Collapse them, keeping the newest.

3. Give the model an explicit resolution policy. Present each chunk with a labelled header ([S3 | Policy Manual | v4 | 2025-06-01]) and instruct: prefer the most recent authoritative source; if sources genuinely disagree on a material point, state both positions with their citations rather than silently choosing.

4. For multi-hop questions, decompose. If the answer requires combining facts rather than choosing between them — "which of our EU customers is affected by the clause added in v4?" — a single retrieval pass is the wrong tool. Break the query into sub-questions, retrieve per sub-question, then synthesise.

5. Surface, don't hide. Returning “Source A says 30 days, Source B (newer) says 14 days — I've used 14” is a better product outcome than a confident single number, and it's usually what a compliance stakeholder actually wants.

A vector database is built around one operation a relational database cannot do efficiently: find the k items whose high-dimensional embeddings are closest to a query vector. Relational databases answer “which rows match exactly?”; vector databases answer “which rows are most similar?”

The deeper reason B-trees don't work: they rely on a total ordering of one dimension. In 768 or 1536 dimensions there is no meaningful sort order, and classic spatial structures like KD-trees degenerate to brute force — the “curse of dimensionality.” So the field gave up on exactness and built approximate structures instead.

An embedding is a fixed-length list of floats — typically 384 to 3072 dimensions — that represents the meaning of a piece of text, arranged so that texts with similar meaning end up close together geometrically.

How it's produced: the text is tokenised, the tokens go through a transformer encoder, and the resulting per-token hidden states are pooled into a single vector — usually mean pooling or the [CLS] token. That vector is then normalised so cosine similarity reduces to a dot product.

Why the geometry means anything is the part people skip: it's the training objective. Embedding models are trained contrastively — pull matched pairs (a question and its answer, a sentence and its paraphrase) together, push mismatched pairs apart, typically with InfoNCE loss over in-batch negatives. The geometry is learned, not inherent.

Three practical consequences:

• Similarity is model-specific. Two models' vectors are not comparable, even at the same dimensionality.
• The training data defines "similar." A model trained on web Q&A may cluster legal clauses badly; domain models exist for a reason.
• Asymmetric tasks need asymmetric handling. Many models expect prefixes like query: / passage:; omitting them quietly costs you recall.

You can store them anywhere — a float array is just bytes. What you can't do in a standard database is search them fast, because there's no index that makes similarity search sublinear without a purpose-built ANN structure.

Put ten million 1536-dimension vectors in a plain table and every query becomes a full scan: ten million dot products of 1536 floats each. That's tens of gigabytes of memory bandwidth per query — hundreds of milliseconds to seconds, single-user. A vector index turns that into a few thousand comparisons and single-digit milliseconds.

Beyond the index, purpose-built stores give you things you'd otherwise build yourself:

• Filtered ANN — combining metadata predicates with similarity search without destroying recall (harder than it sounds; naive post-filtering can return nothing).
• Quantization and compression — scalar/product/binary quantization to fit the index in RAM.
• Hybrid scoring — built-in BM25 + dense fusion.
• Operational tooling — sharding, replication, namespaces/multi-tenancy, live re-indexing.

Every index is a point on the same triangle: recall, latency, and memory. Flat maximises recall, HNSW maximises speed at high recall but costs RAM, IVF is the memory-and-build-time compromise, and quantization is the lever you pull on any of them.

The knobs to name: HNSW has M (edges per node — more edges, better recall, more memory), ef_construction (build-time effort) and ef_search (query-time effort — the runtime recall/latency dial). IVF has nlist and nprobe. Being able to say “I'd fix M and ef_construction at build time and tune ef_search against a recall target measured on a Flat baseline” is what separates a read-about-it answer from a used-it answer.

ANN deliberately gives up the guarantee of finding the true nearest neighbours in exchange for sublinear search time — and it's the right trade because in RAG a 98%-correct neighbour list produces an indistinguishable answer at a fraction of the cost.

The mechanism is always some form of "don't look at most of the data":

• Graph-based (HNSW): vectors become nodes connected to their neighbours across layers of decreasing sparsity. A search enters at the top layer, greedily hops toward the query, drops a layer, repeats. Long edges at the top cover distance; short edges at the bottom give precision. Roughly O(log N).
• Partition-based (IVF): cluster the space once, then only scan the few cells nearest the query.
• Hash-based (LSH): hash functions that collide for nearby vectors; check only the matching buckets.

Why exact search loses: at 10M × 1536 dims, brute force is ~15 GFLOPs of dot products per query. That's fine offline, impossible at 100 QPS with a 50 ms budget. And the accuracy you're buying is largely illusory — the embedding itself is a lossy approximation of meaning, so insisting on the exact top-10 of an approximate representation is false precision.

Answer this with a decision framework plus honest specifics — interviewers can tell instantly when someone lists six databases they've only read about. Name what you actually used, then show you know the landscape.

My selection criteria, in order: (1) scale and expected growth — under a million vectors, the answer is usually "the database you already have"; (2) filtering requirements, especially multi-tenant permission filters, where naive post-filtering silently destroys recall; (3) hybrid search support; (4) ops model — managed vs. self-hosted vs. a team that can run a distributed system; (5) data residency and compliance; (6) cost at projected QPS, not at demo scale.

You can't mix them. Vectors from two different models live in incomparable spaces, so a query embedded with v2 searched against a v1 index returns effectively random results — and it fails silently, with plausible-looking scores. The only correct answer is a full re-embed, executed as a versioned migration.

The migration pattern:

1. Version everything. Every collection and every vector row carries embedding_model and model_version. Queries assert the version they expect and fail loudly on mismatch.
2. Build alongside, don't mutate. Create a new v2 collection and backfill it from the stored raw chunk text — which is why you keep the source text in the payload, not just the vector.
3. Evaluate before switching. Run the golden query set against v1 and v2. A newer model with better MTEB scores can still be worse on your domain; this step is not optional.
4. Shadow, then cut over. Serve from v1 while mirroring traffic to v2, compare retrieved sets and downstream answer quality, then flip behind a feature flag with a one-line rollback.
5. Retire v1 after a bake period; keep the option to roll back until you're confident.

Cost control: backfill in batches off-peak, prioritise hot documents, and use a queue so ingest of new documents continues writing to both collections during the transition.

Work the pipeline in order of leverage: representation first, then retrieval strategy, then ranking, then the index parameters. Tuning ef_search is the last thing I'd do, not the first — index tuning buys recall against your own embeddings, not relevance.

1 · Representation. Pick the embedding model empirically on your data, not from a leaderboard. Fix the chunking — structure-aware splits that respect headings, tables and code blocks, with overlap; prepend document title and section path to each chunk so a fragment carries its context. Consider parent-document retrieval: embed small precise chunks, return the larger parent for generation.

2 · Retrieval strategy. Hybrid dense + BM25 fused with Reciprocal Rank Fusion. Query rewriting for conversational follow-ups ("what about the second one?" is meaningless as an embedding). Multi-query expansion for recall. Metadata pre-filtering to shrink the candidate space before scoring.

3 · Ranking. Retrieve wide (k=50–100), rerank narrow (keep 3–8) with a cross-encoder. This is typically the single largest relevance gain available, because a cross-encoder sees query and passage jointly rather than comparing two independently-computed vectors. Add business-aware boosts: recency, authority, document type.

4 · Index and infra. Now tune ef_search/nprobe against measured recall; quantize to fit in RAM; shard by tenant so filters become cheap; cache embeddings for repeated queries.

Distinguish two things that both get called "hybrid": fusion of two ranked result sets (dense + sparse), and filtering by structured metadata. They're different problems with different failure modes.

Fusion — combining BM25 and vector results. The scores aren't comparable (BM25 is unbounded, cosine is [-1,1]), so don't naively add them. Use Reciprocal Rank Fusion: score = Σ 1/(k + rank_i), typically k=60. It's rank-based, needs no score normalisation, and is remarkably hard to beat. If you do want weighted score fusion, normalise per-query (min-max over the returned set) and expose the weight as a tunable α — then tune α on your golden set, not by intuition.

Filtering — the part that actually bites. Three strategies:

The design I'd propose: permissions and tenancy as hard pre-filters (never post-filter security — it's a correctness and leak risk, and thin results are the least of it); soft attributes like recency and doc type as reranking boosts; dense + BM25 fused with RRF; cross-encoder rerank over the fused top-50; return top-5 with citations. Then measure each stage's contribution by ablation.

Temperature is a divisor applied to the logits before the softmax. It controls how sharply peaked the next-token probability distribution is — low temperature concentrates probability on the model's top candidates, high temperature flattens the distribution and lets unlikely tokens through.

Mathematically: p_i = softmax(z_i / T). As T → 0 the distribution collapses onto the argmax (greedy decoding). At T = 1 you sample from the model's raw learned distribution. Above 1 you're deliberately flattening it.

Max tokens is a hard ceiling on how many tokens the model may generate. It is not a length instruction — the model doesn't plan around it. When the limit is hit, generation stops mid-sentence and the API returns a length finish reason.

Three things to get right:

• It's output-only, but it competes with input. Input + output must fit the context window. Many APIs reserve the max-tokens budget up front, so an over-generous setting can cause a request to fail even though the actual answer would have been short.
• It's a cost and latency cap, not a style control. To get short answers, instruct the model ("answer in two sentences") and set max tokens as a safety net. Relying on the cap alone gives you truncated garbage.
• Always check the finish reason. A response truncated at the limit is a bug you should detect and handle — retry with a larger budget, or stream and continue — not silently ship. This is especially dangerous with JSON output, where truncation produces unparseable results.

Reasoning models add a wrinkle worth mentioning: thinking tokens consume the output budget before any visible text is produced, so a limit that was comfortable for a non-reasoning model can leave nothing for the actual answer.

Both truncate the candidate set before sampling. Top-K keeps a fixed number of tokens; Top-P keeps however many tokens are needed to reach a cumulative probability mass. The difference is that Top-P adapts to the shape of the distribution and Top-K doesn't.

Worked example. Suppose the model is very confident — the top token has p = 0.92. With Top-K = 40 you've just admitted 39 tokens with a combined 8% probability, giving nonsense a chance it didn't earn. With Top-P = 0.9 the nucleus is a single token; you sample the obvious continuation. Now suppose the model is genuinely uncertain, with 200 tokens each around 0.4%. Top-K = 40 arbitrarily amputates a legitimate long tail; Top-P = 0.9 expands to include ~180 of them.

Which to use: Top-P is the sensible default and is what most APIs expose. Top-K is useful as a hard safety cap on the tail, and some stacks apply both (K first, then P). The practical guidance most providers give — and a good line to repeat — is tune temperature or top-p, not both simultaneously, because you can't attribute the effect.

Both subtract from the logits of tokens that have already appeared, discouraging repetition — but they differ in whether the penalty scales with how often the token appeared.

frequency_penalty

Penalty proportional to the count of prior occurrences. A word used five times is penalised five times as hard as one used once. Targets verbatim loops and overused words.

presence_penalty

Flat penalty applied once a token has appeared at all, regardless of count. Pushes the model toward introducing new vocabulary and new topics.

Typical ranges are −2.0 to 2.0, with useful values usually between 0.1 and 0.8. Rules of thumb: use a small frequency penalty when output degenerates into repeated phrases; use a small presence penalty when you want topical breadth, e.g. "give me 20 distinct ideas" where the model keeps circling the same three.

Note also that not every provider exposes both; some expose a single repetition penalty (a multiplicative variant common in open-source stacks), which is worth flagging if you're asked about portability across models.

They optimise for opposite properties: creative writing wants variance and surprise, factual Q&A wants reproducibility and grounding. Almost every parameter moves in the opposite direction.

Beyond sampling, the whole configuration differs. Factual Q&A gets a strict system prompt with an abstain instruction, structured output / JSON schema enforcement, low top-k retrieval with citations, and a faithfulness check on the output. Creative writing gets a rich persona prompt, few-shot style examples, no schema, and human judgement instead of automated scoring.

Determinism means identical inputs produce identical outputs. Note the word "near" in the question — that's deliberate, and acknowledging why full determinism is unattainable with a hosted model is the point of the question.

What you control:

• temperature = 0 (greedy decoding) and a fixed seed where the provider supports one.
• Pin the model version — model-2026-05-01, never a floating -latest alias. Silent model upgrades are the most common cause of "it changed and we didn't touch anything."
• Freeze the prompt, and everything that composes it: template version, retrieved chunks, chunk order. In RAG this is the big one — non-deterministic retrieval makes the LLM's determinism irrelevant.
• Constrain the output space — JSON schema / structured outputs / grammar-constrained decoding removes whole classes of variation.
• Cache — for a genuinely fixed input, serving a cached response is exact determinism, and it's cheaper.

Why residual non-determinism remains: floating-point addition isn't associative, so GPU kernel reductions vary with batch composition and hardware; providers batch requests from many tenants, so your numerics depend on who else is calling. Ties in the argmax then break differently, and a single divergent token cascades. Mixture-of-experts routing adds another batch-dependent source. Providers expose system_fingerprint precisely so you can detect when the backend configuration changed.

Yes — they compose in a specific order, and several common combinations are either redundant or actively self-defeating. The pipeline is: logits → penalties → temperature scaling → top-k truncation → top-p truncation → renormalise → sample.

The interactions that matter:

• Temperature 0 makes top-p and top-k inert. Greedy decoding picks the argmax; truncating a distribution you never sample from changes nothing. Setting temperature=0, top_p=0.7 isn't wrong, it's just theatre — and it misleads the next engineer into thinking top_p is doing something.
• High temperature + low top-p partially cancel. You flatten the distribution, then cut off the tail you just created. The result is more uniform sampling among a few candidates — not the same as either setting alone, and hard to reason about. This is why the standard advice is to tune one.
• Low temperature + high penalties fight each other. The temperature says "be confident and consistent"; the penalty forcibly suppresses the confident token because it appeared before. In structured output this produces malformed JSON — the model is banned from repeating a key it needs.
• Top-k and top-p together: the more restrictive one wins for that token, so the effective behaviour flips between them unpredictably as the distribution shape changes.
• max_tokens interacts with everything downstream: higher temperature tends toward more rambling output, which hits the ceiling more often, which shows up as truncation bugs rather than as a sampling problem.

I'd separate the two symptoms first, because they have different causes. Inconsistency is usually a sampling or version problem and parameters fix it. Hallucination is usually a grounding problem, and parameters barely touch it — so answering “lower the temperature” alone is the wrong answer here.

For inconsistency — parameters first, in this order:

1. temperature → 0–0.2. Biggest single lever on variance.
2. Check for a floating model alias; pin the version. Then check system_fingerprint for a silent backend change.
3. Neutralise redundant knobs: top_p = 1, penalties = 0, so one thing controls variance.
4. Set a seed if available; add structured output constraints.
5. Verify the prompt is actually identical — retrieved chunks, chunk ordering, timestamps and conversation history injected into the template are common hidden sources of variation.

For hallucination — parameters are step zero, not the fix:

1. Drop temperature to near 0 and zero the penalties. Cheap, worth doing, rarely sufficient.
2. Inspect the retrieved context. Was the answer even in the prompt? Most "hallucinations" in RAG are retrieval failures wearing a disguise.
3. Tighten the prompt: explicit abstain instruction, require citations per claim, clearly delimit the context block.
4. Reduce noise: rerank and keep fewer, better chunks; irrelevant context increases fabrication.
5. Add a verification pass — faithfulness scoring or a claim-checking step — and route low-confidence answers to abstention or a human.
6. Consider a stronger model for the generation step; some are markedly better at declining to answer.

The context window is the maximum number of tokens a model can attend to in a single forward pass — system prompt, conversation history, retrieved documents, the user's question, and the generated output all share it. It is the model's entire working memory for that request.

Why it matters, in four directions:

• Capability. It bounds what the model can reason over at once — a 200-page contract, a whole codebase, a long support thread.
• Statelessness. LLMs have no memory between calls. A multi-turn conversation feels continuous only because you resend the history every turn. The context window is therefore the hard limit on apparent memory.
• Cost and latency. Attention cost grows quadratically with sequence length in the classic formulation, and you pay per input token. Prefill time scales with input size, so a large context directly increases time-to-first-token.
• Quality. The advertised maximum is not the useful maximum. Performance typically degrades well before the limit, so "it fits" and "it works well" are different claims.

Give tiers and ranges rather than precise numbers — they change every few months, and the strong answer signals that you know the difference between advertised and effective window size.

The two caveats that make this a good answer: first, most models deliver reliable quality at only roughly 60–70% of their stated maximum, so treat the number as a hard limit rather than a working target. Second, output limits are separate and much smaller than input limits — a model with a 1M input window may cap generation at a few tens of thousands of tokens.

The request fails — you get a hard API error (a context-length-exceeded 400), not a graceful degradation. The model cannot process the sequence at all, because positional and attention structures are defined only up to the trained length.

The important nuance is that input + reserved output must fit, so you can be rejected while the input alone is under the limit. And behaviour differs by layer:

• Raw API: immediate error. Clean, and preferable — it's visible.
• Frameworks and chat products: often silently truncate the oldest messages or drop retrieved chunks. This is far more dangerous, because the system keeps answering while quietly forgetting the system prompt or the instruction that mattered.

What I'd do about it: count tokens before sending, using the model's actual tokenizer rather than a character heuristic. Budget the window explicitly — a fixed reserve for the system prompt and output, then allocate what's left between retrieved context and history. When the budget is exceeded, degrade deliberately and log it: summarise older turns, drop the lowest-scoring chunks first, or fall back to a longer-context model. Never truncate blindly from the front, which is exactly how systems lose their safety instructions.

Match the strategy to the task, because the right answer differs for "answer a question about this document" versus "summarise the whole thing" versus "keep this conversation coherent."

For long documents:

• Retrieval (RAG) — the default for targeted questions. Don't read the document; retrieve the relevant few thousand tokens.
• Map-reduce — for tasks that genuinely need the whole document (summarise, extract every clause of type X): process chunks independently, then combine.
• Refine / iterative — carry a running answer through the chunks sequentially. Better for narrative coherence, worse for latency, and drifts over many steps.
• Hierarchical summarisation — summarise sections, then summarise the summaries. Good for whole-corpus overviews; lossy by construction.

For long conversations:

• Sliding window — keep the last N turns verbatim. Simple, cheap, forgets abruptly.
• Rolling summary + recent window — the workhorse: a compact running summary of everything older, plus the last few turns in full.
• Retrieval over history — embed past turns and pull back only what's relevant now. Scales indefinitely.
• Structured state — extract durable facts (user's name, constraints, decisions) into a small key–value store that's always injected, so they never fall out of the window.

Cross-cutting: use prompt caching for the stable prefix (system prompt, long shared documents) — it cuts cost dramatically on repeated calls; compress retrieved context; and always leave headroom for the output.

Chunking is the highest-leverage and most under-tuned decision in RAG. The chunk is simultaneously the unit of embedding, the unit of retrieval, and the unit of context — so a bad split degrades all three at once, and no downstream component can repair it.

The core tension:

What actually fixes it isn't finding a magic number — it's respecting structure. Split on semantic boundaries (headings, sections, paragraphs, function definitions), never mid-table or mid-sentence. Add 10–20% overlap so a fact spanning a boundary survives. Prepend the document title and heading path to every chunk so a retrieved fragment still knows what it's about. And consider parent-document / small-to-big retrieval: embed small precise chunks for matching, but hand the LLM the surrounding parent section — which resolves the tension rather than compromising on it.

They differ in what they throw away: sliding window discards by age, hierarchical summarisation discards by detail, and retrieval discards by relevance. That framing answers the question in one sentence — then compare.

The failure each one has: sliding windows forget a constraint the user set twenty turns ago; summarisation smooths away the exact number someone will later be held to; retrieval can't answer questions whose answer isn't localised in any chunk ("what's the overall tone of this contract?").

Models attend unevenly across position: accuracy at retrieving a fact from the context follows a U-shape, highest at the beginning and end and measurably worse in the middle. So a relevant document buried mid-prompt can be effectively invisible even though it's technically there.

Why it happens: a mix of primacy from causal attention and positional encoding behaviour, recency bias, and training-data distribution — instructions typically appear at the start of documents and conclusions at the end, so the model learns those positions carry weight. The effect worsens as context length grows.

Mitigations, roughly in order of impact:

1. Use less context. The most effective fix. Rerank aggressively and pass 3–5 excellent chunks instead of 20 mediocre ones — the U-curve barely matters when there's no middle.
2. Order strategically. Put the highest-scoring chunks first and last, weakest in the middle (sometimes called "lost-in-the-middle reordering"). Cheap and measurable.
3. Repeat the instruction at the end. Restate the question after the context block so it occupies a high-attention position.
4. Structure the context. Numbered, delimited chunks with headers and source labels make retrieval-within-the-prompt easier than an undifferentiated wall of text.
5. Compress. Extractive filtering of retrieved passages before they enter the prompt.
6. Decompose. For genuinely long material, several focused calls beat one enormous one — map-reduce sidesteps the problem entirely.

Cost scales roughly linearly with input tokens; latency is worse than linear because prefill attention scales quadratically with sequence length. A 10× bigger prompt costs about 10× more and adds disproportionately to time-to-first-token — and often produces a worse answer because of the middle-of-context effect.

Where the money and milliseconds go:

• Prefill — processing the input. Dominates TTFT and scales with prompt size.
• Decode — generating tokens, roughly linear in output length, and where most of the wall-clock time goes for long answers.
• KV cache memory — grows with sequence length and directly limits how many concurrent requests a served model can handle. On self-hosted infra this is the real constraint on throughput.

Optimisations that actually move the numbers:

1. Prompt caching — the single biggest win for repeated stable prefixes. Cached reads are typically ~90% cheaper than fresh input tokens, so structure prompts with the static part (system prompt, long shared document, few-shot examples) first and the variable part last.
2. Retrieve less, rerank harder. Fewer, better chunks cut cost and improve quality simultaneously — the rare free lunch.
3. Route by difficulty. Send easy queries to a small cheap model, escalate only what needs a frontier model. Often a 5–10× blended cost reduction.
4. Stream — doesn't reduce total latency, but cuts perceived latency dramatically.
5. Cap and monitor — token budgets per request, plus dashboards on tokens-per-query and cost-per-resolved-question rather than raw spend.
6. Semantic caching of whole responses for repeated or near-duplicate questions.

I'd design it as tiers with different lifetimes and different retrieval rules, modelled loosely on working / episodic / semantic memory — because "keep the last N turns" and "remember the user is vegetarian" are different problems that shouldn't share a mechanism.

Write path: after each turn, an extraction step decides what is worth promoting from working memory into the profile — and, critically, handles updates and contradictions. "I moved to Berlin" must overwrite the old city, not sit alongside it. Every fact carries a timestamp, a source turn, and a confidence, so newer explicit statements win.

Read path: assemble the prompt in a fixed budget — system prompt, profile, rolling summary, retrieved episodic snippets, recent turns verbatim, current message. Order matters: put durable instructions where attention is strongest.

The hard parts to name — these are what the interviewer is listening for: forgetting and decay (memory that only grows becomes noise), contradiction resolution, privacy and deletion (a user must be able to say "forget that", and GDPR erasure has to reach the vector store too), and evaluation — you need tests for "does it remember X after 50 turns" and for false memories, which are worse than forgetting.

At the frontier the capability gaps are narrower than the marketing suggests and they change with every release — so answer in terms of ecosystem, integration and durable design differences rather than claiming one is smarter.

Where genuine differences persist: input modalities supported natively; context window and effective long-context behaviour; the shape of tool-calling and structured-output APIs; safety and refusal behaviour, which materially affects some enterprise domains; data-residency and compliance options; and price per million tokens, which can differ severalfold for comparable quality.

Proprietary models are served to you through an API — you rent capability. Open-weight models give you the weights to run yourself — you own the deployment and everything that comes with it. Note the terminology point: most "open-source" LLMs are open-weight; the training data and code usually aren't released, and licences vary.

The crossover logic: API models win on time-to-market, spiky traffic and frontier capability. Self-hosting wins on strict data residency, very high steady volume, heavy customisation, or a need for reproducibility. The break-even is genuinely a spreadsheet exercise — a GPU costs the same whether it's busy or idle, so utilisation is the deciding variable.

I'd work backwards from the constraints that can disqualify a model outright, then benchmark only the survivors on real tasks. Starting from "which model is best?" is the wrong direction — most decisions are settled by requirements, not by leaderboard position.

Step 1 — hard constraints (these eliminate candidates): data residency and compliance; whether data may leave the perimeter at all; required modalities; latency SLA; maximum acceptable cost per request; licence terms for commercial use.

Step 2 — task requirements: what does the task actually demand? Extraction and classification need accuracy and structured output, not reasoning depth. Agentic workflows need reliable tool calling. Long-document analysis needs effective long context. Match the requirement, not the reputation.

Step 3 — build a task-specific evaluation. 50–200 real examples with expected outputs, scored automatically where possible plus human review of a sample. Run every surviving candidate, including a small cheap one — you will often find a model a tenth of the price is indistinguishable on your task.

Step 4 — score on the full picture: quality, p95 latency, cost per 1,000 requests at projected volume, failure modes (does it fail loudly or plausibly?), rate limits and availability, and switching cost.

Step 5 — design for change. Abstract the provider behind an interface, keep prompts versioned and portable, and re-run the eval when new models ship. The right model in six months is probably not today's.

The difference reduces to the attention mask and the training objective. Encoders attend bidirectionally and are trained to understand; decoders attend causally and are trained to continue; encoder-decoders do both and are trained to transform one sequence into another.

Why decoder-only won is a good thing to be able to explain: it's a single stack, so it scales more simply; every token is a training signal, making pre-training data-efficient; the KV cache makes autoregressive inference efficient; and in-context learning turned out to make explicit task-specific architectures unnecessary — you just prompt.

When the task is narrow, high-volume and well-specified. A fine-tuned 7B model can match or beat a frontier model on a single task at a fraction of the cost and latency — the frontier model's advantage is breadth, and on a narrow task you're paying for breadth you don't use.

Choose the small fine-tuned model when:

• The task is stable and repetitive — classification, entity extraction, routing, structured parsing, tone rewriting, moderation.
• Volume is high. Millions of calls a day is where per-token pricing hurts and a fixed GPU cost wins.
• Latency matters. A small model on your own hardware can be an order of magnitude faster to first token.
• You have labelled data — ideally a few thousand examples, which you can bootstrap by distilling from a frontier model.
• Data can't leave your perimeter, or you need the exact same weights available in five years.
• Output format must be rigid — fine-tuning is very effective at locking in a schema, and it also shortens prompts, since the instructions are baked into the weights.

Stay with the large general model when the task space is open-ended, requirements change weekly, you lack training data, reasoning depth is the actual product, or volume is low enough that engineering time dwarfs inference cost.

A base model is a raw next-token predictor trained on unlabelled text — ask it a question and it may well continue with more questions, because that's what documents do. An instruction-tuned model has been post-trained on instruction–response pairs (SFT) and usually aligned with preference optimisation (RLHF/DPO) so it treats input as a request to fulfil.

The practical rule: if you're building a product, use the instruction-tuned variant. Reach for the base model only when you intend to do your own substantial fine-tuning — starting from base avoids inheriting another team's alignment and formatting habits, and gives you a cleaner canvas for a heavily domain-specific model.

Frame it as three cost structures, not three quality levels — and lead with the observation that for a cost-sensitive application the right architecture is usually a router across all three, not a single choice.

The economics to state explicitly: a self-hosted GPU costs the same whether it serves one request or a million, so self-hosting only wins above a utilisation threshold — and that threshold is a real calculation involving GPU-hour price, tokens/second throughput, and the vendor's per-million-token rate. Below it, the API is cheaper and better. Above it, self-hosting can be several times cheaper. Add engineering time honestly: an ML platform engineer costs more than a lot of API tokens.

Public benchmarks are close to useless for this decision — they're saturated, subject to training-set contamination, and measure tasks that aren't yours. I'd build a task-specific evaluation set and treat the comparison as an experiment with controls.

Quality: accuracy against a golden set of 100–300 real examples; task-appropriate metrics (exact match for extraction, faithfulness and citation accuracy for RAG, pass@1 for code, LLM-as-judge with a human-validated rubric for open-ended text). Report distributions, not just means — a model that's usually great and occasionally catastrophic may be worse for you than a consistent one.

Reliability: structured-output validity rate, tool-call correctness, instruction adherence, refusal rate on legitimate requests, and behaviour under adversarial or malformed input. Crucially: how does it fail — loudly, or with a confident wrong answer?

Performance: p50/p95/p99 latency and time-to-first-token measured at realistic concurrency, not single-request; throughput; rate limits and observed availability.

Cost: cost per completed task (not per token — a chattier model with cheaper tokens can lose), including retries and failed attempts. Model prompt-cache savings if the workload has a stable prefix.

Fairness of the comparison — this is where candidates slip: prompts must be optimised for each model separately. Handing model B a prompt tuned for model A and declaring B worse is the most common benchmarking error. Fix sampling parameters, run multiple trials per example to measure variance, use held-out data neither model was tuned on, and blind the human raters.

Quantization stores weights at lower numerical precision — 16-bit floats down to 8 or 4 bits — which shrinks the model roughly proportionally. Since LLM inference is memory-bandwidth-bound rather than compute-bound, smaller weights mean both a smaller footprint and faster generation, at some accuracy cost.

Terminology to keep straight — a common differentiator: GGUF is a file format (the llama.cpp ecosystem) that happens to package quantized weights at various levels like Q4_K_M or Q8_0, and it supports CPU and hybrid CPU/GPU offload. INT8/INT4 are precisions. GPTQ and AWQ are post-training quantization methods for GPU serving; QLoRA is fine-tuning on a quantized base. Conflating the format with the precision is the mistake to avoid.

Deployment consequences: a 70B model at INT4 fits on a single 48 GB GPU instead of needing multiple cards — that's the difference between a feasible and infeasible deployment, and it's usually the actual motivation. It also frees VRAM for a bigger KV cache, which raises concurrency.