LLMs & AI Engineering · How LLMs work
Preview: how LLMs work (tokens, attention, training)
A big-picture preview for the FDE path: an LLM is a next-token predictor trained on huge text, then tuned to be helpful. Python part 2 stays next in the main sequence.
Step 1 of 7 · 1. Text → tokens → numbers35 min read ·
1. Text → tokens → numbers
A model never sees letters. Text is split into tokens (common words or word pieces); each token becomes an id, then a vector (embedding). Roughly, 1 token ≈ ¾ of an English word.
"Trade T2 qty mismatch"
your text
Tokens
Trade | T | 2 | qty | mismatch
Embeddings
vectors
Transformer layers
attention + MLP
Next-token probabilities
'between' 31%, ':' 18% …
Next: 2. Attention