LLMs

I am currently still working through this post and its not final. There could be erros and wrong assumtions until the final draft.

In this blog post I will formulate the idea behind the building blocks of LLMs that led to models like GPT2, LLama, and Gemini and more.

Tokenization

We begin first with the process know as tokenization of text. For context LLMs can’t simply reason with just language for exmaple if we give a prompt to an llm:

How can I multiply two matrices of different size? 

We have to give this english sentence a mapping from text \(\rightarrow\) number which gives a sequence of numerical tokens. We also need to adjust some vocabulary of words to numerical tokens so the llm can reason with a sequence of numerical tokens rather than words.

So given a \(\operatorname{Vocabulary}\) and a sequence of words \(\operatorname{Word} \in \operatorname{Vocabulary}\) we have a mapping to a \(\operatorname{Token}\in \mathbb{Z}\)

Now if we look into the above prompt we can see for now how we can map the context of our sentece into numerical tokens. For now I use the tokenizer from openai tiktoken for reference and we will use it without knowing how this arrow \( \rightarrow \) happens and treat it as a block box for now.

[4438, 649, 358, 31370, 1403, 36295, 315, 2204, 1404, 30, 220]

Tokens 11 Characters 51

The output of the tokeinzer for our sentence is a 11 length sequence of integers given 51 characters.

I am doing a lot of handwavy assumtions that I will clarify in a moment but for now I wanna reference the most important detail of a tokenizer. The toknezier outputs a sequence of numbers that identifies each word in the vocabulary. In a high level this is all the functionality that we want from our tokenizer. Now how the tokenizer does this is far more complicated and involved and we will step through the detials and see how the process works.

A transformer computes attention using \(QK^T\).

For a standalone equation: