I messed up so you don't: Surprisal and the headache of tokenizer encodings in LLMs!
About this talk
Abstract
What can go wrong with tokenizer encodings? Everything! I will share my experience of understanding, misunderstanding, and ultimately learning to work with tokenization in LLMs. I will discuss what surprisal is, its relevance to my research, and its connection to tokenization. The talk will include various examples illustrating how misunderstandings of tokenization can arise, as well as strategies for debugging and preventing these issues.
Prerequisites
Basic Python, and interest in NLP/LLMs
