#60 - How to input text into your model? Understanding tokenizers.

Life with AI

English - December 01, 2022 07:20 - 14 minutes - 13.9 MB - ★★★★★ - 1 rating
Technology Homepage Download Apple Podcasts Google Podcasts Overcast Castro Pocket Casts RSS feed

Previous Episode: #59 - How to better prospect clients on LinkdIn with Guillaume from Waalaxy.

Next Episode: #61 - ChatGPT.

Hello everyone, in this episode I explain how tokenizers work. They are basically what enables us to input the text into a NLP algorithm like BERT or GPT. In the episode I explain 3 types of tokenizers, word based, character based and sub-word based representation.

Instagram: https://www.instagram.com/podcast.lifewithai/

Linkedin: https://www.linkedin.com/company/life-with-ai

Huuging Face blog about tokenizers: https://huggingface.co/docs/transformers/tokenizer_summary