李理的博客

Building and Optimizing a BPE Tokenizer from Scratch—Part 0: Introduction

This series of articles implements a subtask of Stanford’s CS336 Assignment 1: building an efficient training algorithm for a BPE Tokenizer. Through a series of optimizations, our algorithm’s training time on OpenWebText was reduced from over 10 hours to less than 10 minutes. This series explains these optimizations, including algorithmic improvements, data structure enhancements, parallelization with OpenMP, Cython optimization, and implementing key code in C++ along with its integration via Cython. This first article covers the task’s introduction, how to get the source code, and how to set up the development environment.

Posted by lili on September 5, 2025

Posted by lili on July 6, 2025

人工智能深度学习 chatbot PyTorch Java BERT git 编程 OCR 汪曾祺语音识别 Kaldi Linux XLNet 情感分析 sentiment analysis 语法纠错 Transformer Tensorflow Huggingface Ubuntu TensorFlow 深度学习框架 Tensor2Tensor 机器翻译微信 wechat automation selenium webdriver pywinauto CentOS GPU Appium t2t 代码阅读中英翻译公众号爬虫 ocr tesseract pytesseract python 默认参数位置参数 VPN JSON Jackson huggingface RoPE PagedAttention vLLM Pre-training LLM CPT weather forecasting graph neural networks qlora quantization transformers cmake pip pipenv conda padding vscode debug source code build deep learning Speech ASR linux pytorch extension Deep Learning DeepSeek Attention MoE cs336 bpe tokenizer

ABOUT ME

读读论文，写写代码。

FRIENDS

Li Li

李理的博客

Building and Optimizing a BPE Tokenizer from Scratch—Part 0: Introduction

动手实现和优化BPE Tokenizer的训练——第0部分：简介

模型优化

负对数似然和交叉熵

翻译：The Log-Sum-Exp Trick

翻译：DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning

翻译：YaRN: Efficient Context Window Extension of Large Language Models

翻译：DeepSeek-R1: Advancing LLM Reasoning with Reinforcement Learning

翻译：DeepSeek Explained 6: All you need to know about Reinforcement Learning in LLM training

翻译：DeepSeek Explained 5: DeepSeek-V3-Base

FEATURED TAGS

ABOUT ME

FRIENDS