{"cells":[{"cell_type":"markdown","metadata":{},"source":["

从头开始训练自己的分词器

"]},{"cell_type":"markdown","metadata":{"id":"X4cRE8IbIrIV"},"source":["

如果您在 Openbayes 上打开此笔记，您可能需要安装 🤗 Transformers 和 🤗 Datasets。请取消以下单元格的注释并执行它：

"]},{"cell_type":"code","execution_count":1,"metadata":{"colab":{"base_uri":"https://localhost:8080/","height":1000},"execution":{"iopub.execute_input":"2023-09-11T13:14:44.485090Z","iopub.status.busy":"2023-09-11T13:14:44.484796Z","iopub.status.idle":"2023-09-11T13:14:56.088986Z","shell.execute_reply":"2023-09-11T13:14:56.088054Z","shell.execute_reply.started":"2023-09-11T13:14:44.485056Z"},"id":"MOsHUjgdIrIW","outputId":"f84a093e-147f-470e-aad9-80fb51193c8e","scrolled":true,"tags":[]},"outputs":[{"name":"stdout","output_type":"stream","text":["Looking in indexes: https://pypi.tuna.tsinghua.edu.cn/simple\n","Collecting datasets\n"," Downloading https://pypi.tuna.tsinghua.edu.cn/packages/09/7e/fd4d6441a541dba61d0acb3c1fd5df53214c2e9033854e837a99dd9e0793/datasets-2.14.5-py3-none-any.whl (519 kB)\n","\u001B[2K \u001B[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001B[0m \u001B[32m519.6/519.6 kB\u001B[0m \u001B[31m34.7 MB/s\u001B[0m eta \u001B[36m0:00:00\u001B[0m\n","\u001B[?25hCollecting transformers[sentencepiece]\n"," Downloading https://pypi.tuna.tsinghua.edu.cn/packages/13/30/54b59e73400df3de506ad8630284e9fd63f4b94f735423d55fc342181037/transformers-4.33.1-py3-none-any.whl (7.6 MB)\n","\u001B[2K \u001B[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001B[0m \u001B[32m7.6/7.6 MB\u001B[0m \u001B[31m132.8 MB/s\u001B[0m eta \u001B[36m0:00:00\u001B[0ma \u001B[36m0:00:01\u001B[0m\n","\u001B[?25hRequirement already satisfied: packaging in /usr/local/lib/python3.8/site-packages (from datasets) (23.0)\n","Collecting fsspec[http]<2023.9.0,>=2023.1.0\n"," Downloading https://pypi.tuna.tsinghua.edu.cn/packages/e3/bd/4c0a4619494188a9db5d77e2100ab7d544a42e76b2447869d8e124e981d8/fsspec-2023.6.0-py3-none-any.whl (163 kB)\n","\u001B[2K \u001B[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001B[0m \u001B[32m163.8/163.8 kB\u001B[0m \u001B[31m77.7 MB/s\u001B[0m eta \u001B[36m0:00:00\u001B[0m\n","\u001B[?25hRequirement already satisfied: tqdm>=4.62.1 in /usr/local/lib/python3.8/site-packages (from datasets) (4.64.1)\n","Requirement already satisfied: pyyaml>=5.1 in /usr/local/lib/python3.8/site-packages (from datasets) (6.0)\n","Requirement already satisfied: pandas in /usr/local/lib/python3.8/site-packages (from datasets) (1.5.3)\n","Collecting dill<0.3.8,>=0.3.0\n"," Downloading https://pypi.tuna.tsinghua.edu.cn/packages/f5/3a/74a29b11cf2cdfcd6ba89c0cecd70b37cd1ba7b77978ce611eb7a146a832/dill-0.3.7-py3-none-any.whl (115 kB)\n","\u001B[2K \u001B[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001B[0m \u001B[32m115.3/115.3 kB\u001B[0m \u001B[31m63.0 MB/s\u001B[0m eta \u001B[36m0:00:00\u001B[0m\n","\u001B[?25hRequirement already satisfied: requests>=2.19.0 in /usr/local/lib/python3.8/site-packages (from datasets) (2.28.1)\n","Collecting pyarrow>=8.0.0\n"," Downloading https://pypi.tuna.tsinghua.edu.cn/packages/16/4f/1e721c6cee69b62251c996266b0cf256faad38028cb3b6d71cc86a86109e/pyarrow-13.0.0-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (40.1 MB)\n","\u001B[2K \u001B[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001B[0m \u001B[32m40.1/40.1 MB\u001B[0m \u001B[31m72.2 MB/s\u001B[0m eta \u001B[36m0:00:00\u001B[0m00:01\u001B[0m00:01\u001B[0m\n","\u001B[?25hCollecting huggingface-hub<1.0.0,>=0.14.0\n"," Downloading https://pypi.tuna.tsinghua.edu.cn/packages/2a/b3/63f423056b99df8d23c269b2df9374ea12a526e981709be84c7491c9f1cc/huggingface_hub-0.17.0-py3-none-any.whl (294 kB)\n","\u001B[2K \u001B[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001B[0m \u001B[32m294.8/294.8 kB\u001B[0m \u001B[31m82.2 MB/s\u001B[0m eta \u001B[36m0:00:00\u001B[0m\n","\u001B[?25hCollecting xxhash\n"," Downloading https://pypi.tuna.tsinghua.edu.cn/packages/b5/3e/c03b156f182702468a65adb54f64b68b1ea6953bfd2c6371b2fc9d6d2b6f/xxhash-3.3.0-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (194 kB)\n","\u001B[2K \u001B[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001B[0m \u001B[32m194.6/194.6 kB\u001B[0m \u001B[31m80.8 MB/s\u001B[0m eta \u001B[36m0:00:00\u001B[0m\n","\u001B[?25hCollecting multiprocess\n"," Downloading https://pypi.tuna.tsinghua.edu.cn/packages/c2/a6/c5cb599d917904878f220a4dbdfdcc4ef291dd3956c35b3b0dc6fc42fb6d/multiprocess-0.70.15-py38-none-any.whl (132 kB)\n","\u001B[2K \u001B[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001B[0m \u001B[32m132.6/132.6 kB\u001B[0m \u001B[31m76.3 MB/s\u001B[0m eta \u001B[36m0:00:00\u001B[0m\n","\u001B[?25hCollecting aiohttp\n"," Downloading https://pypi.tuna.tsinghua.edu.cn/packages/84/22/20c41a698eb5435ff7cd3bc6884542d2a40f3c91a221f713d77217facc39/aiohttp-3.8.5-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (1.1 MB)\n","\u001B[2K \u001B[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001B[0m \u001B[32m1.1/1.1 MB\u001B[0m \u001B[31m124.8 MB/s\u001B[0m eta \u001B[36m0:00:00\u001B[0m\n","\u001B[?25hRequirement already satisfied: numpy>=1.17 in /usr/local/lib/python3.8/site-packages (from datasets) (1.23.5)\n","Collecting tokenizers!=0.11.3,<0.14,>=0.11.1\n"," Downloading https://pypi.tuna.tsinghua.edu.cn/packages/4e/f2/017bf57106b845e31ef6179bf204042720a53629cf599ef9464da990e0e5/tokenizers-0.13.3-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (7.8 MB)\n","\u001B[2K \u001B[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001B[0m \u001B[32m7.8/7.8 MB\u001B[0m \u001B[31m137.8 MB/s\u001B[0m eta \u001B[36m0:00:00\u001B[0ma \u001B[36m0:00:01\u001B[0m\n","\u001B[?25hCollecting regex!=2019.12.17\n"," Downloading https://pypi.tuna.tsinghua.edu.cn/packages/1f/5c/374ac3fa3c7ed9a967ad273a5e841897ef6b10aa6aad938ff10717a3e2a3/regex-2023.8.8-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (774 kB)\n","\u001B[2K \u001B[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001B[0m \u001B[32m774.3/774.3 kB\u001B[0m \u001B[31m128.8 MB/s\u001B[0m eta \u001B[36m0:00:00\u001B[0m\n","\u001B[?25hCollecting safetensors>=0.3.1\n"," Downloading https://pypi.tuna.tsinghua.edu.cn/packages/21/12/d95158b4fdd0422faf019038be0be874d7bf3d9f9bd0b1b529f73853cec2/safetensors-0.3.3-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (1.3 MB)\n","\u001B[2K \u001B[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001B[0m \u001B[32m1.3/1.3 MB\u001B[0m \u001B[31m130.6 MB/s\u001B[0m eta \u001B[36m0:00:00\u001B[0m\n","\u001B[?25hCollecting filelock\n"," Downloading https://pypi.tuna.tsinghua.edu.cn/packages/52/90/45223db4e1df30ff14e8aebf9a1bf0222da2e7b49e53692c968f36817812/filelock-3.12.3-py3-none-any.whl (11 kB)\n","Collecting sentencepiece!=0.1.92,>=0.1.91\n"," Downloading https://pypi.tuna.tsinghua.edu.cn/packages/c9/58/4fbd3f33a38c9809fedf57bbef7e086b9909d6807148f35d68c0c90896d3/sentencepiece-0.1.99-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (1.3 MB)\n","\u001B[2K \u001B[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001B[0m \u001B[32m1.3/1.3 MB\u001B[0m \u001B[31m139.3 MB/s\u001B[0m eta \u001B[36m0:00:00\u001B[0m\n","\u001B[?25hRequirement already satisfied: protobuf in /usr/local/lib/python3.8/site-packages (from transformers[sentencepiece]) (3.20.3)\n","Collecting async-timeout<5.0,>=4.0.0a3\n"," Downloading https://pypi.tuna.tsinghua.edu.cn/packages/a7/fa/e01228c2938de91d47b307831c62ab9e4001e747789d0b05baf779a6488c/async_timeout-4.0.3-py3-none-any.whl (5.7 kB)\n","Collecting aiosignal>=1.1.2\n"," Downloading https://pypi.tuna.tsinghua.edu.cn/packages/76/ac/a7305707cb852b7e16ff80eaf5692309bde30e2b1100a1fcacdc8f731d97/aiosignal-1.3.1-py3-none-any.whl (7.6 kB)\n","Requirement already satisfied: charset-normalizer<4.0,>=2.0 in /usr/local/lib/python3.8/site-packages (from aiohttp->datasets) (2.0.4)\n","Collecting frozenlist>=1.1.1\n"," Downloading https://pypi.tuna.tsinghua.edu.cn/packages/0b/36/c276486f89bee9098332710af2207344f360c6c6f104a4931a7566039b1d/frozenlist-1.4.0-cp38-cp38-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl (220 kB)\n","\u001B[2K \u001B[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001B[0m \u001B[32m220.1/220.1 kB\u001B[0m \u001B[31m79.7 MB/s\u001B[0m eta \u001B[36m0:00:00\u001B[0m\n","\u001B[?25hCollecting yarl<2.0,>=1.0\n"," Downloading https://pypi.tuna.tsinghua.edu.cn/packages/c4/0c/7898c35ca4945fdd416e1dadeda985cc391e4f9298ae5e71c3a5cd88e82d/yarl-1.9.2-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (266 kB)\n","\u001B[2K \u001B[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001B[0m \u001B[32m266.9/266.9 kB\u001B[0m \u001B[31m92.1 MB/s\u001B[0m eta \u001B[36m0:00:00\u001B[0m\n","\u001B[?25hCollecting multidict<7.0,>=4.5\n"," Downloading https://pypi.tuna.tsinghua.edu.cn/packages/fe/0c/8469202f8f4b0e65816f91c3febc4bda7316c995b59ecdf3b15c574f7a24/multidict-6.0.4-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (121 kB)\n","\u001B[2K \u001B[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001B[0m \u001B[32m121.3/121.3 kB\u001B[0m \u001B[31m61.5 MB/s\u001B[0m eta \u001B[36m0:00:00\u001B[0m\n","\u001B[?25hRequirement already satisfied: attrs>=17.3.0 in /usr/local/lib/python3.8/site-packages (from aiohttp->datasets) (22.2.0)\n","Requirement already satisfied: typing-extensions>=3.7.4.3 in /usr/local/lib/python3.8/site-packages (from huggingface-hub<1.0.0,>=0.14.0->datasets) (4.5.0)\n","Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.8/site-packages (from requests>=2.19.0->datasets) (2022.12.7)\n","Requirement already satisfied: urllib3<1.27,>=1.21.1 in /usr/local/lib/python3.8/site-packages (from requests>=2.19.0->datasets) (1.26.14)\n","Requirement already satisfied: idna<4,>=2.5 in /usr/local/lib/python3.8/site-packages (from requests>=2.19.0->datasets) (3.4)\n","Collecting typing-extensions>=3.7.4.3\n"," Downloading https://pypi.tuna.tsinghua.edu.cn/packages/ec/6b/63cc3df74987c36fe26157ee12e09e8f9db4de771e0f3404263117e75b95/typing_extensions-4.7.1-py3-none-any.whl (33 kB)\n","Requirement already satisfied: python-dateutil>=2.8.1 in /usr/local/lib/python3.8/site-packages (from pandas->datasets) (2.8.2)\n","Requirement already satisfied: pytz>=2020.1 in /usr/local/lib/python3.8/site-packages (from pandas->datasets) (2022.7.1)\n","Requirement already satisfied: six>=1.5 in /usr/local/lib/python3.8/site-packages (from python-dateutil>=2.8.1->pandas->datasets) (1.16.0)\n","Installing collected packages: tokenizers, sentencepiece, safetensors, xxhash, typing-extensions, regex, pyarrow, multidict, fsspec, frozenlist, dill, async-timeout, yarl, multiprocess, filelock, aiosignal, huggingface-hub, aiohttp, transformers, datasets\n"," Attempting uninstall: typing-extensions\n"," Found existing installation: typing_extensions 4.5.0\n"," Uninstalling typing_extensions-4.5.0:\n"," Successfully uninstalled typing_extensions-4.5.0\n","Successfully installed aiohttp-3.8.5 aiosignal-1.3.1 async-timeout-4.0.3 datasets-2.14.5 dill-0.3.7 filelock-3.12.3 frozenlist-1.4.0 fsspec-2023.6.0 huggingface-hub-0.17.0 multidict-6.0.4 multiprocess-0.70.15 pyarrow-13.0.0 regex-2023.8.8 safetensors-0.3.3 sentencepiece-0.1.99 tokenizers-0.13.3 transformers-4.33.1 typing-extensions-4.7.1 xxhash-3.3.0 yarl-1.9.2\n","\u001B[33mWARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv\u001B[0m\u001B[33m\n","\u001B[0m"]}],"source":["!pip install datasets transformers[sentencepiece]"]},{"cell_type":"markdown","metadata":{"id":"HFASsisvIrIb"},"source":["

如果您正在本地打开此笔记，请确保您的环境已安装最新版本的 Datasets 和 Transformers 的源安装。

"]},{"cell_type":"code","execution_count":2,"metadata":{"execution":{"iopub.execute_input":"2023-09-11T13:14:56.091287Z","iopub.status.busy":"2023-09-11T13:14:56.091089Z","iopub.status.idle":"2023-09-11T13:14:56.787704Z","shell.execute_reply":"2023-09-11T13:14:56.787254Z","shell.execute_reply.started":"2023-09-11T13:14:56.091267Z"}},"outputs":[],"source":["from transformers.utils import send_example_telemetry\n","\n","send_example_telemetry(\"tokenizer_training_notebook\", framework=\"none\")"]},{"cell_type":"markdown","metadata":{},"source":["

在这个笔记中，我们将看到几种从头开始在给定语料库上训练自己的分词器的方法，以便您可以使用它来从头开始训练语言模型。

为什么需要训练分词器？那是因为 Transformer 模型经常使用子词分词算法，并且需要训练以识别在您使用的语料库中经常出现的单词部分。我们建议您查看 Hugging Face 课程的分词章节以获取有关分词器的一般介绍，并查看分词器总结以了解子词分词算法之间的差异。

"]},{"cell_type":"markdown","metadata":{},"source":["

获取语料库

"]},{"cell_type":"markdown","metadata":{},"source":["

我们需要文本来训练我们的分词器。我们将使用 🤗 Datasets 库来下载我们的文本数据，可以使用load_dataset函数轻松完成：

"]},{"cell_type":"code","execution_count":3,"metadata":{"execution":{"iopub.execute_input":"2023-09-11T13:14:56.788643Z","iopub.status.busy":"2023-09-11T13:14:56.788463Z","iopub.status.idle":"2023-09-11T13:14:57.307175Z","shell.execute_reply":"2023-09-11T13:14:57.306381Z","shell.execute_reply.started":"2023-09-11T13:14:56.788627Z"}},"outputs":[],"source":["from datasets import load_dataset"]},{"cell_type":"markdown","metadata":{},"source":["

对于这个例子，我们将使用 Wikitext-2（其中包含 4.5MB 的文本，因此我们的示例训练速度很快），但您可以使用任何您想要的数据集（以及任何语言，除了英语）。

"]},{"cell_type":"code","execution_count":4,"metadata":{"execution":{"iopub.execute_input":"2023-09-11T13:14:57.308242Z","iopub.status.busy":"2023-09-11T13:14:57.308000Z","iopub.status.idle":"2023-09-11T13:15:00.506548Z","shell.execute_reply":"2023-09-11T13:15:00.506094Z","shell.execute_reply.started":"2023-09-11T13:14:57.308223Z"}},"outputs":[],"source":["dataset = load_dataset(\"wikitext\", name=\"wikitext-2-raw-v1\", split=\"train\")"]},{"cell_type":"markdown","metadata":{},"source":["

我们可以查看数据集，其中包含 36,718 个文本：

"]},{"cell_type":"code","execution_count":5,"metadata":{"execution":{"iopub.execute_input":"2023-09-11T13:15:00.507433Z","iopub.status.busy":"2023-09-11T13:15:00.507184Z","iopub.status.idle":"2023-09-11T13:15:00.512228Z","shell.execute_reply":"2023-09-11T13:15:00.511852Z","shell.execute_reply.started":"2023-09-11T13:15:00.507416Z"}},"outputs":[{"data":{"text/plain":["Dataset({\n"," features: ['text'],\n"," num_rows: 36718\n","})"]},"execution_count":5,"metadata":{},"output_type":"execute_result"}],"source":["dataset"]},{"cell_type":"markdown","metadata":{},"source":["

要访问一个元素，我们只需要提供它的索引：

"]},{"cell_type":"code","execution_count":6,"metadata":{"execution":{"iopub.execute_input":"2023-09-11T13:15:00.513078Z","iopub.status.busy":"2023-09-11T13:15:00.512752Z","iopub.status.idle":"2023-09-11T13:15:00.516016Z","shell.execute_reply":"2023-09-11T13:15:00.515649Z","shell.execute_reply.started":"2023-09-11T13:15:00.513062Z"}},"outputs":[{"data":{"text/plain":["{'text': ' = Valkyria Chronicles III = \\n'}"]},"execution_count":6,"metadata":{},"output_type":"execute_result"}],"source":["dataset[1]"]},{"cell_type":"markdown","metadata":{},"source":["

我们也可以直接访问一个切片，这样我们就会得到一个字典，其中键为「text」，值为文本列表：

"]},{"cell_type":"code","execution_count":7,"metadata":{"execution":{"iopub.execute_input":"2023-09-11T13:15:00.517920Z","iopub.status.busy":"2023-09-11T13:15:00.517615Z","iopub.status.idle":"2023-09-11T13:15:00.520752Z","shell.execute_reply":"2023-09-11T13:15:00.520382Z","shell.execute_reply.started":"2023-09-11T13:15:00.517892Z"}},"outputs":[{"data":{"text/plain":["{'text': ['',\n"," ' = Valkyria Chronicles III = \\n',\n"," '',\n"," ' Senjō no Valkyria 3 : Unrecorded Chronicles ( Japanese : 戦場のヴァルキュリア3 , lit . Valkyria of the Battlefield 3 ) , commonly referred to as Valkyria Chronicles III outside Japan , is a tactical role @-@ playing video game developed by Sega and Media.Vision for the PlayStation Portable . Released in January 2011 in Japan , it is the third game in the Valkyria series . Employing the same fusion of tactical and real @-@ time gameplay as its predecessors , the story runs parallel to the first game and follows the \" Nameless \" , a penal military unit serving the nation of Gallia during the Second Europan War who perform secret black operations and are pitted against the Imperial unit \" Calamaty Raven \" . \\n',\n"," \" The game began development in 2010 , carrying over a large portion of the work done on Valkyria Chronicles II . While it retained the standard features of the series , it also underwent multiple adjustments , such as making the game more forgiving for series newcomers . Character designer Raita Honjou and composer Hitoshi Sakimoto both returned from previous entries , along with Valkyria Chronicles II director Takeshi Ozawa . A large team of writers handled the script . The game 's opening theme was sung by May 'n . \\n\"]}"]},"execution_count":7,"metadata":{},"output_type":"execute_result"}],"source":["dataset[:5]"]},{"cell_type":"markdown","metadata":{},"source":["

训练我们的分词器所需的 API 将需要一批文本的迭代器，例如文本列表的列表：

"]},{"cell_type":"code","execution_count":8,"metadata":{"execution":{"iopub.execute_input":"2023-09-11T13:15:00.521370Z","iopub.status.busy":"2023-09-11T13:15:00.521222Z","iopub.status.idle":"2023-09-11T13:15:00.569332Z","shell.execute_reply":"2023-09-11T13:15:00.568935Z","shell.execute_reply.started":"2023-09-11T13:15:00.521355Z"}},"outputs":[],"source":["batch_size = 1000\n","all_texts = [dataset[i : i + batch_size][\"text\"] for i in range(0, len(dataset), batch_size)]"]},{"cell_type":"markdown","metadata":{},"source":["

为了避免将所有内容加载到内存中（因为 Datasets 库将元素保留在磁盘上，只有在请求时才将它们加载到内存中），我们定义一个 Python 迭代器。如果您有一个巨大的数据集，这将特别有用：

"]},{"cell_type":"code","execution_count":9,"metadata":{"execution":{"iopub.execute_input":"2023-09-11T13:15:00.570270Z","iopub.status.busy":"2023-09-11T13:15:00.569864Z","iopub.status.idle":"2023-09-11T13:15:00.572530Z","shell.execute_reply":"2023-09-11T13:15:00.572156Z","shell.execute_reply.started":"2023-09-11T13:15:00.570254Z"}},"outputs":[],"source":["def batch_iterator():\n"," for i in range(0, len(dataset), batch_size):\n"," yield dataset[i : i + batch_size][\"text\"]"]},{"cell_type":"markdown","metadata":{},"source":["

现在让我们看看如何使用这个语料库来训练一个新的分词器！有两个 API 可以做到这一点：第一个使用现有的分词器，在一行代码中训练一个新版本的分词器，第二个是实际上逐步构建您的分词器，因此可以自定义每个步骤！

"]},{"cell_type":"markdown","metadata":{},"source":["

使用现有的分词器

"]},{"cell_type":"markdown","metadata":{},"source":["

如果您想使用与现有标记化器完全相同的算法和参数来训练标记化器，您可以使用 train_new_from_iterator API。例如，让我们使用相同的标记化算法在 Wikitext-2 上训练 GPT-2 标记化器的新版本。

首先，我们需要将要使用的标记化器作为模型加载：

"]},{"cell_type":"code","execution_count":10,"metadata":{"execution":{"iopub.execute_input":"2023-09-11T13:15:00.573352Z","iopub.status.busy":"2023-09-11T13:15:00.573038Z","iopub.status.idle":"2023-09-11T13:15:01.832793Z","shell.execute_reply":"2023-09-11T13:15:01.831933Z","shell.execute_reply.started":"2023-09-11T13:15:00.573337Z"}},"outputs":[],"source":["from transformers import AutoTokenizer\n","\n","tokenizer = AutoTokenizer.from_pretrained(\"gpt2\")"]},{"cell_type":"markdown","metadata":{},"source":["

请确保您选择的分词器是快速版本（由 🤗 Tokenizers 库支持），否则笔记本的其余部分将无法运行：

"]},{"cell_type":"code","execution_count":11,"metadata":{"execution":{"iopub.execute_input":"2023-09-11T13:15:01.834050Z","iopub.status.busy":"2023-09-11T13:15:01.833775Z","iopub.status.idle":"2023-09-11T13:15:01.837675Z","shell.execute_reply":"2023-09-11T13:15:01.837280Z","shell.execute_reply.started":"2023-09-11T13:15:01.834023Z"}},"outputs":[{"data":{"text/plain":["True"]},"execution_count":11,"metadata":{},"output_type":"execute_result"}],"source":["tokenizer.is_fast"]},{"cell_type":"markdown","metadata":{},"source":["

然后我们将训练语料库（无论是之前定义的列表还是迭代器）提供给train_new_from_iterator方法。我们还需要指定要使用的词汇表大小：

"]},{"cell_type":"code","execution_count":12,"metadata":{"execution":{"iopub.execute_input":"2023-09-11T13:15:01.838451Z","iopub.status.busy":"2023-09-11T13:15:01.838199Z","iopub.status.idle":"2023-09-11T13:15:03.525847Z","shell.execute_reply":"2023-09-11T13:15:03.525234Z","shell.execute_reply.started":"2023-09-11T13:15:01.838434Z"}},"outputs":[{"name":"stdout","output_type":"stream","text":["\n","\n","\n"]}],"source":["new_tokenizer = tokenizer.train_new_from_iterator(batch_iterator(), vocab_size=25000)"]},{"cell_type":"markdown","metadata":{},"source":["

这就是全部内容了！由 Rust 支持的 🤗 Tokenizers 库使得训练非常快速。

现在你已经有了一个新的分词器，可以预处理你的数据并训练语言模型。你可以像往常一样输入文本。

"]},{"cell_type":"code","execution_count":13,"metadata":{"execution":{"iopub.execute_input":"2023-09-11T13:15:03.526948Z","iopub.status.busy":"2023-09-11T13:15:03.526762Z","iopub.status.idle":"2023-09-11T13:15:03.532052Z","shell.execute_reply":"2023-09-11T13:15:03.531662Z","shell.execute_reply.started":"2023-09-11T13:15:03.526930Z"}},"outputs":[{"data":{"text/plain":["{'input_ids': [[], [301, 8639, 9504, 3050, 301, 315], [], [4720, 74, 4825, 889, 8639, 491, 529, 672, 6944, 475, 267, 9504, 374, 2809, 529, 10879, 231, 100, 162, 255, 113, 14391, 4046, 113, 4509, 95, 18351, 4509, 256, 4046, 99, 4046, 22234, 96, 19, 264, 6437, 272, 8639, 281, 261, 3518, 2035, 491, 373, 264, 5162, 3305, 290, 344, 8639, 9504, 3050, 2616, 1822, 264, 364, 259, 14059, 1559, 340, 2393, 1527, 737, 1961, 370, 805, 3604, 288, 7577, 14, 54, 782, 337, 261, 4840, 15585, 272, 19958, 284, 1404, 1696, 284, 1822, 264, 385, 364, 261, 1431, 737, 284, 261, 8639, 906, 272, 2531, 1858, 286, 261, 1112, 9658, 281, 14059, 288, 1626, 340, 645, 6556, 344, 520, 14434, 264, 261, 1485, 3436, 7515, 290, 261, 518, 737, 288, 4750, 261, 302, 22039, 302, 264, 259, 21720, 1743, 3836, 5654, 261, 4259, 281, 4742, 490, 724, 261, 3581, 1351, 283, 1114, 579, 952, 4010, 1985, 2563, 288, 453, 2128, 807, 935, 261, 7655, 3836, 302, 2038, 314, 271, 89, 22414, 302, 272, 315], [324, 737, 1022, 1984, 284, 1525, 264, 7663, 610, 259, 1241, 4816, 281, 261, 693, 3654, 326, 8639, 9504, 1243, 272, 1894, 385, 7631, 261, 3684, 2303, 281, 261, 906, 264, 385, 534, 9638, 5354, 16654, 1030, 264, 844, 344, 1878, 261, 737, 667, 10407, 1315, 337, 906, 727, 3210, 383, 272, 13353, 8814, 8187, 2591, 6086, 74, 298, 288, 7508, 10103, 17447, 304, 11550, 9013, 920, 1898, 403, 1445, 22645, 264, 1071, 359, 8639, 9504, 1243, 2499, 21197, 5400, 19526, 5224, 272, 303, 1241, 990, 281, 3839, 8713, 261, 3418, 272, 324, 737, 331, 83, 2574, 3535, 321, 8351, 370, 1073, 331, 78, 272, 315]], 'attention_mask': [[], [1, 1, 1, 1, 1, 1], [], [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1], [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]]}"]},"execution_count":13,"metadata":{},"output_type":"execute_result"}],"source":["new_tokenizer(dataset[:5][\"text\"])"]},{"cell_type":"markdown","metadata":{},"source":["

你可以使用 save_pretrained 方法将其保存在本地：

"]},{"cell_type":"code","execution_count":14,"metadata":{"execution":{"iopub.execute_input":"2023-09-11T13:15:03.532754Z","iopub.status.busy":"2023-09-11T13:15:03.532602Z","iopub.status.idle":"2023-09-11T13:15:03.568388Z","shell.execute_reply":"2023-09-11T13:15:03.568004Z","shell.execute_reply.started":"2023-09-11T13:15:03.532739Z"}},"outputs":[{"data":{"text/plain":["('my-new-tokenizer/tokenizer_config.json',\n"," 'my-new-tokenizer/special_tokens_map.json',\n"," 'my-new-tokenizer/vocab.json',\n"," 'my-new-tokenizer/merges.txt',\n"," 'my-new-tokenizer/added_tokens.json',\n"," 'my-new-tokenizer/tokenizer.json')"]},"execution_count":14,"metadata":{},"output_type":"execute_result"}],"source":["new_tokenizer.save_pretrained(\"my-new-tokenizer\")"]},{"cell_type":"markdown","metadata":{},"source":["

这台机器现在可以重新加载分词器：

"]},{"cell_type":"code","execution_count":18,"metadata":{"execution":{"iopub.execute_input":"2023-09-11T13:15:03.577523Z","iopub.status.busy":"2023-09-11T13:15:03.577181Z","iopub.status.idle":"2023-09-11T13:15:03.601174Z","shell.execute_reply":"2023-09-11T13:15:03.600787Z","shell.execute_reply.started":"2023-09-11T13:15:03.577507Z"}},"outputs":[],"source":["tok = new_tokenizer.from_pretrained(\"my-new-tokenizer\")"]},{"cell_type":"markdown","metadata":{},"source":["

现在，如果您想创建和训练一个全新的分词器，它与现有的任何分词器都不相似，您需要使用🤗 Tokenizers 库从头开始构建它。

"]},{"cell_type":"markdown","metadata":{},"source":["

从头开始构建您的分词器

"]},{"cell_type":"markdown","metadata":{},"source":["

为了了解如何从头开始构建您的分词器，我们需要深入了解 🤗 Tokenizers 库和分词流程。该流程包括以下几个步骤：

规范化：对初始输入字符串执行所有初始转换。例如，当您需要将一些文本转换为小写时，可能会剥离它，甚至应用常见的 Unicode 规范化过程之一，您将添加一个规范化器。
预分词：负责拆分初始输入字符串。这是决定在哪里以及如何预分段原始字符串的组件。最简单的例子是简单地按空格拆分。
模型：处理所有子标记的发现和生成，这是可训练的部分，真正依赖于您的输入数据。
后处理：提供高级构造功能，以与一些基于 Transformers 的 SoTA 模型兼容。例如，对于 BERT，它将在 [CLS] 和 [SEP] 标记周围包装标记化的句子。

而要朝另一个方向前进：

解码：负责将标记化的输入映射回原始字符串。解码器通常根据我们之前使用的PreTokenizer进行选择。

对于模型的训练，🤗 Tokenizers 库提供了一个Trainer类，我们将使用它。

所有这些构建块都可以组合在一起创建工作的分词流程。为了给您一些例子，我们将在这里展示三个完整的流程：如何复制 GPT-2、BERT 和 T5（这将为您提供 BPE、WordPiece 和 Unigram 分词器的示例）。

"]},{"cell_type":"markdown","metadata":{},"source":["

像BERT一样的WordPiece模型

"]},{"cell_type":"markdown","metadata":{},"source":["

让我们看看如何创建一个类似于用于训练 BERT 的 WordPiece 分词器。第一步是创建一个带有空的 WordPiece 模型的Tokenizer：

"]},{"cell_type":"code","execution_count":19,"metadata":{"execution":{"iopub.execute_input":"2023-09-11T13:15:03.602093Z","iopub.status.busy":"2023-09-11T13:15:03.601710Z","iopub.status.idle":"2023-09-11T13:15:03.611243Z","shell.execute_reply":"2023-09-11T13:15:03.610846Z","shell.execute_reply.started":"2023-09-11T13:15:03.602077Z"}},"outputs":[{"name":"stdout","output_type":"stream","text":["Ignored unknown kwargs option unl_token\n"]}],"source":["from tokenizers import decoders, models, normalizers, pre_tokenizers, processors, trainers, Tokenizer\n","\n","tokenizer = Tokenizer(models.WordPiece(unl_token=\"[UNK]\"))"]},{"cell_type":"markdown","metadata":{},"source":["

这个 tokenizer 还没有准备好进行训练。我们需要添加一些预处理步骤：规范化（可选）和预分词器，它将把输入拆分成我们称之为单词的块。然后，令牌将成为这些单词的一部分（但不能比它们更大）。

在 BERT 的情况下，规范化是小写。由于 BERT 是如此流行的模型，它有自己的规范化器：

"]},{"cell_type":"code","execution_count":20,"metadata":{"execution":{"iopub.execute_input":"2023-09-11T13:15:03.611975Z","iopub.status.busy":"2023-09-11T13:15:03.611826Z","iopub.status.idle":"2023-09-11T13:15:03.614263Z","shell.execute_reply":"2023-09-11T13:15:03.613859Z","shell.execute_reply.started":"2023-09-11T13:15:03.611961Z"}},"outputs":[],"source":["tokenizer.normalizer = normalizers.BertNormalizer(lowercase=True)"]},{"cell_type":"markdown","metadata":{},"source":["

如果您想自定义它，您可以使用现有的块并按顺序组合它们：例如在这里我们将其转换为小写，应用 NFD 规范化并去除重音符号：

"]},{"cell_type":"code","execution_count":21,"metadata":{"execution":{"iopub.execute_input":"2023-09-11T13:15:03.615101Z","iopub.status.busy":"2023-09-11T13:15:03.614773Z","iopub.status.idle":"2023-09-11T13:15:03.617202Z","shell.execute_reply":"2023-09-11T13:15:03.616830Z","shell.execute_reply.started":"2023-09-11T13:15:03.615085Z"}},"outputs":[],"source":["tokenizer.normalizer = normalizers.Sequence(\n"," [normalizers.NFD(), normalizers.Lowercase(), normalizers.StripAccents()]\n",")"]},{"cell_type":"markdown","metadata":{},"source":["

还有一个BertPreTokenizer，我们可以直接使用它。它使用空格和标点进行预分词：

"]},{"cell_type":"code","execution_count":22,"metadata":{"execution":{"iopub.execute_input":"2023-09-11T13:15:03.620229Z","iopub.status.busy":"2023-09-11T13:15:03.619926Z","iopub.status.idle":"2023-09-11T13:15:03.622294Z","shell.execute_reply":"2023-09-11T13:15:03.621896Z","shell.execute_reply.started":"2023-09-11T13:15:03.620213Z"}},"outputs":[],"source":["tokenizer.pre_tokenizer = pre_tokenizers.BertPreTokenizer()"]},{"cell_type":"markdown","metadata":{},"source":["

与规范化器一样，我们可以在「Sequence」中组合几个预分词器。如果我们想快速查看它如何预处理输入，我们可以调用「pre_tokenize_str」方法：

"]},{"cell_type":"code","execution_count":23,"metadata":{"execution":{"iopub.execute_input":"2023-09-11T13:15:03.623056Z","iopub.status.busy":"2023-09-11T13:15:03.622790Z","iopub.status.idle":"2023-09-11T13:15:03.626251Z","shell.execute_reply":"2023-09-11T13:15:03.625830Z","shell.execute_reply.started":"2023-09-11T13:15:03.623039Z"}},"outputs":[{"data":{"text/plain":["[('This', (0, 4)),\n"," ('is', (5, 7)),\n"," ('an', (8, 10)),\n"," ('example', (11, 18)),\n"," ('!', (18, 19))]"]},"execution_count":23,"metadata":{},"output_type":"execute_result"}],"source":["tokenizer.pre_tokenizer.pre_tokenize_str(\"This is an example!\")"]},{"cell_type":"markdown","metadata":{},"source":["

请注意，预分词器不仅将文本分成单词，还保留了偏移量，即这些单词在原始文本中的开头和结尾。这将允许最终的分词器能够将每个标记与其来自的文本部分匹配（这是我们用于问答或标记分类任务的功能）。

现在我们可以训练我们的分词器（管道还没有完全完成，但我们需要一个训练有素的分词器来构建后处理器），我们使用WordPieceTrainer进行训练。记住要将特殊标记传递给训练器，因为它们不会在语料库中出现。

"]},{"cell_type":"code","execution_count":24,"metadata":{"execution":{"iopub.execute_input":"2023-09-11T13:15:03.626909Z","iopub.status.busy":"2023-09-11T13:15:03.626762Z","iopub.status.idle":"2023-09-11T13:15:03.629298Z","shell.execute_reply":"2023-09-11T13:15:03.628907Z","shell.execute_reply.started":"2023-09-11T13:15:03.626894Z"}},"outputs":[],"source":["special_tokens = [\"[UNK]\", \"[PAD]\", \"[CLS]\", \"[SEP]\", \"[MASK]\"]\n","trainer = trainers.WordPieceTrainer(vocab_size=25000, special_tokens=special_tokens)"]},{"cell_type":"markdown","metadata":{},"source":["

要实际训练分词器，方法与之前使用的相同：我们可以传递一些文本文件，或者是一批文本的迭代器：

"]},{"cell_type":"code","execution_count":25,"metadata":{"execution":{"iopub.execute_input":"2023-09-11T13:15:03.630088Z","iopub.status.busy":"2023-09-11T13:15:03.629799Z","iopub.status.idle":"2023-09-11T13:15:04.858855Z","shell.execute_reply":"2023-09-11T13:15:04.858369Z","shell.execute_reply.started":"2023-09-11T13:15:03.630071Z"}},"outputs":[{"name":"stdout","output_type":"stream","text":["\n","\n","\n"]}],"source":["tokenizer.train_from_iterator(batch_iterator(), trainer=trainer)"]},{"cell_type":"markdown","metadata":{},"source":["

现在我们已经训练好了分词器，我们可以定义后处理器：我们需要在开头添加 CLS 标记，在结尾添加 SEP 标记（对于单个句子）或多个 SEP 标记（对于句子对）。我们使用TemplateProcessing来完成这个任务，这需要知道 CLS 和 SEP 标记的 ID（这就是为什么我们要等到训练完成的原因）。

所以让我们首先获取这两个特殊标记的 ID：

"]},{"cell_type":"code","execution_count":26,"metadata":{"execution":{"iopub.execute_input":"2023-09-11T13:15:04.859849Z","iopub.status.busy":"2023-09-11T13:15:04.859524Z","iopub.status.idle":"2023-09-11T13:15:04.862699Z","shell.execute_reply":"2023-09-11T13:15:04.862289Z","shell.execute_reply.started":"2023-09-11T13:15:04.859832Z"}},"outputs":[{"name":"stdout","output_type":"stream","text":["2 3\n"]}],"source":["cls_token_id = tokenizer.token_to_id(\"[CLS]\")\n","sep_token_id = tokenizer.token_to_id(\"[SEP]\")\n","print(cls_token_id, sep_token_id)"]},{"cell_type":"markdown","metadata":{},"source":["

以下是我们如何构建后处理器的方法。我们必须在模板中指示如何使用一句话（$A）或两句话（$A和$B）来组织特殊标记。冒号后面跟着一个数字表示要为每个部分分配的标记类型 ID。

"]},{"cell_type":"code","execution_count":27,"metadata":{"execution":{"iopub.execute_input":"2023-09-11T13:15:04.863532Z","iopub.status.busy":"2023-09-11T13:15:04.863259Z","iopub.status.idle":"2023-09-11T13:15:04.865996Z","shell.execute_reply":"2023-09-11T13:15:04.865563Z","shell.execute_reply.started":"2023-09-11T13:15:04.863516Z"}},"outputs":[],"source":["tokenizer.post_processor = processors.TemplateProcessing(\n"," single=f\"[CLS]:0 $A:0 [SEP]:0\",\n"," pair=f\"[CLS]:0 $A:0 [SEP]:0 $B:1 [SEP]:1\",\n"," special_tokens=[\n"," (\"[CLS]\", cls_token_id),\n"," (\"[SEP]\", sep_token_id),\n"," ],\n",")"]},{"cell_type":"markdown","metadata":{},"source":["

我们可以通过编码一对句子来检查是否获得了预期的结果，例如：

"]},{"cell_type":"code","execution_count":28,"metadata":{"execution":{"iopub.execute_input":"2023-09-11T13:15:04.866777Z","iopub.status.busy":"2023-09-11T13:15:04.866521Z","iopub.status.idle":"2023-09-11T13:15:04.868932Z","shell.execute_reply":"2023-09-11T13:15:04.868550Z","shell.execute_reply.started":"2023-09-11T13:15:04.866762Z"}},"outputs":[],"source":["encoding = tokenizer.encode(\"This is one sentence.\", \"With this one we have a pair.\")"]},{"cell_type":"markdown","metadata":{},"source":["

我们可以查看标记来检查特殊标记是否已经插入到正确的位置：

"]},{"cell_type":"code","execution_count":29,"metadata":{"execution":{"iopub.execute_input":"2023-09-11T13:15:04.869595Z","iopub.status.busy":"2023-09-11T13:15:04.869427Z","iopub.status.idle":"2023-09-11T13:15:04.872776Z","shell.execute_reply":"2023-09-11T13:15:04.872397Z","shell.execute_reply.started":"2023-09-11T13:15:04.869580Z"}},"outputs":[{"data":{"text/plain":["['[CLS]',\n"," 'this',\n"," 'is',\n"," 'one',\n"," 'sentence',\n"," '.',\n"," '[SEP]',\n"," 'with',\n"," 'this',\n"," 'one',\n"," 'we',\n"," 'have',\n"," 'a',\n"," 'pair',\n"," '.',\n"," '[SEP]']"]},"execution_count":29,"metadata":{},"output_type":"execute_result"}],"source":["encoding.tokens"]},{"cell_type":"markdown","metadata":{},"source":["

我们可以检查令牌类型 ID 是否正确：

"]},{"cell_type":"code","execution_count":30,"metadata":{"execution":{"iopub.execute_input":"2023-09-11T13:15:04.873439Z","iopub.status.busy":"2023-09-11T13:15:04.873266Z","iopub.status.idle":"2023-09-11T13:15:04.876287Z","shell.execute_reply":"2023-09-11T13:15:04.875922Z","shell.execute_reply.started":"2023-09-11T13:15:04.873424Z"}},"outputs":[{"data":{"text/plain":["[0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1]"]},"execution_count":30,"metadata":{},"output_type":"execute_result"}],"source":["encoding.type_ids"]},{"cell_type":"markdown","metadata":{},"source":["

这个分词器中的最后一部分是解码器，我们使用 WordPiece 解码器，并指定特殊前缀 ##：

"]},{"cell_type":"code","execution_count":31,"metadata":{"execution":{"iopub.execute_input":"2023-09-11T13:15:04.876952Z","iopub.status.busy":"2023-09-11T13:15:04.876780Z","iopub.status.idle":"2023-09-11T13:15:04.879042Z","shell.execute_reply":"2023-09-11T13:15:04.878669Z","shell.execute_reply.started":"2023-09-11T13:15:04.876937Z"}},"outputs":[],"source":["tokenizer.decoder = decoders.WordPiece(prefix=\"##\")"]},{"cell_type":"markdown","metadata":{},"source":["

现在我们的分词器已经完成，我们需要将其包装在一个 Transformers 对象中，以便能够在 Transformers 库中使用它。更具体地说，我们必须将其放在与我们想要使用的模型相对应的 tokenizer fast 类中，这里是BertTokenizerFast。

"]},{"cell_type":"code","execution_count":32,"metadata":{"execution":{"iopub.execute_input":"2023-09-11T13:15:04.879775Z","iopub.status.busy":"2023-09-11T13:15:04.879539Z","iopub.status.idle":"2023-09-11T13:15:04.895006Z","shell.execute_reply":"2023-09-11T13:15:04.894617Z","shell.execute_reply.started":"2023-09-11T13:15:04.879760Z"}},"outputs":[],"source":["from transformers import BertTokenizerFast\n","\n","new_tokenizer = BertTokenizerFast(tokenizer_object=tokenizer)"]},{"cell_type":"markdown","metadata":{},"source":["

和之前一样，我们可以将这个分词器作为普通的 Transformers 分词器使用，并使用 save_pretrained 或 push_to_hub 方法。

如果您正在构建的分词器与 Transformers 中的任何类都不匹配，因为它非常特殊，您可以将其包装在 PreTrainedTokenizerFast 中。

"]},{"cell_type":"markdown","metadata":{},"source":["

BPE 模型类似于 GPT-2

"]},{"cell_type":"markdown","metadata":{},"source":["

现在让我们来看看如何创建一个像用于训练 GPT-2 的 BPE 分词器。第一步是创建一个带有空 BPE 模型的Tokenizer：

"]},{"cell_type":"code","execution_count":33,"metadata":{"execution":{"iopub.execute_input":"2023-09-11T13:15:04.895809Z","iopub.status.busy":"2023-09-11T13:15:04.895632Z","iopub.status.idle":"2023-09-11T13:15:04.899536Z","shell.execute_reply":"2023-09-11T13:15:04.899158Z","shell.execute_reply.started":"2023-09-11T13:15:04.895794Z"}},"outputs":[],"source":["tokenizer = Tokenizer(models.BPE())"]},{"cell_type":"markdown","metadata":{},"source":["

与之前一样，我们需要添加可选的规范化（在 GPT-2 的情况下未使用），并且在训练之前需要指定一个预分词器。在 GPT-2 的情况下，使用的预分词器是字节级预分词器：

"]},{"cell_type":"code","execution_count":34,"metadata":{"execution":{"iopub.execute_input":"2023-09-11T13:15:04.900536Z","iopub.status.busy":"2023-09-11T13:15:04.900363Z","iopub.status.idle":"2023-09-11T13:15:04.902607Z","shell.execute_reply":"2023-09-11T13:15:04.902234Z","shell.execute_reply.started":"2023-09-11T13:15:04.900521Z"}},"outputs":[],"source":["tokenizer.pre_tokenizer = pre_tokenizers.ByteLevel(add_prefix_space=False)"]},{"cell_type":"markdown","metadata":{},"source":["

如果我们想快速查看它如何预处理输入，我们可以调用 pre_tokenize_str 方法：

"]},{"cell_type":"code","execution_count":35,"metadata":{"execution":{"iopub.execute_input":"2023-09-11T13:15:04.903322Z","iopub.status.busy":"2023-09-11T13:15:04.903092Z","iopub.status.idle":"2023-09-11T13:15:04.906295Z","shell.execute_reply":"2023-09-11T13:15:04.905932Z","shell.execute_reply.started":"2023-09-11T13:15:04.903308Z"}},"outputs":[{"data":{"text/plain":["[('This', (0, 4)),\n"," ('Ġis', (4, 7)),\n"," ('Ġan', (7, 10)),\n"," ('Ġexample', (10, 18)),\n"," ('!', (18, 19))]"]},"execution_count":35,"metadata":{},"output_type":"execute_result"}],"source":["tokenizer.pre_tokenizer.pre_tokenize_str(\"This is an example!\")"]},{"cell_type":"markdown","metadata":{},"source":["

我们使用了与 GPT-2 相同的默认前缀空格，因此您可以看到每个单词在开头都会添加一个初始的「'Ġ'」，除了第一个单词。

现在我们可以训练我们的分词器！这次我们使用BpeTrainer。

"]},{"cell_type":"code","execution_count":36,"metadata":{"execution":{"iopub.execute_input":"2023-09-11T13:15:04.907019Z","iopub.status.busy":"2023-09-11T13:15:04.906787Z","iopub.status.idle":"2023-09-11T13:15:06.292556Z","shell.execute_reply":"2023-09-11T13:15:06.292027Z","shell.execute_reply.started":"2023-09-11T13:15:04.907005Z"}},"outputs":[{"name":"stdout","output_type":"stream","text":["\n","\n","\n"]}],"source":["trainer = trainers.BpeTrainer(vocab_size=25000, special_tokens=[\"<|endoftext|>\"])\n","tokenizer.train_from_iterator(batch_iterator(), trainer=trainer)"]},{"cell_type":"markdown","metadata":{},"source":["

为了完成整个流程，我们必须包括后处理器和解码器：

"]},{"cell_type":"code","execution_count":37,"metadata":{"execution":{"iopub.execute_input":"2023-09-11T13:15:06.294074Z","iopub.status.busy":"2023-09-11T13:15:06.293740Z","iopub.status.idle":"2023-09-11T13:15:06.296620Z","shell.execute_reply":"2023-09-11T13:15:06.296179Z","shell.execute_reply.started":"2023-09-11T13:15:06.294056Z"}},"outputs":[],"source":["tokenizer.post_processor = processors.ByteLevel(trim_offsets=False)\n","tokenizer.decoder = decoders.ByteLevel()"]},{"cell_type":"markdown","metadata":{},"source":["

和之前一样，我们最后将其包装在一个 Transformers tokenizer 对象中：

"]},{"cell_type":"code","execution_count":38,"metadata":{"execution":{"iopub.execute_input":"2023-09-11T13:15:06.297340Z","iopub.status.busy":"2023-09-11T13:15:06.297154Z","iopub.status.idle":"2023-09-11T13:15:06.331285Z","shell.execute_reply":"2023-09-11T13:15:06.330881Z","shell.execute_reply.started":"2023-09-11T13:15:06.297324Z"}},"outputs":[],"source":["from transformers import GPT2TokenizerFast\n","\n","new_tokenizer = GPT2TokenizerFast(tokenizer_object=tokenizer)"]},{"cell_type":"markdown","metadata":{},"source":["

像Albert一样的Unigram模型

"]},{"cell_type":"markdown","metadata":{},"source":["

现在让我们看一下如何创建一个像T5训练中使用的 Unigram tokenizer。第一步是创建一个带有空的 Unigram 模型的 Tokenizer：

"]},{"cell_type":"code","execution_count":39,"metadata":{"execution":{"iopub.execute_input":"2023-09-11T13:15:06.332530Z","iopub.status.busy":"2023-09-11T13:15:06.332343Z","iopub.status.idle":"2023-09-11T13:15:06.336738Z","shell.execute_reply":"2023-09-11T13:15:06.336348Z","shell.execute_reply.started":"2023-09-11T13:15:06.332515Z"}},"outputs":[],"source":["tokenizer = Tokenizer(models.Unigram())"]},{"cell_type":"markdown","metadata":{},"source":["

与之前一样，我们需要添加可选的规范化（这里是一些替换和小写），并在训练之前需要指定一个预分词器。使用的预分词器是 Metaspace 预分词器：它将所有空格替换为特殊字符（默认为 ▁），然后在该字符上进行分割。

"]},{"cell_type":"code","execution_count":40,"metadata":{"execution":{"iopub.execute_input":"2023-09-11T13:15:06.337999Z","iopub.status.busy":"2023-09-11T13:15:06.337816Z","iopub.status.idle":"2023-09-11T13:15:06.340458Z","shell.execute_reply":"2023-09-11T13:15:06.340072Z","shell.execute_reply.started":"2023-09-11T13:15:06.337983Z"}},"outputs":[],"source":["tokenizer.normalizer = normalizers.Sequence(\n"," [normalizers.Replace(\"``\", '\"'), normalizers.Replace(\"''\", '\"'), normalizers.Lowercase()]\n",")\n","tokenizer.pre_tokenizer = pre_tokenizers.Metaspace()"]},{"cell_type":"markdown","metadata":{},"source":["

如果我们想快速查看它如何预处理输入，我们可以调用 pre_tokenize_str 方法：

"]},{"cell_type":"code","execution_count":41,"metadata":{"execution":{"iopub.execute_input":"2023-09-11T13:15:06.341143Z","iopub.status.busy":"2023-09-11T13:15:06.340967Z","iopub.status.idle":"2023-09-11T13:15:06.344298Z","shell.execute_reply":"2023-09-11T13:15:06.343922Z","shell.execute_reply.started":"2023-09-11T13:15:06.341128Z"}},"outputs":[{"data":{"text/plain":["[('▁This', (0, 4)), ('▁is', (4, 7)), ('▁an', (7, 10)), ('▁example!', (10, 19))]"]},"execution_count":41,"metadata":{},"output_type":"execute_result"}],"source":["tokenizer.pre_tokenizer.pre_tokenize_str(\"This is an example!\")"]},{"cell_type":"markdown","metadata":{},"source":["

你可以看到每个单词在开头都加了一个 ▁，这通常是由 sentencepiece 完成的。

现在我们可以训练我们的分词器了！这次我们使用 UnigramTrainer。\"我们必须在这个训练器中明确设置未知标记，否则它会在之后忘记它。

"]},{"cell_type":"code","execution_count":42,"metadata":{"execution":{"iopub.execute_input":"2023-09-11T13:15:06.345058Z","iopub.status.busy":"2023-09-11T13:15:06.344799Z","iopub.status.idle":"2023-09-11T13:15:11.665437Z","shell.execute_reply":"2023-09-11T13:15:11.664721Z","shell.execute_reply.started":"2023-09-11T13:15:06.345042Z"}},"outputs":[{"name":"stdout","output_type":"stream","text":["\n","\n"]}],"source":["trainer = trainers.UnigramTrainer(vocab_size=25000, special_tokens=[\"[CLS]\", \"[SEP]\", \"\", \"\", \"[MASK]\"], unk_token=\"\")\n","tokenizer.train_from_iterator(batch_iterator(), trainer=trainer)"]},{"cell_type":"markdown","metadata":{},"source":["

为了完成整个流程，我们必须包括后处理器和解码器。后处理器与我们在BERT中看到的非常相似，解码器只是像预分词器一样的「Metaspace」。

"]},{"cell_type":"code","execution_count":43,"metadata":{"execution":{"iopub.execute_input":"2023-09-11T13:15:11.666587Z","iopub.status.busy":"2023-09-11T13:15:11.666399Z","iopub.status.idle":"2023-09-11T13:15:11.669642Z","shell.execute_reply":"2023-09-11T13:15:11.669218Z","shell.execute_reply.started":"2023-09-11T13:15:11.666569Z"}},"outputs":[],"source":["cls_token_id = tokenizer.token_to_id(\"[CLS]\")\n","sep_token_id = tokenizer.token_to_id(\"[SEP]\")"]},{"cell_type":"code","execution_count":44,"metadata":{"execution":{"iopub.execute_input":"2023-09-11T13:15:11.670386Z","iopub.status.busy":"2023-09-11T13:15:11.670240Z","iopub.status.idle":"2023-09-11T13:15:11.673068Z","shell.execute_reply":"2023-09-11T13:15:11.672651Z","shell.execute_reply.started":"2023-09-11T13:15:11.670371Z"}},"outputs":[],"source":["tokenizer.post_processor = processors.TemplateProcessing(\n"," single=\"[CLS]:0 $A:0 [SEP]:0\",\n"," pair=\"[CLS]:0 $A:0 [SEP]:0 $B:1 [SEP]:1\",\n"," special_tokens=[\n"," (\"[CLS]\", cls_token_id),\n"," (\"[SEP]\", sep_token_id),\n"," ],\n",")\n","tokenizer.decoder = decoders.Metaspace()"]},{"cell_type":"markdown","metadata":{},"source":["

和之前一样，我们最后将其包装在一个 Transformers tokenizer 对象中：

"]},{"cell_type":"code","execution_count":45,"metadata":{"execution":{"iopub.execute_input":"2023-09-11T13:15:11.673767Z","iopub.status.busy":"2023-09-11T13:15:11.673594Z","iopub.status.idle":"2023-09-11T13:15:11.700985Z","shell.execute_reply":"2023-09-11T13:15:11.700547Z","shell.execute_reply.started":"2023-09-11T13:15:11.673752Z"}},"outputs":[],"source":["from transformers import AlbertTokenizerFast\n","\n","new_tokenizer = AlbertTokenizerFast(tokenizer_object=tokenizer)"]},{"cell_type":"markdown","metadata":{},"source":["

使用你的新分词器来训练语言模型！

"]},{"cell_type":"markdown","metadata":{},"source":["

你可以在从头开始的语言建模笔记中使用你的新分词器，或者在语言建模脚本中使用 --tokenizer_name 参数来训练一个从头开始的模型。

"]}],"metadata":{"colab":{"name":"Train your tokenizer","provenance":[]},"kernelspec":{"display_name":"Python 3 (ipykernel)","language":"python","name":"python3"},"language_info":{"codemirror_mode":{"name":"ipython","version":3},"file_extension":".py","mimetype":"text/x-python","name":"python","nbconvert_exporter":"python","pygments_lexer":"ipython3","version":"3.8.16"}},"nbformat":4,"nbformat_minor":4}

Command Palette

独自のトークナイザーを最初からトレーニングする

Build AI with AI

HyperAI Newsletters

Command Palette

独自のトークナイザーを最初からトレーニングする

Build AI with AI

HyperAI Newsletters

Command Palette

独自のトークナイザーを最初からトレーニングする

Build AI with AI

HyperAI Newsletters