vukrosic
/

essential-web-16k-tokenizer

Model card Files Files and versions

essential-web-16k-tokenizer

1.59 MB

1 contributor

History: 6 commits

vukrosic's picture

Add essential_web_500k_tokens.pkl - Tokenized version (pickle format) of 500K chars from Essential-Web

4ca13e5 verified 8 months ago

.gitattributes

1.52 kB

initial commit 8 months ago
README.md

4.6 kB

Add tokenizer usage documentation 8 months ago
bpe_tokenizer_16k_n1000000.pkl
Pickle imports
- No problematic imports detected
What is a pickle import?
194 kB
xet

Upload bpe_tokenizer_16k_n1000000.pkl 8 months ago
essential_web_500k_text.txt

504 kB

Add essential_web_500k_text.txt - 500K characters of raw text from Essential-Web dataset 8 months ago
essential_web_500k_tokens.pkl
Pickle imports
- No problematic imports detected
What is a pickle import?
350 kB
xet

Add essential_web_500k_tokens.pkl - Tokenized version (pickle format) of 500K chars from Essential-Web 8 months ago
essential_web_500k_tokens.txt

538 kB

Add essential_web_500k_tokens.txt - Tokenized version (one token per line) of 500K chars from Essential-Web 8 months ago