python-tokenizers 0.23.1-1
| Status: | SUCCESS |
|---|---|
| Last updated: | 2026-06-02 07:12 |
| Description: | Fast State-of-the-Art Tokenizers optimized for Research and Production |
| Upstream: | AUR |
Build Output
==> Making package: python-tokenizers 0.23.1-1 (Tue Jun 2 07:07:07 2026)
==> Checking runtime dependencies...
==> Checking buildtime dependencies...
==> Installing missing dependencies...
resolving dependencies...
looking for conflicting packages...
Packages (103) abseil-cpp-20260107.1-1 apache-orc-2.3.0-3 arrow-24.0.0-1 aws-c-auth-0.10.1-1 aws-c-cal-0.9.13-1 aws-c-common-0.12.6-1 aws-c-compression-0.3.2-1 aws-c-event-stream-0.7.0-1 aws-c-http-0.10.14-1 aws-c-io-0.26.3-1 aws-c-mqtt-0.15.2-1 aws-c-s3-0.12.2-1 aws-c-sdkutils-0.2.4-1 aws-checksums-0.2.10-1 aws-crt-cpp-0.38.5-1 aws-sdk-cpp-core-1.11.792-1 aws-sdk-cpp-iam-1.11.792-1 aws-sdk-cpp-s3-1.11.792-1 blas-3.12.1-2 c-ares-1.34.6-1 cblas-3.12.1-2 compiler-rt-22.1.6-1 gflags-2.2.2-6 google-glog-0.7.1-2 grpc-1.80.0-1 gtest-1.17.0-2 hdf5-2.1.1-1 lapack-3.12.1-2 libaec-1.1.6-1 libgit2-1:1.9.4-1 liblzf-3.6-5 lld-22.1.6-1 llhttp-9.3.1-1 llvm-libs-22.1.6-1 maturin-1.13.3-1 protobuf-34.1-1 python-aiohappyeyeballs-2.6.1-4 python-aiohttp-3.13.5-1 python-aiosignal-1.4.0-3 python-annotated-doc-0.0.4-2 python-annotated-types-0.7.0-3 python-anyio-4.13.0-1 python-autocommand-2.2.2-9 python-certifi-2026.05.20-1 python-dateutil-2.9.0-8 python-dill-0.4.1-1 python-filelock-3.29.0-1 python-frozenlist-1.8.0-2 python-fsspec-2026.4.0-1 python-h11-0.16.0-2 python-h5py-3.16.0-1 python-hf-xet-1.5.0-1 python-httpcore-1.0.9-3 python-httpx-0.28.1-7 python-huggingface-hub-1:1.16.0-1 python-idna-3.17-1 python-iniconfig-2.3.0-1 python-jaraco.collections-5.1.0-3 python-jaraco.context-6.1.2-1 python-jaraco.functools-4.1.0-3 python-jaraco.text-4.0.0-4 python-markdown-it-py-4.0.0-2 python-mdurl-0.1.2-9 python-more-itertools-11.1.0-1 python-multidict-6.7.1-1 python-multiprocess-0.70.19-1 python-pandas-2.3.3-2 python-pkg_resources-81.0.0-1 python-platformdirs-4.10.0-1 python-pluggy-1.6.0-3 python-propcache-0.4.1-2 python-pydantic-2.13.4-1 python-pydantic-core-3:2.46.4-1 python-pygments-2.20.0-1 python-pyproject-hooks-1.2.0-6 python-pytz-2026.1-1 python-rich-15.0.0-1 python-semantic-version-2.10.0-9 python-setuptools-1:82.0.1-1 python-shellingham-1.5.4-4 python-six-1.17.0-3 python-typer-0.26.0-1 python-typing-inspection-0.4.2-2 python-typing_extensions-4.15.0-3 python-xxhash-3.7.0-1 python-yarl-1.23.0-1 re2-2:2025.11.05-3 rust-1:1.95.0-1 s2n-tls-1.7.2-1 snappy-1.2.2-3 thrift-0.22.0-6 clang-22.1.6-1 python-build-1.4.3-1 python-datasets-4.8.5-1 python-installer-1.0.0-1 python-maturin-1.13.3-1 python-numpy-2.4.6-1 python-pyarrow-24.0.0-1 python-pytest-1:9.0.3-1 python-requests-2.34.2-1 python-setuptools-rust-1.12.1-1 python-wheel-0.47.0-1 rust-bindgen-0.72.1-2
Total Download Size: 146.07 MiB
Total Installed Size: 1234.10 MiB
:: Proceed with installation? [Y/n]
:: Retrieving packages...
clang-22.1.6-1-x86_64 downloading...
python-pandas-2.3.3-2-x86_64 downloading...
arrow-24.0.0-1-x86_64 downloading...
python-numpy-2.4.6-1-x86_64 downloading...
maturin-1.13.3-1-x86_64 downloading...
grpc-1.80.0-1-x86_64 downloading...
lapack-3.12.1-2-x86_64 downloading...
compiler-rt-22.1.6-1-x86_64 downloading...
hdf5-2.1.1-1-x86_64 downloading...
python-pyarrow-24.0.0-1-x86_64 downloading...
python-hf-xet-1.5.0-1-x86_64 downloading...
lld-22.1.6-1-x86_64 downloading...
rust-bindgen-0.72.1-2-x86_64 downloading...
thrift-0.22.0-6-x86_64 downloading...
aws-sdk-cpp-s3-1.11.792-1-x86_64 downloading...
python-pydantic-core-3:2.46.4-1-x86_64 downloading...
python-h5py-3.16.0-1-x86_64 downloading...
python-huggingface-hub-1:1.16.0-1-any downloading...
libgit2-1:1.9.4-1-x86_64 downloading...
python-datasets-4.8.5-1-any downloading...
python-pydantic-2.13.4-1-any downloading...
python-aiohttp-3.13.5-1-x86_64 downloading...
aws-sdk-cpp-iam-1.11.792-1-x86_64 downloading...
aws-sdk-cpp-core-1.11.792-1-x86_64 downloading...
python-rich-15.0.0-1-any downloading...
apache-orc-2.3.0-3-x86_64 downloading...
python-fsspec-2026.4.0-1-any downloading...
python-httpx-0.28.1-7-any downloading...
python-multiprocess-0.70.19-1-any downloading...
s2n-tls-1.7.2-1-x86_64 downloading...
python-dateutil-2.9.0-8-any downloading...
aws-crt-cpp-0.38.5-1-x86_64 downloading...
python-anyio-4.13.0-1-any downloading...
python-typer-0.26.0-1-any downloading...
blas-3.12.1-2-x86_64 downloading...
python-dill-0.4.1-1-any downloading...
re2-2:2025.11.05-3-x86_64 downloading...
aws-c-common-0.12.6-1-x86_64 downloading...
aws-c-http-0.10.14-1-x86_64 downloading...
aws-c-mqtt-0.15.2-1-x86_64 downloading...
gflags-2.2.2-6-x86_64 downloading...
python-markdown-it-py-4.0.0-2-any downloading...
python-requests-2.34.2-1-any downloading...
aws-c-io-0.26.3-1-x86_64 downloading...
google-glog-0.7.1-2-x86_64 downloading...
python-httpcore-1.0.9-3-any downloading...
aws-c-s3-0.12.2-1-x86_64 downloading...
python-yarl-1.23.0-1-x86_64 downloading...
aws-c-auth-0.10.1-1-x86_64 downloading...
python-idna-3.17-1-any downloading...
aws-checksums-0.2.10-1-x86_64 downloading...
python-filelock-3.29.0-1-any downloading...
python-multidict-6.7.1-1-x86_64 downloading...
python-h11-0.16.0-2-any downloading...
python-setuptools-rust-1.12.1-1-any downloading...
cblas-3.12.1-2-x86_64 downloading...
python-pytz-2026.1-1-any downloading...
python-frozenlist-1.8.0-2-x86_64 downloading...
python-propcache-0.4.1-2-x86_64 downloading...
aws-c-event-stream-0.7.0-1-x86_64 downloading...
aws-c-sdkutils-0.2.4-1-x86_64 downloading...
aws-c-cal-0.9.13-1-x86_64 downloading...
python-semantic-version-2.10.0-9-any downloading...
snappy-1.2.2-3-x86_64 downloading...
python-six-1.17.0-3-any downloading...
python-aiohappyeyeballs-2.6.1-4-any downloading...
python-typing-inspection-0.4.2-2-any downloading...
libaec-1.1.6-1-x86_64 downloading...
python-maturin-1.13.3-1-x86_64 downloading...
python-annotated-types-0.7.0-3-any downloading...
python-shellingham-1.5.4-4-any downloading...
python-mdurl-0.1.2-9-any downloading...
python-xxhash-3.7.0-1-x86_64 downloading...
python-aiosignal-1.4.0-3-any downloading...
aws-c-compression-0.3.2-1-x86_64 downloading...
python-annotated-doc-0.0.4-2-any downloading...
python-certifi-2026.05.20-1-any downloading...
checking keyring...
checking package integrity...
loading package files...
checking for file conflicts...
checking available disk space...
:: Processing package changes...
installing llvm-libs...
installing compiler-rt...
installing clang...
Optional dependencies for clang
openmp: OpenMP support in clang with -fopenmp
python: for scan-view and git-clang-format [installed]
llvm: referenced by some clang headers
installing rust-bindgen...
installing python-pyproject-hooks...
installing python-build...
Optional dependencies for python-build
python-pip: to use as the Python package installer (default)
python-uv: to use as the Python package installer
python-virtualenv: to use virtualenv for build isolation
installing python-installer...
installing llhttp...
installing libgit2...
installing lld...
installing rust...
Optional dependencies for rust
gdb: rust-gdb script [installed]
lldb: rust-lldb script
installing maturin...
installing python-maturin...
installing python-more-itertools...
installing python-jaraco.functools...
installing python-jaraco.context...
installing python-autocommand...
installing python-jaraco.text...
Optional dependencies for python-jaraco.text
python-inflect: for show-newlines script
installing python-jaraco.collections...
installing python-platformdirs...
installing python-wheel...
Optional dependencies for python-wheel
python-keyring: for wheel.signatures
python-xdg: for wheel.signatures
python-setuptools: for legacy bdist_wheel subcommand [pending]
installing python-typing_extensions...
installing python-pkg_resources...
installing python-setuptools...
installing python-semantic-version...
installing python-setuptools-rust...
installing python-aiohappyeyeballs...
installing python-frozenlist...
installing python-aiosignal...
installing python-multidict...
installing python-propcache...
installing python-idna...
installing python-yarl...
installing python-aiohttp...
Optional dependencies for python-aiohttp
gunicorn: to deploy using Gunicorn
python-aiodns: for fast DNS resolving
python-brotli: for Brotli transfer-encodings support
installing python-dill...
Optional dependencies for python-dill
python-objgraph: graph support
installing python-filelock...
installing python-fsspec...
Optional dependencies for python-fsspec
python-aiohttp: HTTP support [installed]
python-distributed: Dask support
python-libarchive-c: archives support
python-lz4: LZ4 compression support
python-paramiko: SFTP support
python-pyarrow: Arrow/Parquet support [pending]
python-pygit2: git support
python-requests: web protocols support [pending]
python-smbprotocol: SMB support
python-snappy: snappy compression support
python-tqdm: progress bar support [installed]
installing libaec...
installing hdf5...
installing liblzf...
installing blas...
installing cblas...
installing lapack...
installing python-numpy...
Optional dependencies for python-numpy
blas-openblas: faster linear algebra
installing python-h5py...
installing python-certifi...
installing python-h11...
installing python-httpcore...
Optional dependencies for python-httpcore
python-h2: for HTTP/2 support
python-socksio: for SOCKS support
python-anyio: for asyncio backend [pending]
python-trio: for trio backend
python-sniffio: for async support
installing python-anyio...
Optional dependencies for python-anyio
python-trio: trio backend
python-outcome: trio backend
python-uvloop: use uvloop for asyncio backend
python-pytest: pytest plugin [pending]
installing python-httpx...
Optional dependencies for python-httpx
python-brotli: for brotli response decompression
python-brotlicffi: for brotli response decompression
python-zstandard: for zstd response decompression
python-h2: HTTP/2 support
python-socksio: SOCKS proxy support
python-click: command line client support [installed]
python-rich: command line client support [pending]
python-pygments: command line client support [pending]
python-trio: alternative async library
installing python-hf-xet...
installing python-annotated-types...
installing python-typing-inspection...
installing python-pydantic-core...
installing python-pydantic...
Optional dependencies for python-pydantic
mypy: for type validation with mypy
python-dotenv: for .env file support
python-email-validator: for email validation
python-hypothesis: for hypothesis plugin when using legacy v1
installing python-mdurl...
installing python-markdown-it-py...
Optional dependencies for python-markdown-it-py
python-mdit_py_plugins: core plugins
python-linkify-it-py: linkify extension
installing python-pygments...
installing python-rich...
installing python-shellingham...
installing python-annotated-doc...
installing python-typer...
installing python-huggingface-hub...
Optional dependencies for python-huggingface-hub
python-authlib: for OAuth
python-fastapi: for OAuth
python-itsdangerous: for OAuth
python-gradio: for the webhooks server
python-duckdb: for hf datasets sql
python-graphviz
python-jinja [installed]
python-mcp
python-pillow
python-pydot
python-pytorch
python-safetensors
python-fastai
installing python-multiprocess...
installing python-six...
installing python-dateutil...
installing python-pytz...
installing python-pandas...
Optional dependencies for python-pandas
python-pandas-datareader: pandas.io.data replacement (recommended)
python-numexpr: accelerating certain numerical operations (recommended)
python-bottleneck: accelerating certain types of nan evaluations (recommended)
python-matplotlib: plotting
python-jinja: conditional formatting with DataFrame.style [installed]
python-tabulate: printing in Markdown-friendly format
python-scipy: miscellaneous statistical functions
python-numba: alternative execution engine
python-xarray: pandas-like API for N-dimensional data
python-xlrd: Excel XLS input
python-xlwt: Excel XLS output
python-openpyxl: Excel XLSX input/output
python-xlsxwriter: alternative Excel XLSX output
python-beautifulsoup4: read_html function (in any case)
python-html5lib: read_html function (and/or python-lxml)
python-lxml: read_xml, to_xml and read_html function (and/or python-html5lib)
python-sqlalchemy: SQL database support
python-psycopg2: PostgreSQL engine for sqlalchemy
python-pymysql: MySQL engine for sqlalchemy
python-pytables: HDF5-based reading / writing
python-blosc: for msgpack compression using blosc
zlib: compression for msgpack [installed]
python-pyarrow: Parquet, ORC and feather reading/writing [pending]
python-fsspec: handling files aside from local and HTTP [installed]
python-qtpy: read_clipboard function (only one needed)
xclip: read_clipboard function (only one needed)
xsel: read_clipboard function (only one needed)
python-brotli: Brotli compression
python-snappy: Snappy compression
python-zstandard: Zstandard (zstd) compression
installing gtest...
Optional dependencies for gtest
python: gmock generator [installed]
installing abseil-cpp...
installing protobuf...
installing snappy...
installing apache-orc...
installing aws-c-common...
installing aws-c-cal...
installing aws-c-compression...
installing s2n-tls...
installing aws-c-io...
installing aws-c-http...
installing aws-c-sdkutils...
installing aws-c-auth...
installing aws-checksums...
installing aws-c-event-stream...
installing aws-c-mqtt...
installing aws-c-s3...
installing aws-crt-cpp...
installing aws-sdk-cpp-core...
installing aws-sdk-cpp-iam...
installing aws-sdk-cpp-s3...
installing gflags...
installing google-glog...
installing c-ares...
installing re2...
installing grpc...
installing thrift...
Optional dependencies for thrift
qt5-base: TQTcpServer (Qt5) support
installing arrow...
installing python-pyarrow...
Optional dependencies for python-pyarrow
python-cffi: interact with C code
python-pandas: Pandas integration [installed]
python-fsspec: Filesystem Spec support [installed]
installing python-requests...
Optional dependencies for python-requests
python-chardet: alternative character encoding library
python-pysocks: SOCKS proxy support
installing python-xxhash...
installing python-datasets...
Optional dependencies for python-datasets
python-librosa: Audio datasets
python-soxr: Audio datasets
python-torchcodec: Audio datasets
python-pillow: Vision datasets
python-tensorflow: TensorFlow support
python-pytorch: PyTorch support
installing python-iniconfig...
installing python-pluggy...
installing python-pytest...
:: Running post-transaction hooks...
(1/1) Arming ConditionNeedsUpdate...
==> Retrieving sources...
-> Downloading tokenizers-0.23.1.tar.gz...
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0
100 1.53M 0 1.53M 0 0 2.28M 0 0
100 1.53M 0 1.53M 0 0 2.28M 0 0
100 1.53M 0 1.53M 0 0 2.28M 0 0
-> Downloading norvig-big.txt...
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 0
56 6.18M 56 3.48M 0 0 2.53M 0 00:02 00:01 00:01 3.32M
100 6.18M 100 6.18M 0 0 4.18M 0 00:01 00:01 3.32M
100 6.18M 100 6.18M 0 0 4.18M 0 00:01 00:01 3.32M
100 6.18M 100 6.18M 0 0 4.18M 0 00:01 00:01 3.32M
-> Downloading roberta.json...
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 0
100 84 100 84 0 0 596 0 0
100 84 100 84 0 0 595 0 0
0 0 0 0 0 0 0 0 0
100 1.29M 100 1.29M 0 0 2.27M 0 0
100 1.29M 100 1.29M 0 0 2.27M 0 0
100 1.29M 100 1.29M 0 0 2.27M 0 0
==> WARNING: Skipping verification of source file PGP signatures.
==> Validating source files with sha256sums...
tokenizers-0.23.1.tar.gz ... Passed
norvig-big.txt ... Passed
roberta.json ... Passed
==> Extracting sources...
-> Extracting tokenizers-0.23.1.tar.gz with bsdtar
==> Starting prepare()...
Updating crates.io index
Locking 22 packages to latest compatible versions
Updating autocfg v1.5.0 -> v1.5.1
Updating bumpalo v3.20.2 -> v3.20.3
Updating cc v1.2.61 -> v1.2.63
Updating compact_str v0.9.0 -> v0.9.1
Updating either v1.15.0 -> v1.16.0
Updating hashbrown v0.17.0 -> v0.17.1
Updating jiff v0.2.24 -> v0.2.28
Updating jiff-static v0.2.24 -> v0.2.28
Updating js-sys v0.3.95 -> v0.3.99
Updating log v0.4.29 -> v0.4.30
Updating memchr v2.8.0 -> v2.8.1
Updating mio v1.2.0 -> v1.2.1
Updating serde_json v1.0.149 -> v1.0.150
Updating shlex v1.3.0 -> v2.0.1
Updating tokio v1.52.1 -> v1.52.3
Updating unicode-segmentation v1.13.2 -> v1.13.3
Updating wasm-bindgen v0.2.118 -> v0.2.122
Updating wasm-bindgen-macro v0.2.118 -> v0.2.122
Updating wasm-bindgen-macro-support v0.2.118 -> v0.2.122
Updating wasm-bindgen-shared v0.2.118 -> v0.2.122
Updating zerocopy v0.8.48 -> v0.8.50
Updating zerocopy-derive v0.8.48 -> v0.8.50
note: pass `--verbose` to see 6 unchanged dependencies behind latest
Downloading crates ...
Downloaded rustc-hash v2.1.2
Downloaded futures-macro v0.3.32
Downloaded tokio-macros v2.7.0
Downloaded quote v1.0.45
Downloaded numpy v0.28.0
Downloaded futures-core v0.3.32
Downloaded unicode-ident v1.0.24
Downloaded futures-channel v0.3.32
Downloaded proc-macro2 v1.0.106
Downloaded futures-task v0.3.32
Downloaded heck v0.5.0
Downloaded anstyle v1.0.14
Downloaded env_filter v1.0.1
Downloaded colorchoice v1.0.5
Downloaded anstyle-query v1.1.5
Downloaded utf8parse v0.2.2
Downloaded num-complex v0.4.6
Downloaded matrixmultiply v0.3.10
Downloaded derive_builder_macro v0.20.2
Downloaded itoa v1.0.18
Downloaded is_terminal_polyfill v1.70.2
Downloaded rawpointer v0.2.1
Downloaded strsim v0.11.1
Downloaded anstyle-parse v1.0.0
Downloaded num-integer v0.1.46
Downloaded anstream v1.0.0
Downloaded cfg-if v1.0.4
Downloaded env_logger v0.11.10
Downloaded autocfg v1.5.1
Downloaded macro_rules_attribute v0.2.2
Downloaded macro_rules_attribute-proc_macro v0.2.2
Downloaded num-traits v0.2.19
Downloaded errno v0.3.14
Downloaded fnv v1.0.7
Downloaded rayon-cond v0.4.0
Downloaded monostate v0.1.18
Downloaded version_check v0.9.5
Downloaded castaway v0.2.4
Downloaded ndarray v0.17.2
Downloaded monostate-impl v0.1.18
Downloaded unit-prefix v0.5.2
Downloaded ident_case v1.0.1
Downloaded ndarray v0.16.1
Downloaded jiff v0.2.28
Downloaded slab v0.4.12
Downloaded pyo3-build-config v0.28.2
Downloaded rustversion v1.0.22
Downloaded darling v0.20.11
Downloaded console v0.16.3
Downloaded bitflags v2.11.1
Downloaded once_cell v1.21.4
Downloaded rand_chacha v0.9.0
Downloaded rand_core v0.9.5
Downloaded crossbeam-utils v0.8.21
Downloaded zmij v1.0.21
Downloaded thiserror-impl v2.0.18
Downloaded thiserror v2.0.18
Downloaded pin-project-lite v0.2.17
Downloaded signal-hook-registry v1.4.8
Downloaded pyo3-macros v0.28.2
Downloaded target-lexicon v0.13.5
Downloaded fastrand v2.4.1
Downloaded static_assertions v1.1.0
Downloaded dary_heap v0.3.9
Downloaded darling_macro v0.20.11
Downloaded derive_builder_core v0.20.2
Downloaded find-msvc-tools v0.1.9
Downloaded pkg-config v0.3.33
Downloaded onig v6.5.3
Downloaded paste v1.0.15
Downloaded ppv-lite86 v0.2.21
Downloaded either v1.16.0
Downloaded crossbeam-deque v0.8.6
Downloaded smallvec v1.15.1
Downloaded shlex v2.0.1
Downloaded ryu v1.0.23
Downloaded derive_builder v0.20.2
Downloaded crossbeam-epoch v0.9.18
Downloaded ahash v0.8.12
Downloaded tempfile v3.27.0
Downloaded pyo3-async-runtimes v0.28.0
Downloaded getrandom v0.4.2
Downloaded base64 v0.13.1
Downloaded getrandom v0.3.4
Downloaded daachorse v1.0.1
Downloaded log v0.4.30
Downloaded darling_core v0.20.11
Downloaded serde_core v1.0.228
Downloaded serde_derive v1.0.228
Downloaded rayon-core v1.13.0
Downloaded cc v1.2.63
Downloaded indicatif v0.18.4
Downloaded pyo3-macros-backend v0.28.2
Downloaded unicode_categories v0.1.1
Downloaded compact_str v0.9.1
Downloaded unicode-normalization-alignments v0.1.12
Downloaded pyo3-ffi v0.28.2
Downloaded minimal-lexical v0.2.1
Downloaded rand v0.9.4
Downloaded memchr v2.8.1
Downloaded serde v1.0.228
Downloaded nom v7.1.3
Downloaded unicode-segmentation v1.13.3
Downloaded mio v1.2.1
Downloaded itertools v0.14.0
Downloaded futures-util v0.3.32
Downloaded portable-atomic v1.13.1
Downloaded rayon v1.12.0
Downloaded esaxx-rs v0.1.10
Downloaded aho-corasick v1.1.4
Downloaded syn v2.0.117
Downloaded unicode-width v0.2.2
Downloaded regex v1.12.3
Downloaded zerocopy v0.8.50
Downloaded serde_json v1.0.150
Downloaded regex-syntax v0.8.10
Downloaded rustix v1.1.4
Downloaded spm_precompiled v0.1.4
Downloaded regex-automata v0.4.14
Downloaded onig_sys v69.9.3
Downloaded tokio v1.52.3
Downloaded libc v0.2.186
Downloaded pyo3 v0.28.2
Downloaded linux-raw-sys v0.12.1
==> Starting build()...
Compiling proc-macro2 v1.0.106
Compiling quote v1.0.45
Compiling unicode-ident v1.0.24
Compiling target-lexicon v0.13.5
Compiling libc v0.2.186
Compiling autocfg v1.5.1
Compiling serde_core v1.0.228
Compiling pyo3-build-config v0.28.2
Compiling syn v2.0.117
Compiling memchr v2.8.1
Compiling find-msvc-tools v0.1.9
Compiling shlex v2.0.1
Compiling cc v1.2.63
Compiling crossbeam-utils v0.8.21
Compiling num-traits v0.2.19
Compiling ident_case v1.0.1
Compiling getrandom v0.3.4
Compiling zerocopy v0.8.50
Compiling strsim v0.11.1
Compiling cfg-if v1.0.4
Compiling serde v1.0.228
Compiling fnv v1.0.7
Compiling darling_core v0.20.11
Compiling pyo3-macros-backend v0.28.2
Compiling pyo3-ffi v0.28.2
Compiling once_cell v1.21.4
Compiling crossbeam-epoch v0.9.18
Compiling serde_derive v1.0.228
Compiling aho-corasick v1.1.4
Compiling darling_macro v0.20.11
Compiling matrixmultiply v0.3.10
Compiling pkg-config v0.3.33
Compiling heck v0.5.0
Compiling rayon-core v1.13.0
Compiling regex-syntax v0.8.10
Compiling rustversion v1.0.22
Compiling onig_sys v69.9.3
Compiling darling v0.20.11
Compiling regex-automata v0.4.14
Compiling crossbeam-deque v0.8.6
Compiling pyo3 v0.28.2
Compiling version_check v0.9.5
Compiling either v1.16.0
Compiling zmij v1.0.21
Compiling portable-atomic v1.13.1
Compiling rawpointer v0.2.1
Compiling paste v1.0.15
Compiling ahash v0.8.12
Compiling pyo3-macros v0.28.2
Compiling regex v1.12.3
Compiling derive_builder_core v0.20.2
Compiling ppv-lite86 v0.2.21
Compiling rand_core v0.9.5
Compiling num-complex v0.4.6
Compiling num-integer v0.1.46
Compiling esaxx-rs v0.1.10
Compiling errno v0.3.14
Compiling futures-core v0.3.32
Compiling thiserror v2.0.18
Compiling log v0.4.30
Compiling unicode-width v0.2.2
Compiling pin-project-lite v0.2.17
Compiling utf8parse v0.2.2
Compiling serde_json v1.0.150
Compiling minimal-lexical v0.2.1
Compiling itoa v1.0.18
Compiling nom v7.1.3
Compiling console v0.16.3
Compiling anstyle-parse v1.0.0
Compiling signal-hook-registry v1.4.8
Compiling rand_chacha v0.9.0
Compiling castaway v0.2.4
Compiling derive_builder_macro v0.20.2
Compiling rayon v1.12.0
Compiling itertools v0.14.0
Compiling thiserror-impl v2.0.18
Compiling futures-macro v0.3.32
Compiling tokio-macros v2.7.0
Compiling monostate-impl v0.1.18
Compiling numpy v0.28.0
Compiling mio v1.2.1
Compiling anstyle v1.0.14
Compiling smallvec v1.15.1
Compiling colorchoice v1.0.5
Compiling static_assertions v1.1.0
Compiling macro_rules_attribute-proc_macro v0.2.2
Compiling unicode-segmentation v1.13.3
Compiling anstyle-query v1.1.5
Compiling ryu v1.0.23
Compiling base64 v0.13.1
Compiling slab v0.4.12
Compiling bitflags v2.11.1
Compiling futures-task v0.3.32
Compiling is_terminal_polyfill v1.70.2
Compiling unit-prefix v0.5.2
Compiling onig v6.5.3
Compiling indicatif v0.18.4
Compiling futures-util v0.3.32
Compiling anstream v1.0.0
Compiling spm_precompiled v0.1.4
Compiling compact_str v0.9.1
Compiling macro_rules_attribute v0.2.2
Compiling unicode-normalization-alignments v0.1.12
Compiling tokio v1.52.3
Compiling rayon-cond v0.4.0
Compiling monostate v0.1.18
Compiling derive_builder v0.20.2
Compiling rand v0.9.4
Compiling env_filter v1.0.1
Compiling futures-channel v0.3.32
Compiling ndarray v0.17.2
Compiling dary_heap v0.3.9
Compiling jiff v0.2.28
Compiling daachorse v1.0.1
Compiling rustc-hash v2.1.2
Compiling unicode_categories v0.1.1
Compiling tokenizers v0.23.1 (/builder/src/tokenizers-0.23.1/tokenizers)
Compiling env_logger v0.11.10
Compiling pyo3-async-runtimes v0.28.0
Compiling ndarray v0.16.1
Compiling tokenizers-python v0.23.1 (/builder/src/tokenizers-0.23.1/bindings/python)
Finished `release` profile [optimized] target(s) in 3m 09s
* Getting build dependencies for wheel...
* Building wheel...
Running `maturin pep517 build-wheel -i /usr/sbin/python --compatibility off`
Downloading crates ...
Downloaded equivalent v1.0.2
Downloaded anstyle-wincon v3.0.11
Downloaded anyhow v1.0.102
Downloaded once_cell_polyfill v1.70.2
Downloaded encode_unicode v1.0.0
Downloaded foldhash v0.1.5
Downloaded id-arena v2.3.0
Downloaded unicode-xid v0.2.6
Downloaded wasm-metadata v0.244.0
Downloaded wasm-bindgen v0.2.122
Downloaded windows-link v0.2.1
Downloaded wit-bindgen-rust-macro v0.51.0
Downloaded wasip3 v0.4.0+wasi-0.3.0-rc-2026-01-06
Downloaded wit-bindgen-core v0.51.0
Downloaded zerocopy-derive v0.8.50
Downloaded wasm-encoder v0.244.0
Downloaded wasm-bindgen-macro-support v0.2.122
Downloaded wit-parser v0.244.0
Downloaded wit-bindgen v0.57.1
Downloaded wasmparser v0.244.0
Downloaded wit-component v0.244.0
Downloaded wit-bindgen v0.51.0
Downloaded hashbrown v0.15.5
Downloaded hashbrown v0.17.1
Downloaded wasip2 v1.0.3+wasi-0.2.9
Downloaded js-sys v0.3.99
Downloaded wit-bindgen-rust v0.51.0
Downloaded web-time v1.1.0
Downloaded wasm-bindgen-shared v0.2.122
Downloaded wasm-bindgen-macro v0.2.122
Downloaded jiff-static v0.2.28
Downloaded indexmap v2.14.0
Downloaded r-efi v6.0.0
Downloaded prettyplease v0.2.37
Downloaded wasi v0.11.1+wasi-snapshot-preview1
Downloaded r-efi v5.3.0
Downloaded portable-atomic-util v0.2.7
Downloaded bumpalo v3.20.3
Downloaded semver v1.0.28
Downloaded leb128fmt v0.1.0
Downloaded windows-sys v0.61.2
🍹 Building a mixed python/rust project
🐍 Found CPython 3.14 at /usr/sbin/python
🔗 Found pyo3 bindings with abi3 support
📡 Using build options features, bindings from pyproject.toml
Compiling pyo3-build-config v0.28.2
Compiling pyo3-ffi v0.28.2
Compiling pyo3-macros-backend v0.28.2
Compiling pyo3 v0.28.2
Compiling numpy v0.28.0
Compiling pyo3-macros v0.28.2
Compiling pyo3-async-runtimes v0.28.0
Compiling tokenizers-python v0.23.1 (/builder/src/tokenizers-0.23.1/bindings/python)
Finished `release` profile [optimized] target(s) in 40.46s
📦 Built wheel for abi3 Python ≥ 3.10 to /builder/src/tokenizers-0.23.1/bindings/python/target/wheels/tokenizers-0.23.1-cp310-abi3-linux_x86_64.whl
/builder/src/tokenizers-0.23.1/bindings/python/target/wheels/tokenizers-0.23.1-cp310-abi3-linux_x86_64.whl
Successfully built tokenizers-0.23.1-cp310-abi3-linux_x86_64.whl
==> Starting check()...
============================= test session starts ==============================
platform linux -- Python 3.14.5, pytest-9.0.3, pluggy-1.6.0 -- /builder/src/tokenizers-0.23.1/bindings/python/test-env/bin/python
cachedir: .pytest_cache
rootdir: /builder/src/tokenizers-0.23.1/bindings/python
configfile: pytest.ini (WARNING: ignoring pytest config in setup.cfg!)
plugins: anyio-4.13.0
collecting ... collected 198 items / 1 skipped
tests/bindings/test_decoders.py::TestByteLevel::test_instantiate PASSED
tests/bindings/test_decoders.py::TestByteLevel::test_decoding PASSED
tests/bindings/test_decoders.py::TestByteLevel::test_manual_reload PASSED
tests/bindings/test_decoders.py::TestReplace::test_instantiate PASSED
tests/bindings/test_decoders.py::TestReplace::test_decoding PASSED
tests/bindings/test_decoders.py::TestWordPiece::test_instantiate PASSED
tests/bindings/test_decoders.py::TestWordPiece::test_decoding PASSED
tests/bindings/test_decoders.py::TestWordPiece::test_can_modify PASSED
tests/bindings/test_decoders.py::TestByteFallback::test_instantiate PASSED
tests/bindings/test_decoders.py::TestByteFallback::test_decoding PASSED
tests/bindings/test_decoders.py::TestFuse::test_instantiate PASSED
tests/bindings/test_decoders.py::TestFuse::test_decoding PASSED
tests/bindings/test_decoders.py::TestStrip::test_instantiate PASSED
tests/bindings/test_decoders.py::TestStrip::test_decoding PASSED
tests/bindings/test_decoders.py::TestMetaspace::test_instantiate PASSED
tests/bindings/test_decoders.py::TestMetaspace::test_decoding PASSED
tests/bindings/test_decoders.py::TestMetaspace::test_can_modify PASSED
tests/bindings/test_decoders.py::TestBPEDecoder::test_instantiate PASSED
tests/bindings/test_decoders.py::TestBPEDecoder::test_decoding PASSED
tests/bindings/test_decoders.py::TestBPEDecoder::test_can_modify PASSED
tests/bindings/test_decoders.py::TestCTCDecoder::test_instantiate PASSED
tests/bindings/test_decoders.py::TestCTCDecoder::test_decoding PASSED
tests/bindings/test_decoders.py::TestCTCDecoder::test_can_modify PASSED
tests/bindings/test_decoders.py::TestSequenceDecoder::test_instantiate PASSED
tests/bindings/test_decoders.py::TestSequenceDecoder::test_decoding PASSED
tests/bindings/test_encoding.py::TestEncoding::test_sequence_ids PASSED
tests/bindings/test_encoding.py::TestEncoding::test_n_sequences PASSED
tests/bindings/test_encoding.py::TestEncoding::test_word_to_tokens PASSED
tests/bindings/test_encoding.py::TestEncoding::test_word_to_chars PASSED
tests/bindings/test_encoding.py::TestEncoding::test_token_to_sequence PASSED
tests/bindings/test_encoding.py::TestEncoding::test_token_to_chars PASSED
tests/bindings/test_encoding.py::TestEncoding::test_token_to_word PASSED
tests/bindings/test_encoding.py::TestEncoding::test_char_to_token PASSED
tests/bindings/test_encoding.py::TestEncoding::test_char_to_word PASSED
tests/bindings/test_encoding.py::TestEncoding::test_truncation PASSED
tests/bindings/test_encoding.py::TestEncoding::test_invalid_truncate_direction PASSED
tests/bindings/test_models.py::TestBPE::test_can_modify PASSED
tests/bindings/test_models.py::TestBPE::test_dropout_zero PASSED
tests/bindings/test_models.py::TestUnigram::test_can_modify PASSED
tests/bindings/test_models.py::TestUnigram::test_alpha_zero PASSED
tests/bindings/test_models.py::TestWordPiece::test_instantiate PASSED
tests/bindings/test_models.py::TestWordPiece::test_can_modify PASSED
tests/bindings/test_models.py::TestWordLevel::test_instantiate PASSED
tests/bindings/test_models.py::TestWordLevel::test_can_modify PASSED
tests/bindings/test_normalizers.py::TestBertNormalizer::test_instantiate PASSED
tests/bindings/test_normalizers.py::TestBertNormalizer::test_strip_accents PASSED
tests/bindings/test_normalizers.py::TestBertNormalizer::test_handle_chinese_chars PASSED
tests/bindings/test_normalizers.py::TestBertNormalizer::test_clean_text PASSED
tests/bindings/test_normalizers.py::TestBertNormalizer::test_lowercase PASSED
tests/bindings/test_normalizers.py::TestBertNormalizer::test_can_modify PASSED
tests/bindings/test_normalizers.py::TestSequence::test_instantiate PASSED
tests/bindings/test_normalizers.py::TestSequence::test_can_make_sequences PASSED
tests/bindings/test_normalizers.py::TestSequence::test_set_item PASSED
tests/bindings/test_normalizers.py::TestSequence::test_item_getters_and_setters PASSED
tests/bindings/test_normalizers.py::TestLowercase::test_instantiate PASSED
tests/bindings/test_normalizers.py::TestLowercase::test_lowercase PASSED
tests/bindings/test_normalizers.py::TestStrip::test_instantiate PASSED
tests/bindings/test_normalizers.py::TestStrip::test_left_strip PASSED
tests/bindings/test_normalizers.py::TestStrip::test_right_strip PASSED
tests/bindings/test_normalizers.py::TestStrip::test_full_strip PASSED
tests/bindings/test_normalizers.py::TestStrip::test_can_modify PASSED
tests/bindings/test_normalizers.py::TestPrepend::test_instantiate PASSED
tests/bindings/test_normalizers.py::TestPrepend::test_prepend PASSED
tests/bindings/test_normalizers.py::TestPrepend::test_can_modify PASSED
tests/bindings/test_normalizers.py::TestCustomNormalizer::test_instantiate PASSED
tests/bindings/test_normalizers.py::TestCustomNormalizer::test_normalizer_interface PASSED
tests/bindings/test_pre_tokenizers.py::TestByteLevel::test_instantiate PASSED
tests/bindings/test_pre_tokenizers.py::TestByteLevel::test_has_alphabet PASSED
tests/bindings/test_pre_tokenizers.py::TestByteLevel::test_can_modify PASSED
tests/bindings/test_pre_tokenizers.py::TestByteLevel::test_manual_reload PASSED
tests/bindings/test_pre_tokenizers.py::TestSplit::test_instantiate PASSED
tests/bindings/test_pre_tokenizers.py::TestWhitespace::test_instantiate PASSED
tests/bindings/test_pre_tokenizers.py::TestWhitespaceSplit::test_instantiate PASSED
tests/bindings/test_pre_tokenizers.py::TestBertPreTokenizer::test_instantiate PASSED
tests/bindings/test_pre_tokenizers.py::TestMetaspace::test_instantiate PASSED
tests/bindings/test_pre_tokenizers.py::TestMetaspace::test_can_modify PASSED
tests/bindings/test_pre_tokenizers.py::TestCharDelimiterSplit::test_instantiate PASSED
tests/bindings/test_pre_tokenizers.py::TestCharDelimiterSplit::test_can_modify PASSED
tests/bindings/test_pre_tokenizers.py::TestPunctuation::test_instantiate PASSED
tests/bindings/test_pre_tokenizers.py::TestSequence::test_instantiate PASSED
tests/bindings/test_pre_tokenizers.py::TestSequence::test_bert_like PASSED
tests/bindings/test_pre_tokenizers.py::TestSequence::test_set_item PASSED
tests/bindings/test_pre_tokenizers.py::TestSequence::test_item_getters_and_setters PASSED
tests/bindings/test_pre_tokenizers.py::TestDigits::test_instantiate PASSED
tests/bindings/test_pre_tokenizers.py::TestDigits::test_can_modify PASSED
tests/bindings/test_pre_tokenizers.py::TestFixedLength::test_instantiate PASSED
tests/bindings/test_pre_tokenizers.py::TestFixedLength::test_pre_tokenize_str PASSED
tests/bindings/test_pre_tokenizers.py::TestUnicodeScripts::test_instantiate PASSED
tests/bindings/test_pre_tokenizers.py::TestCustomPreTokenizer::test_instantiate PASSED
tests/bindings/test_pre_tokenizers.py::TestCustomPreTokenizer::test_camel_case PASSED
tests/bindings/test_processors.py::TestBertProcessing::test_instantiate PASSED
tests/bindings/test_processors.py::TestBertProcessing::test_processing PASSED
tests/bindings/test_processors.py::TestRobertaProcessing::test_instantiate PASSED
tests/bindings/test_processors.py::TestRobertaProcessing::test_processing PASSED
tests/bindings/test_processors.py::TestByteLevelProcessing::test_instantiate PASSED
tests/bindings/test_processors.py::TestByteLevelProcessing::test_processing PASSED
tests/bindings/test_processors.py::TestByteLevelProcessing::test_manual_reload PASSED
tests/bindings/test_processors.py::TestTemplateProcessing::test_instantiate PASSED
tests/bindings/test_processors.py::TestTemplateProcessing::test_bert_parity PASSED
tests/bindings/test_processors.py::TestTemplateProcessing::test_roberta_parity PASSED
tests/bindings/test_processors.py::TestSequenceProcessing::test_sequence_processing PASSED
tests/bindings/test_processors.py::TestSequenceProcessing::test_post_process PASSED
tests/bindings/test_processors.py::TestSequenceProcessing::test_items ByteLevel(add_prefix_space=False, trim_offsets=False, use_regex=False)
ByteLevel(add_prefix_space=True, trim_offsets=True, use_regex=True)
PASSED
tests/bindings/test_tokenizer.py::TestAddedToken::test_instantiate_with_content_only PASSED
tests/bindings/test_tokenizer.py::TestAddedToken::test_can_set_rstrip PASSED
tests/bindings/test_tokenizer.py::TestAddedToken::test_can_set_lstrip PASSED
tests/bindings/test_tokenizer.py::TestAddedToken::test_can_set_single_world PASSED
tests/bindings/test_tokenizer.py::TestAddedToken::test_can_set_normalized PASSED
tests/bindings/test_tokenizer.py::TestTokenizer::test_has_expected_type_and_methods PASSED
tests/bindings/test_tokenizer.py::TestTokenizer::test_add_tokens PASSED
tests/bindings/test_tokenizer.py::TestTokenizer::test_add_tokens_with_normalizer PASSED
tests/bindings/test_tokenizer.py::TestTokenizer::test_normalizer_change_refreshes_added_tokens PASSED
tests/bindings/test_tokenizer.py::TestTokenizer::test_add_special_tokens PASSED
tests/bindings/test_tokenizer.py::TestTokenizer::test_encode PASSED
tests/bindings/test_tokenizer.py::TestTokenizer::test_encode_formats PASSED
tests/bindings/test_tokenizer.py::TestTokenizer::test_encode_add_special_tokens PASSED
tests/bindings/test_tokenizer.py::TestTokenizer::test_truncation PASSED
tests/bindings/test_tokenizer.py::TestTokenizer::test_padding PASSED
tests/bindings/test_tokenizer.py::TestTokenizer::test_decode PASSED
tests/bindings/test_tokenizer.py::TestTokenizer::test_decode_stream_copy_and_prefix_ids PASSED
tests/bindings/test_tokenizer.py::TestTokenizer::test_decode_stream_fallback Warning: You are sending unauthenticated requests to the HF Hub. Please set a HF_TOKEN to enable higher rate limits and faster downloads.
PASSED
tests/bindings/test_tokenizer.py::TestTokenizer::test_decode_skip_special_tokens PASSED
tests/bindings/test_tokenizer.py::TestTokenizer::test_decode_stream PASSED
tests/bindings/test_tokenizer.py::TestTokenizer::test_get_vocab PASSED
tests/bindings/test_tokenizer.py::TestTokenizer::test_get_vocab_size PASSED
tests/bindings/test_tokenizer.py::TestTokenizer::test_post_process PASSED
tests/bindings/test_tokenizer.py::TestTokenizer::test_multiprocessing_with_parallelism PASSED
tests/bindings/test_tokenizer.py::TestTokenizer::test_multithreaded_concurrency PASSED
tests/bindings/test_tokenizer.py::TestTokenizer::test_from_pretrained PASSED
tests/bindings/test_tokenizer.py::TestTokenizer::test_from_pretrained_revision PASSED
tests/bindings/test_tokenizer.py::TestTokenizer::test_unigram_byte_fallback PASSED
tests/bindings/test_tokenizer.py::TestTokenizer::test_encode_special_tokens PASSED
tests/bindings/test_tokenizer.py::TestTokenizer::test_splitting PASSED
tests/bindings/test_tokenizer.py::TestTokenizer::test_decode_special PASSED
tests/bindings/test_tokenizer.py::TestTokenizer::test_weakref_support PASSED
tests/bindings/test_tokenizer.py::TestTokenizer::test_weakref_with_multiple_references PASSED
tests/bindings/test_tokenizer.py::TestTokenizer::test_setting_to_none PASSED
tests/bindings/test_tokenizer.py::TestTokenizerRepr::test_repr PASSED
tests/bindings/test_tokenizer.py::TestTokenizerRepr::test_repr_complete PASSED
tests/bindings/test_tokenizer.py::TestAsyncTokenizer::test_basic_encoding FAILED
tests/bindings/test_tokenizer.py::TestAsyncTokenizer::test_encode FAILED
tests/bindings/test_tokenizer.py::TestAsyncTokenizer::test_with_special_tokens FAILED
tests/bindings/test_tokenizer.py::TestAsyncTokenizer::test_with_truncation_padding FAILED
tests/bindings/test_tokenizer.py::TestAsyncTokenizer::test_various_input_formats FAILED
tests/bindings/test_tokenizer.py::TestAsyncTokenizer::test_error_handling FAILED
tests/bindings/test_tokenizer.py::TestAsyncTokenizer::test_concurrency FAILED
tests/bindings/test_tokenizer.py::TestAsyncTokenizer::test_decode FAILED
tests/bindings/test_tokenizer.py::TestAsyncTokenizer::test_large_batch FAILED
tests/bindings/test_tokenizer.py::TestAsyncTokenizer::test_numpy_inputs FAILED
tests/bindings/test_tokenizer.py::TestAsyncTokenizer::test_async_methods_existence PASSED
tests/bindings/test_tokenizer.py::TestAsyncTokenizer::test_performance_comparison FAILED
tests/bindings/test_trainers.py::TestBpeTrainer::test_can_modify PASSED
tests/bindings/test_trainers.py::TestBpeTrainer::test_can_pickle PASSED
tests/bindings/test_trainers.py::TestWordPieceTrainer::test_can_modify PASSED
tests/bindings/test_trainers.py::TestWordPieceTrainer::test_can_pickle PASSED
tests/bindings/test_trainers.py::TestWordLevelTrainer::test_can_modify PASSED
tests/bindings/test_trainers.py::TestWordLevelTrainer::test_can_pickle PASSED
tests/bindings/test_trainers.py::TestUnigram::test_train PASSED
tests/bindings/test_trainers.py::TestUnigram::test_train_parallelism_with_custom_pretokenizer PASSED
tests/bindings/test_trainers.py::TestUnigram::test_can_pickle PASSED
tests/bindings/test_trainers.py::TestUnigram::test_train_with_special_tokens PASSED
tests/bindings/test_trainers.py::TestUnigram::test_cannot_train_different_model PASSED
tests/bindings/test_trainers.py::TestUnigram::test_can_modify PASSED
tests/bindings/test_trainers.py::TestUnigram::test_continuing_prefix_trainer_mismatch
PASSED
tests/documentation/test_pipeline.py::TestPipeline::test_pipeline PASSED
tests/documentation/test_pipeline.py::TestPipeline::test_bert_example PASSED
tests/documentation/test_quicktour.py::TestQuicktour::test_quicktour PASSED
tests/documentation/test_tutorial_train_from_iterators.py::TestTrainFromIterators::test_train_basic PASSED
tests/documentation/test_tutorial_train_from_iterators.py::TestTrainFromIterators::test_datasets FAILED
tests/documentation/test_tutorial_train_from_iterators.py::TestTrainFromIterators::test_gzip PASSED
tests/implementations/test_base_tokenizer.py::TestBaseTokenizer::test_get_set_components PASSED
tests/implementations/test_bert_wordpiece.py::TestBertWordPieceTokenizer::test_basic_encode PASSED
tests/implementations/test_bert_wordpiece.py::TestBertWordPieceTokenizer::test_multiprocessing_with_parallelism PASSED
tests/implementations/test_bert_wordpiece.py::TestBertWordPieceTokenizer::test_train_from_iterator PASSED
tests/implementations/test_byte_level_bpe.py::TestByteLevelBPE::test_basic_encode PASSED
tests/implementations/test_byte_level_bpe.py::TestByteLevelBPE::test_add_prefix_space PASSED
tests/implementations/test_byte_level_bpe.py::TestByteLevelBPE::test_lowerspace PASSED
tests/implementations/test_byte_level_bpe.py::TestByteLevelBPE::test_multiprocessing_with_parallelism PASSED
tests/implementations/test_byte_level_bpe.py::TestByteLevelBPE::test_train_from_iterator PASSED
tests/implementations/test_char_bpe.py::TestCharBPETokenizer::test_basic_encode PASSED
tests/implementations/test_char_bpe.py::TestCharBPETokenizer::test_lowercase PASSED
tests/implementations/test_char_bpe.py::TestCharBPETokenizer::test_decoding PASSED
tests/implementations/test_char_bpe.py::TestCharBPETokenizer::test_multiprocessing_with_parallelism PASSED
tests/implementations/test_char_bpe.py::TestCharBPETokenizer::test_train_from_iterator PASSED
tests/implementations/test_sentencepiece.py::TestSentencePieceBPE::test_train_from_iterator PASSED
tests/implementations/test_sentencepiece.py::TestSentencePieceUnigram::test_train PASSED
tests/implementations/test_sentencepiece.py::TestSentencePieceUnigram::test_train_with_unk_token PASSED
tests/implementations/test_sentencepiece.py::TestSentencePieceUnigram::test_train_from_iterator PASSED
tests/implementations/test_sentencepiece.py::TestSentencePieceUnigram::test_train_from_iterator_with_unk_token PASSED
tests/test_freethreaded.py::TestEncodeUnderConcurrentSetters::test_encode_while_swapping_post_processor PASSED
tests/test_freethreaded.py::TestEncodeUnderConcurrentSetters::test_encode_while_mutating_trainer_fields PASSED
tests/test_freethreaded.py::TestEncodeUnderConcurrentSetters::test_concurrent_setters_no_lock_poisoning PASSED
tests/test_freethreaded.py::TestFreeThreadedSpecific::test_gil_actually_disabled_on_import SKIPPED
tests/test_serialization.py::TestSerialization::test_full_serialization_albert PASSED
tests/test_serialization.py::TestSerialization::test_str_big PASSED
tests/test_serialization.py::TestSerialization::test_repr_str PASSED
tests/test_serialization.py::TestSerialization::test_repr_str_ellipsis PASSED
tests/test_serialization.py::TestFullDeserialization::test_full_deserialization_hub SKIPPED
=================================== FAILURES ===================================
____________________ TestAsyncTokenizer.test_basic_encoding ____________________
async def functions are not natively supported.
You need to install a suitable plugin for your async framework, for example:
- anyio
- pytest-asyncio
- pytest-tornasync
- pytest-trio
- pytest-twisted
________________________ TestAsyncTokenizer.test_encode ________________________
async def functions are not natively supported.
You need to install a suitable plugin for your async framework, for example:
- anyio
- pytest-asyncio
- pytest-tornasync
- pytest-trio
- pytest-twisted
_________________ TestAsyncTokenizer.test_with_special_tokens __________________
async def functions are not natively supported.
You need to install a suitable plugin for your async framework, for example:
- anyio
- pytest-asyncio
- pytest-tornasync
- pytest-trio
- pytest-twisted
_______________ TestAsyncTokenizer.test_with_truncation_padding ________________
async def functions are not natively supported.
You need to install a suitable plugin for your async framework, for example:
- anyio
- pytest-asyncio
- pytest-tornasync
- pytest-trio
- pytest-twisted
________________ TestAsyncTokenizer.test_various_input_formats _________________
async def functions are not natively supported.
You need to install a suitable plugin for your async framework, for example:
- anyio
- pytest-asyncio
- pytest-tornasync
- pytest-trio
- pytest-twisted
____________________ TestAsyncTokenizer.test_error_handling ____________________
async def functions are not natively supported.
You need to install a suitable plugin for your async framework, for example:
- anyio
- pytest-asyncio
- pytest-tornasync
- pytest-trio
- pytest-twisted
_____________________ TestAsyncTokenizer.test_concurrency ______________________
async def functions are not natively supported.
You need to install a suitable plugin for your async framework, for example:
- anyio
- pytest-asyncio
- pytest-tornasync
- pytest-trio
- pytest-twisted
________________________ TestAsyncTokenizer.test_decode ________________________
async def functions are not natively supported.
You need to install a suitable plugin for your async framework, for example:
- anyio
- pytest-asyncio
- pytest-tornasync
- pytest-trio
- pytest-twisted
_____________________ TestAsyncTokenizer.test_large_batch ______________________
async def functions are not natively supported.
You need to install a suitable plugin for your async framework, for example:
- anyio
- pytest-asyncio
- pytest-tornasync
- pytest-trio
- pytest-twisted
_____________________ TestAsyncTokenizer.test_numpy_inputs _____________________
async def functions are not natively supported.
You need to install a suitable plugin for your async framework, for example:
- anyio
- pytest-asyncio
- pytest-tornasync
- pytest-trio
- pytest-twisted
________________ TestAsyncTokenizer.test_performance_comparison ________________
async def functions are not natively supported.
You need to install a suitable plugin for your async framework, for example:
- anyio
- pytest-asyncio
- pytest-tornasync
- pytest-trio
- pytest-twisted
_____________________ TestTrainFromIterators.test_datasets _____________________
self = <tests.documentation.test_tutorial_train_from_iterators.TestTrainFromIterators object at 0x731572794190>
@pytest.mark.network
def test_datasets(self):
tokenizer, trainer = self.get_tokenizer_trainer()
# In order to keep tests fast, we only use the first 100 examples
os.environ["TOKENIZERS_PARALLELISM"] = "true"
> dataset = datasets.load_dataset("wikitext", "wikitext-103-raw-v1", split="train[0:100]")
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
tests/documentation/test_tutorial_train_from_iterators.py:71:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
/usr/lib/python3.14/site-packages/datasets/load.py:1688: in load_dataset
builder_instance = load_dataset_builder(
/usr/lib/python3.14/site-packages/datasets/load.py:1315: in load_dataset_builder
dataset_module = dataset_module_factory(
/usr/lib/python3.14/site-packages/datasets/load.py:1207: in dataset_module_factory
raise e1 from None
/usr/lib/python3.14/site-packages/datasets/load.py:1182: in dataset_module_factory
).get_module()
^^^^^^^^^^^^
/usr/lib/python3.14/site-packages/datasets/load.py:598: in get_module
standalone_yaml_path = cached_path(
/usr/lib/python3.14/site-packages/datasets/utils/file_utils.py:180: in cached_path
).resolve_path(url_or_filename)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
/usr/lib/python3.14/site-packages/huggingface_hub/hf_file_system.py:305: in resolve_path
parsed = parse_hf_uri(f"{constants.HF_PROTOCOL}{path}")
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
/usr/lib/python3.14/site-packages/huggingface_hub/utils/_hf_uris.py:240: in parse_hf_uri
return _parse_repo_body(location, type_, raw=raw)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
location = 'wikitext@b08601e04326c79dfdd32d625aee71d232d685c3/.huggingface.yaml'
type_ = 'dataset'
def _parse_repo_body(
location: str,
type_: constants.HfUriType,
*,
raw: str,
) -> HfUri:
"""Parse the body of a repo URI: '<repo_id>[@<revision>][/<path>]'."""
location = location.strip("/")
if not location:
raise HfUriError(uri=raw, msg="Missing repository id.")
# The '@' separates the repo_id from the revision, but only when it
# appears right after 'namespace/name' (at most one '/' before it).
# An '@' deeper in the path (e.g. in a filename like 'file@1.txt') is literal.
at_idx = location.find("@")
revision: str | None
if at_idx == -1 or location[:at_idx].count("/") > 1:
# No '@' at all, or the '@' is past the repo_id portion (in a filename).
revision = None
parts = location.split("/", 2)
if len(parts) < 2:
raise HfUriError(uri=raw, msg=f"Repository id must be 'namespace/name', got '{location}'. ")
repo_id = f"{parts[0]}/{parts[1]}"
path_in_repo = parts[2] if len(parts) > 2 else ""
else:
repo_id = location[:at_idx]
rev_and_path = location[at_idx + 1 :]
if not repo_id:
raise HfUriError(uri=raw, msg="Missing repository id before '@'.")
if repo_id.count("/") != 1:
> raise HfUriError(uri=raw, msg=f"Repository id must be 'namespace/name', got '{repo_id}'.")
E huggingface_hub.errors.HfUriError: Invalid HF URI 'hf://datasets/wikitext@b08601e04326c79dfdd32d625aee71d232d685c3/.huggingface.yaml'. Repository id must be 'namespace/name', got 'wikitext'.
/usr/lib/python3.14/site-packages/huggingface_hub/utils/_hf_uris.py:417: HfUriError
=============================== warnings summary ===============================
tests/bindings/test_tokenizer.py:895
/builder/src/tokenizers-0.23.1/bindings/python/tests/bindings/test_tokenizer.py:895: PytestUnknownMarkWarning: Unknown pytest.mark.asyncio - is this a typo? You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/stable/how-to/mark.html
@pytest.mark.asyncio
tests/bindings/test_tokenizer.py:907
/builder/src/tokenizers-0.23.1/bindings/python/tests/bindings/test_tokenizer.py:907: PytestUnknownMarkWarning: Unknown pytest.mark.asyncio - is this a typo? You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/stable/how-to/mark.html
@pytest.mark.asyncio
tests/bindings/test_tokenizer.py:917
/builder/src/tokenizers-0.23.1/bindings/python/tests/bindings/test_tokenizer.py:917: PytestUnknownMarkWarning: Unknown pytest.mark.asyncio - is this a typo? You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/stable/how-to/mark.html
@pytest.mark.asyncio
tests/bindings/test_tokenizer.py:936
/builder/src/tokenizers-0.23.1/bindings/python/tests/bindings/test_tokenizer.py:936: PytestUnknownMarkWarning: Unknown pytest.mark.asyncio - is this a typo? You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/stable/how-to/mark.html
@pytest.mark.asyncio
tests/bindings/test_tokenizer.py:948
/builder/src/tokenizers-0.23.1/bindings/python/tests/bindings/test_tokenizer.py:948: PytestUnknownMarkWarning: Unknown pytest.mark.asyncio - is this a typo? You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/stable/how-to/mark.html
@pytest.mark.asyncio
tests/bindings/test_tokenizer.py:963
/builder/src/tokenizers-0.23.1/bindings/python/tests/bindings/test_tokenizer.py:963: PytestUnknownMarkWarning: Unknown pytest.mark.asyncio - is this a typo? You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/stable/how-to/mark.html
@pytest.mark.asyncio
tests/bindings/test_tokenizer.py:980
/builder/src/tokenizers-0.23.1/bindings/python/tests/bindings/test_tokenizer.py:980: PytestUnknownMarkWarning: Unknown pytest.mark.asyncio - is this a typo? You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/stable/how-to/mark.html
@pytest.mark.asyncio
tests/bindings/test_tokenizer.py:1000
/builder/src/tokenizers-0.23.1/bindings/python/tests/bindings/test_tokenizer.py:1000: PytestUnknownMarkWarning: Unknown pytest.mark.asyncio - is this a typo? You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/stable/how-to/mark.html
@pytest.mark.asyncio
tests/bindings/test_tokenizer.py:1015
/builder/src/tokenizers-0.23.1/bindings/python/tests/bindings/test_tokenizer.py:1015: PytestUnknownMarkWarning: Unknown pytest.mark.asyncio - is this a typo? You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/stable/how-to/mark.html
@pytest.mark.asyncio
tests/bindings/test_tokenizer.py:1028
/builder/src/tokenizers-0.23.1/bindings/python/tests/bindings/test_tokenizer.py:1028: PytestUnknownMarkWarning: Unknown pytest.mark.asyncio - is this a typo? You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/stable/how-to/mark.html
@pytest.mark.asyncio
tests/bindings/test_tokenizer.py:1046
/builder/src/tokenizers-0.23.1/bindings/python/tests/bindings/test_tokenizer.py:1046: PytestUnknownMarkWarning: Unknown pytest.mark.asyncio - is this a typo? You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/stable/how-to/mark.html
@pytest.mark.asyncio
tests/test_freethreaded.py:32
/builder/src/tokenizers-0.23.1/bindings/python/tests/test_freethreaded.py:32: PytestUnknownMarkWarning: Unknown pytest.mark.timeout - is this a typo? You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/stable/how-to/mark.html
pytestmark = pytest.mark.timeout(60) # any of these hanging means a deadlock
tests/bindings/test_tokenizer.py::TestTokenizer::test_decode_skip_special_tokens
tests/bindings/test_tokenizer.py::TestAsyncTokenizer::test_basic_encoding
/usr/lib/python3.14/site-packages/huggingface_hub/file_download.py:1855: DeprecationWarning: hf_xet.download_files() is deprecated. Use XetSession().new_file_download_group().start_download_file() instead.
xet_get(
tests/bindings/test_tokenizer.py::TestTokenizer::test_multiprocessing_with_parallelism
tests/bindings/test_tokenizer.py::TestTokenizer::test_multiprocessing_with_parallelism
tests/implementations/test_bert_wordpiece.py::TestBertWordPieceTokenizer::test_multiprocessing_with_parallelism
tests/implementations/test_bert_wordpiece.py::TestBertWordPieceTokenizer::test_multiprocessing_with_parallelism
tests/implementations/test_byte_level_bpe.py::TestByteLevelBPE::test_multiprocessing_with_parallelism
tests/implementations/test_byte_level_bpe.py::TestByteLevelBPE::test_multiprocessing_with_parallelism
tests/implementations/test_char_bpe.py::TestCharBPETokenizer::test_multiprocessing_with_parallelism
tests/implementations/test_char_bpe.py::TestCharBPETokenizer::test_multiprocessing_with_parallelism
/usr/lib/python3.14/multiprocessing/popen_fork.py:76: DeprecationWarning: This process (pid=4915) is multi-threaded, use of fork() may lead to deadlocks in the child.
self.pid = os.fork()
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
=========================== short test summary info ============================
FAILED tests/bindings/test_tokenizer.py::TestAsyncTokenizer::test_basic_encoding
FAILED tests/bindings/test_tokenizer.py::TestAsyncTokenizer::test_encode - Fa...
FAILED tests/bindings/test_tokenizer.py::TestAsyncTokenizer::test_with_special_tokens
FAILED tests/bindings/test_tokenizer.py::TestAsyncTokenizer::test_with_truncation_padding
FAILED tests/bindings/test_tokenizer.py::TestAsyncTokenizer::test_various_input_formats
FAILED tests/bindings/test_tokenizer.py::TestAsyncTokenizer::test_error_handling
FAILED tests/bindings/test_tokenizer.py::TestAsyncTokenizer::test_concurrency
FAILED tests/bindings/test_tokenizer.py::TestAsyncTokenizer::test_decode - Fa...
FAILED tests/bindings/test_tokenizer.py::TestAsyncTokenizer::test_large_batch
FAILED tests/bindings/test_tokenizer.py::TestAsyncTokenizer::test_numpy_inputs
FAILED tests/bindings/test_tokenizer.py::TestAsyncTokenizer::test_performance_comparison
FAILED tests/documentation/test_tutorial_train_from_iterators.py::TestTrainFromIterators::test_datasets
=========== 12 failed, 184 passed, 3 skipped, 22 warnings in 35.98s ============
Updating crates.io index
Downloading crates ...
Downloaded slab v0.4.12
Downloaded signal-hook-registry v1.4.8
Downloaded anstyle v1.0.14
Downloaded static_assertions v1.1.0
Downloaded smallvec v1.15.1
Downloaded strsim v0.11.1
Downloaded env_filter v1.0.1
Downloaded anstyle-query v1.1.5
Downloaded tokio-macros v2.7.0
Downloaded version_check v0.9.5
Downloaded unit-prefix v0.5.2
Downloaded env_logger v0.11.10
Downloaded target-lexicon v0.13.5
Downloaded zmij v1.0.21
Downloaded unicode-segmentation v1.13.3
Downloaded unicode_categories v0.1.1
Downloaded jiff v0.2.28
Downloaded unicode-normalization-alignments v0.1.12
Downloaded macro_rules_attribute-proc_macro v0.2.2
Downloaded colorchoice v1.0.5
Downloaded pyo3-macros v0.28.2
Downloaded rawpointer v0.2.1
Downloaded rustc-hash v2.1.2
Downloaded rand_chacha v0.9.0
Downloaded pkg-config v0.3.33
Downloaded ryu v1.0.23
Downloaded zerocopy v0.8.50
Downloaded serde_derive v1.0.228
Downloaded syn v2.0.117
Downloaded rayon-core v1.13.0
Downloaded mio v1.2.1
Downloaded rand v0.9.4
Downloaded itertools v0.14.0
Downloaded regex v1.12.3
Downloaded rayon v1.12.0
Downloaded portable-atomic v1.13.1
Downloaded ndarray v0.17.2
Downloaded memchr v2.8.1
Downloaded compact_str v0.9.1
Downloaded spm_precompiled v0.1.4
Downloaded ndarray v0.16.1
Downloaded pyo3-macros-backend v0.28.2
Downloaded nom v7.1.3
Downloaded regex-syntax v0.8.10
Downloaded pyo3-ffi v0.28.2
Downloaded proc-macro2 v1.0.106
Downloaded log v0.4.30
Downloaded rustix v1.1.4
Downloaded getrandom v0.4.2
Downloaded base64 v0.13.1
Downloaded numpy v0.28.0
Downloaded matrixmultiply v0.3.10
Downloaded getrandom v0.3.4
Downloaded console v0.16.3
Downloaded bitflags v2.11.1
Downloaded unicode-width v0.2.2
Downloaded serde_json v1.0.150
Downloaded serde_core v1.0.228
Downloaded regex-automata v0.4.14
Downloaded serde v1.0.228
Downloaded rand_core v0.9.5
Downloaded tokio v1.52.3
Downloaded onig_sys v69.9.3
Downloaded quote v1.0.45
Downloaded libc v0.2.186
Downloaded pyo3-build-config v0.28.2
Downloaded ppv-lite86 v0.2.21
Downloaded num-complex v0.4.6
Downloaded indicatif v0.18.4
Downloaded pin-project-lite v0.2.17
Downloaded paste v1.0.15
Downloaded fastrand v2.4.1
Downloaded onig v6.5.3
Downloaded once_cell v1.21.4
Downloaded num-integer v0.1.46
Downloaded macro_rules_attribute v0.2.2
Downloaded is_terminal_polyfill v1.70.2
Downloaded crossbeam-deque v0.8.6
Downloaded ident_case v1.0.1
Downloaded castaway v0.2.4
Downloaded itoa v1.0.18
Downloaded futures-task v0.3.32
Downloaded futures-macro v0.3.32
Downloaded fnv v1.0.7
Downloaded find-msvc-tools v0.1.9
Downloaded pyo3 v0.28.2
Downloaded rustversion v1.0.22
Downloaded rayon-cond v0.4.0
Downloaded pyo3-async-runtimes v0.28.0
Downloaded num-traits v0.2.19
Downloaded monostate-impl v0.1.18
Downloaded monostate v0.1.18
Downloaded heck v0.5.0
Downloaded futures-util v0.3.32
Downloaded futures-channel v0.3.32
Downloaded futures-core v0.3.32
Downloaded minimal-lexical v0.2.1
Downloaded unicode-ident v1.0.24
Downloaded thiserror-impl v2.0.18
Downloaded tempfile v3.27.0
Downloaded thiserror v2.0.18
Downloaded esaxx-rs v0.1.10
Downloaded either v1.16.0
Downloaded derive_builder v0.20.2
Downloaded daachorse v1.0.1
Downloaded cc v1.2.63
Downloaded dary_heap v0.3.9
Downloaded darling_core v0.20.11
Downloaded ahash v0.8.12
Downloaded derive_builder_core v0.20.2
Downloaded darling_macro v0.20.11
Downloaded darling v0.20.11
Downloaded crossbeam-utils v0.8.21
Downloaded crossbeam-epoch v0.9.18
Downloaded cfg-if v1.0.4
Downloaded autocfg v1.5.1
Downloaded aho-corasick v1.1.4
Downloaded utf8parse v0.2.2
Downloaded errno v0.3.14
Downloaded derive_builder_macro v0.20.2
Downloaded shlex v2.0.1
Downloaded anstyle-parse v1.0.0
Downloaded anstream v1.0.0
Downloaded linux-raw-sys v0.12.1
Compiling proc-macro2 v1.0.106
Compiling unicode-ident v1.0.24
Compiling quote v1.0.45
Compiling libc v0.2.186
Compiling target-lexicon v0.13.5
Compiling autocfg v1.5.1
Compiling serde_core v1.0.228
Compiling pyo3-build-config v0.28.2
Compiling syn v2.0.117
Compiling cfg-if v1.0.4
Compiling shlex v2.0.1
Compiling memchr v2.8.1
Compiling find-msvc-tools v0.1.9
Compiling cc v1.2.63
Compiling crossbeam-utils v0.8.21
Compiling num-traits v0.2.19
Compiling once_cell v1.21.4
Compiling zerocopy v0.8.50
Compiling serde v1.0.228
Compiling fnv v1.0.7
Compiling ident_case v1.0.1
Compiling strsim v0.11.1
Compiling getrandom v0.3.4
Compiling pyo3-ffi v0.28.2
Compiling pyo3-macros-backend v0.28.2
Compiling crossbeam-epoch v0.9.18
Compiling darling_core v0.20.11
Compiling aho-corasick v1.1.4
Compiling serde_derive v1.0.228
Compiling darling_macro v0.20.11
Compiling matrixmultiply v0.3.10
Compiling pkg-config v0.3.33
Compiling rustversion v1.0.22
Compiling regex-syntax v0.8.10
Compiling heck v0.5.0
Compiling rayon-core v1.13.0
Compiling onig_sys v69.9.3
Compiling darling v0.20.11
Compiling crossbeam-deque v0.8.6
Compiling pyo3 v0.28.2
Compiling either v1.16.0
Compiling bitflags v2.11.1
Compiling rawpointer v0.2.1
Compiling portable-atomic v1.13.1
Compiling regex-automata v0.4.14
Compiling paste v1.0.15
Compiling version_check v0.9.5
Compiling zmij v1.0.21
Compiling ahash v0.8.12
Compiling derive_builder_core v0.20.2
Compiling regex v1.12.3
Compiling ppv-lite86 v0.2.21
Compiling rand_core v0.9.5
Compiling pyo3-macros v0.28.2
Compiling num-integer v0.1.46
Compiling num-complex v0.4.6
Compiling esaxx-rs v0.1.10
Compiling errno v0.3.14
Compiling futures-core v0.3.32
Compiling thiserror v2.0.18
Compiling unicode-width v0.2.2
Compiling minimal-lexical v0.2.1
Compiling rustix v1.1.4
Compiling getrandom v0.4.2
Compiling utf8parse v0.2.2
Compiling serde_json v1.0.150
Compiling itoa v1.0.18
Compiling pin-project-lite v0.2.17
Compiling log v0.4.30
Compiling anstyle-parse v1.0.0
Compiling nom v7.1.3
Compiling console v0.16.3
Compiling signal-hook-registry v1.4.8
Compiling rand_chacha v0.9.0
Compiling castaway v0.2.4
Compiling derive_builder_macro v0.20.2
Compiling rayon v1.12.0
Compiling itertools v0.14.0
Compiling monostate-impl v0.1.18
Compiling futures-macro v0.3.32
Compiling tokio-macros v2.7.0
Compiling thiserror-impl v2.0.18
Compiling numpy v0.28.0
Compiling mio v1.2.1
Compiling is_terminal_polyfill v1.70.2
Compiling base64 v0.13.1
Compiling unit-prefix v0.5.2
Compiling colorchoice v1.0.5
Compiling slab v0.4.12
Compiling static_assertions v1.1.0
Compiling smallvec v1.15.1
Compiling anstyle-query v1.1.5
Compiling futures-task v0.3.32
Compiling unicode-segmentation v1.13.3
Compiling linux-raw-sys v0.12.1
Compiling anstyle v1.0.14
Compiling macro_rules_attribute-proc_macro v0.2.2
Compiling ryu v1.0.23
Compiling compact_str v0.9.1
Compiling spm_precompiled v0.1.4
Compiling macro_rules_attribute v0.2.2
Compiling anstream v1.0.0
Compiling futures-util v0.3.32
Compiling unicode-normalization-alignments v0.1.12
Compiling onig v6.5.3
Compiling indicatif v0.18.4
Compiling tokio v1.52.3
Compiling rayon-cond v0.4.0
Compiling monostate v0.1.18
Compiling derive_builder v0.20.2
Compiling rand v0.9.4
Compiling env_filter v1.0.1
Compiling futures-channel v0.3.32
Compiling ndarray v0.17.2
Compiling dary_heap v0.3.9
Compiling jiff v0.2.28
Compiling daachorse v1.0.1
Compiling fastrand v2.4.1
Compiling rustc-hash v2.1.2
Compiling unicode_categories v0.1.1
Compiling tokenizers v0.23.1 (/builder/src/tokenizers-0.23.1/tokenizers)
Compiling tempfile v3.27.0
Compiling pyo3-async-runtimes v0.28.0
Compiling env_logger v0.11.10
Compiling ndarray v0.16.1
Compiling tokenizers-python v0.23.1 (/builder/src/tokenizers-0.23.1/bindings/python)
Finished `test` profile [unoptimized + debuginfo] target(s) in 51.48s
Running unittests src/lib.rs (target/debug/deps/tokenizers-d0ff840e11124795)
running 20 tests
test models::test::serialize ... ok
test normalizers::test::deserialize_sequence ... ok
test models::test::get_subtype ... ok
test decoders::test::serialize ... ok
test decoders::test::get_subtype ... ok
test normalizers::test::get_subtype ... ok
test normalizers::test::serialize ... ok
test pre_tokenizers::test::get_subtype ... ok
test pre_tokenizers::test::serialize ... ok
test processors::test::get_subtype ... ok
test processors::test::serialize ... ok
test tokenizer::test::serde_pyo3 ... ok
test utils::serde_pyo3::test_basic ... ok
test utils::serde_pyo3::test_enum ... ok
test trainers::tests::get_subtype ... ok
test utils::serde_pyo3::test_enum_untagged ... ok
test utils::serde_pyo3::test_flatten ... ok
test utils::serde_pyo3::test_struct ... ok
test utils::serde_pyo3::test_struct_tagged ... ok
test tokenizer::test::serialize ... ok
test result: ok. 20 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.01s
==> Entering fakeroot environment...
==> Starting package()...
==> Tidying install...
-> Removing libtool files...
-> Removing static library files...
-> Purging unwanted files...
-> Stripping unneeded symbols from binaries and libraries...
-> Compressing man and info pages...
==> Checking for packaging issues...
==> WARNING: Package contains reference to $srcdir
usr/lib/python3.14/site-packages/tokenizers/tokenizers.abi3.so
usr/lib/python3.14/site-packages/tokenizers-0.23.1.dist-info/sboms/tokenizers-python.cyclonedx.json
==> Creating package "python-tokenizers"...
-> Generating .PKGINFO file...
-> Generating .BUILDINFO file...
-> Generating .MTREE file...
-> Compressing package...
==> Leaving fakeroot environment.
==> Finished making: python-tokenizers 0.23.1-1 (Tue Jun 2 07:12:48 2026)
==> Checking runtime dependencies...
==> Checking buildtime dependencies...
==> Installing missing dependencies...
resolving dependencies...
looking for conflicting packages...
Packages (103) abseil-cpp-20260107.1-1 apache-orc-2.3.0-3 arrow-24.0.0-1 aws-c-auth-0.10.1-1 aws-c-cal-0.9.13-1 aws-c-common-0.12.6-1 aws-c-compression-0.3.2-1 aws-c-event-stream-0.7.0-1 aws-c-http-0.10.14-1 aws-c-io-0.26.3-1 aws-c-mqtt-0.15.2-1 aws-c-s3-0.12.2-1 aws-c-sdkutils-0.2.4-1 aws-checksums-0.2.10-1 aws-crt-cpp-0.38.5-1 aws-sdk-cpp-core-1.11.792-1 aws-sdk-cpp-iam-1.11.792-1 aws-sdk-cpp-s3-1.11.792-1 blas-3.12.1-2 c-ares-1.34.6-1 cblas-3.12.1-2 compiler-rt-22.1.6-1 gflags-2.2.2-6 google-glog-0.7.1-2 grpc-1.80.0-1 gtest-1.17.0-2 hdf5-2.1.1-1 lapack-3.12.1-2 libaec-1.1.6-1 libgit2-1:1.9.4-1 liblzf-3.6-5 lld-22.1.6-1 llhttp-9.3.1-1 llvm-libs-22.1.6-1 maturin-1.13.3-1 protobuf-34.1-1 python-aiohappyeyeballs-2.6.1-4 python-aiohttp-3.13.5-1 python-aiosignal-1.4.0-3 python-annotated-doc-0.0.4-2 python-annotated-types-0.7.0-3 python-anyio-4.13.0-1 python-autocommand-2.2.2-9 python-certifi-2026.05.20-1 python-dateutil-2.9.0-8 python-dill-0.4.1-1 python-filelock-3.29.0-1 python-frozenlist-1.8.0-2 python-fsspec-2026.4.0-1 python-h11-0.16.0-2 python-h5py-3.16.0-1 python-hf-xet-1.5.0-1 python-httpcore-1.0.9-3 python-httpx-0.28.1-7 python-huggingface-hub-1:1.16.0-1 python-idna-3.17-1 python-iniconfig-2.3.0-1 python-jaraco.collections-5.1.0-3 python-jaraco.context-6.1.2-1 python-jaraco.functools-4.1.0-3 python-jaraco.text-4.0.0-4 python-markdown-it-py-4.0.0-2 python-mdurl-0.1.2-9 python-more-itertools-11.1.0-1 python-multidict-6.7.1-1 python-multiprocess-0.70.19-1 python-pandas-2.3.3-2 python-pkg_resources-81.0.0-1 python-platformdirs-4.10.0-1 python-pluggy-1.6.0-3 python-propcache-0.4.1-2 python-pydantic-2.13.4-1 python-pydantic-core-3:2.46.4-1 python-pygments-2.20.0-1 python-pyproject-hooks-1.2.0-6 python-pytz-2026.1-1 python-rich-15.0.0-1 python-semantic-version-2.10.0-9 python-setuptools-1:82.0.1-1 python-shellingham-1.5.4-4 python-six-1.17.0-3 python-typer-0.26.0-1 python-typing-inspection-0.4.2-2 python-typing_extensions-4.15.0-3 python-xxhash-3.7.0-1 python-yarl-1.23.0-1 re2-2:2025.11.05-3 rust-1:1.95.0-1 s2n-tls-1.7.2-1 snappy-1.2.2-3 thrift-0.22.0-6 clang-22.1.6-1 python-build-1.4.3-1 python-datasets-4.8.5-1 python-installer-1.0.0-1 python-maturin-1.13.3-1 python-numpy-2.4.6-1 python-pyarrow-24.0.0-1 python-pytest-1:9.0.3-1 python-requests-2.34.2-1 python-setuptools-rust-1.12.1-1 python-wheel-0.47.0-1 rust-bindgen-0.72.1-2
Total Download Size: 146.07 MiB
Total Installed Size: 1234.10 MiB
:: Proceed with installation? [Y/n]
:: Retrieving packages...
clang-22.1.6-1-x86_64 downloading...
python-pandas-2.3.3-2-x86_64 downloading...
arrow-24.0.0-1-x86_64 downloading...
python-numpy-2.4.6-1-x86_64 downloading...
maturin-1.13.3-1-x86_64 downloading...
grpc-1.80.0-1-x86_64 downloading...
lapack-3.12.1-2-x86_64 downloading...
compiler-rt-22.1.6-1-x86_64 downloading...
hdf5-2.1.1-1-x86_64 downloading...
python-pyarrow-24.0.0-1-x86_64 downloading...
python-hf-xet-1.5.0-1-x86_64 downloading...
lld-22.1.6-1-x86_64 downloading...
rust-bindgen-0.72.1-2-x86_64 downloading...
thrift-0.22.0-6-x86_64 downloading...
aws-sdk-cpp-s3-1.11.792-1-x86_64 downloading...
python-pydantic-core-3:2.46.4-1-x86_64 downloading...
python-h5py-3.16.0-1-x86_64 downloading...
python-huggingface-hub-1:1.16.0-1-any downloading...
libgit2-1:1.9.4-1-x86_64 downloading...
python-datasets-4.8.5-1-any downloading...
python-pydantic-2.13.4-1-any downloading...
python-aiohttp-3.13.5-1-x86_64 downloading...
aws-sdk-cpp-iam-1.11.792-1-x86_64 downloading...
aws-sdk-cpp-core-1.11.792-1-x86_64 downloading...
python-rich-15.0.0-1-any downloading...
apache-orc-2.3.0-3-x86_64 downloading...
python-fsspec-2026.4.0-1-any downloading...
python-httpx-0.28.1-7-any downloading...
python-multiprocess-0.70.19-1-any downloading...
s2n-tls-1.7.2-1-x86_64 downloading...
python-dateutil-2.9.0-8-any downloading...
aws-crt-cpp-0.38.5-1-x86_64 downloading...
python-anyio-4.13.0-1-any downloading...
python-typer-0.26.0-1-any downloading...
blas-3.12.1-2-x86_64 downloading...
python-dill-0.4.1-1-any downloading...
re2-2:2025.11.05-3-x86_64 downloading...
aws-c-common-0.12.6-1-x86_64 downloading...
aws-c-http-0.10.14-1-x86_64 downloading...
aws-c-mqtt-0.15.2-1-x86_64 downloading...
gflags-2.2.2-6-x86_64 downloading...
python-markdown-it-py-4.0.0-2-any downloading...
python-requests-2.34.2-1-any downloading...
aws-c-io-0.26.3-1-x86_64 downloading...
google-glog-0.7.1-2-x86_64 downloading...
python-httpcore-1.0.9-3-any downloading...
aws-c-s3-0.12.2-1-x86_64 downloading...
python-yarl-1.23.0-1-x86_64 downloading...
aws-c-auth-0.10.1-1-x86_64 downloading...
python-idna-3.17-1-any downloading...
aws-checksums-0.2.10-1-x86_64 downloading...
python-filelock-3.29.0-1-any downloading...
python-multidict-6.7.1-1-x86_64 downloading...
python-h11-0.16.0-2-any downloading...
python-setuptools-rust-1.12.1-1-any downloading...
cblas-3.12.1-2-x86_64 downloading...
python-pytz-2026.1-1-any downloading...
python-frozenlist-1.8.0-2-x86_64 downloading...
python-propcache-0.4.1-2-x86_64 downloading...
aws-c-event-stream-0.7.0-1-x86_64 downloading...
aws-c-sdkutils-0.2.4-1-x86_64 downloading...
aws-c-cal-0.9.13-1-x86_64 downloading...
python-semantic-version-2.10.0-9-any downloading...
snappy-1.2.2-3-x86_64 downloading...
python-six-1.17.0-3-any downloading...
python-aiohappyeyeballs-2.6.1-4-any downloading...
python-typing-inspection-0.4.2-2-any downloading...
libaec-1.1.6-1-x86_64 downloading...
python-maturin-1.13.3-1-x86_64 downloading...
python-annotated-types-0.7.0-3-any downloading...
python-shellingham-1.5.4-4-any downloading...
python-mdurl-0.1.2-9-any downloading...
python-xxhash-3.7.0-1-x86_64 downloading...
python-aiosignal-1.4.0-3-any downloading...
aws-c-compression-0.3.2-1-x86_64 downloading...
python-annotated-doc-0.0.4-2-any downloading...
python-certifi-2026.05.20-1-any downloading...
checking keyring...
checking package integrity...
loading package files...
checking for file conflicts...
checking available disk space...
:: Processing package changes...
installing llvm-libs...
installing compiler-rt...
installing clang...
Optional dependencies for clang
openmp: OpenMP support in clang with -fopenmp
python: for scan-view and git-clang-format [installed]
llvm: referenced by some clang headers
installing rust-bindgen...
installing python-pyproject-hooks...
installing python-build...
Optional dependencies for python-build
python-pip: to use as the Python package installer (default)
python-uv: to use as the Python package installer
python-virtualenv: to use virtualenv for build isolation
installing python-installer...
installing llhttp...
installing libgit2...
installing lld...
installing rust...
Optional dependencies for rust
gdb: rust-gdb script [installed]
lldb: rust-lldb script
installing maturin...
installing python-maturin...
installing python-more-itertools...
installing python-jaraco.functools...
installing python-jaraco.context...
installing python-autocommand...
installing python-jaraco.text...
Optional dependencies for python-jaraco.text
python-inflect: for show-newlines script
installing python-jaraco.collections...
installing python-platformdirs...
installing python-wheel...
Optional dependencies for python-wheel
python-keyring: for wheel.signatures
python-xdg: for wheel.signatures
python-setuptools: for legacy bdist_wheel subcommand [pending]
installing python-typing_extensions...
installing python-pkg_resources...
installing python-setuptools...
installing python-semantic-version...
installing python-setuptools-rust...
installing python-aiohappyeyeballs...
installing python-frozenlist...
installing python-aiosignal...
installing python-multidict...
installing python-propcache...
installing python-idna...
installing python-yarl...
installing python-aiohttp...
Optional dependencies for python-aiohttp
gunicorn: to deploy using Gunicorn
python-aiodns: for fast DNS resolving
python-brotli: for Brotli transfer-encodings support
installing python-dill...
Optional dependencies for python-dill
python-objgraph: graph support
installing python-filelock...
installing python-fsspec...
Optional dependencies for python-fsspec
python-aiohttp: HTTP support [installed]
python-distributed: Dask support
python-libarchive-c: archives support
python-lz4: LZ4 compression support
python-paramiko: SFTP support
python-pyarrow: Arrow/Parquet support [pending]
python-pygit2: git support
python-requests: web protocols support [pending]
python-smbprotocol: SMB support
python-snappy: snappy compression support
python-tqdm: progress bar support [installed]
installing libaec...
installing hdf5...
installing liblzf...
installing blas...
installing cblas...
installing lapack...
installing python-numpy...
Optional dependencies for python-numpy
blas-openblas: faster linear algebra
installing python-h5py...
installing python-certifi...
installing python-h11...
installing python-httpcore...
Optional dependencies for python-httpcore
python-h2: for HTTP/2 support
python-socksio: for SOCKS support
python-anyio: for asyncio backend [pending]
python-trio: for trio backend
python-sniffio: for async support
installing python-anyio...
Optional dependencies for python-anyio
python-trio: trio backend
python-outcome: trio backend
python-uvloop: use uvloop for asyncio backend
python-pytest: pytest plugin [pending]
installing python-httpx...
Optional dependencies for python-httpx
python-brotli: for brotli response decompression
python-brotlicffi: for brotli response decompression
python-zstandard: for zstd response decompression
python-h2: HTTP/2 support
python-socksio: SOCKS proxy support
python-click: command line client support [installed]
python-rich: command line client support [pending]
python-pygments: command line client support [pending]
python-trio: alternative async library
installing python-hf-xet...
installing python-annotated-types...
installing python-typing-inspection...
installing python-pydantic-core...
installing python-pydantic...
Optional dependencies for python-pydantic
mypy: for type validation with mypy
python-dotenv: for .env file support
python-email-validator: for email validation
python-hypothesis: for hypothesis plugin when using legacy v1
installing python-mdurl...
installing python-markdown-it-py...
Optional dependencies for python-markdown-it-py
python-mdit_py_plugins: core plugins
python-linkify-it-py: linkify extension
installing python-pygments...
installing python-rich...
installing python-shellingham...
installing python-annotated-doc...
installing python-typer...
installing python-huggingface-hub...
Optional dependencies for python-huggingface-hub
python-authlib: for OAuth
python-fastapi: for OAuth
python-itsdangerous: for OAuth
python-gradio: for the webhooks server
python-duckdb: for hf datasets sql
python-graphviz
python-jinja [installed]
python-mcp
python-pillow
python-pydot
python-pytorch
python-safetensors
python-fastai
installing python-multiprocess...
installing python-six...
installing python-dateutil...
installing python-pytz...
installing python-pandas...
Optional dependencies for python-pandas
python-pandas-datareader: pandas.io.data replacement (recommended)
python-numexpr: accelerating certain numerical operations (recommended)
python-bottleneck: accelerating certain types of nan evaluations (recommended)
python-matplotlib: plotting
python-jinja: conditional formatting with DataFrame.style [installed]
python-tabulate: printing in Markdown-friendly format
python-scipy: miscellaneous statistical functions
python-numba: alternative execution engine
python-xarray: pandas-like API for N-dimensional data
python-xlrd: Excel XLS input
python-xlwt: Excel XLS output
python-openpyxl: Excel XLSX input/output
python-xlsxwriter: alternative Excel XLSX output
python-beautifulsoup4: read_html function (in any case)
python-html5lib: read_html function (and/or python-lxml)
python-lxml: read_xml, to_xml and read_html function (and/or python-html5lib)
python-sqlalchemy: SQL database support
python-psycopg2: PostgreSQL engine for sqlalchemy
python-pymysql: MySQL engine for sqlalchemy
python-pytables: HDF5-based reading / writing
python-blosc: for msgpack compression using blosc
zlib: compression for msgpack [installed]
python-pyarrow: Parquet, ORC and feather reading/writing [pending]
python-fsspec: handling files aside from local and HTTP [installed]
python-qtpy: read_clipboard function (only one needed)
xclip: read_clipboard function (only one needed)
xsel: read_clipboard function (only one needed)
python-brotli: Brotli compression
python-snappy: Snappy compression
python-zstandard: Zstandard (zstd) compression
installing gtest...
Optional dependencies for gtest
python: gmock generator [installed]
installing abseil-cpp...
installing protobuf...
installing snappy...
installing apache-orc...
installing aws-c-common...
installing aws-c-cal...
installing aws-c-compression...
installing s2n-tls...
installing aws-c-io...
installing aws-c-http...
installing aws-c-sdkutils...
installing aws-c-auth...
installing aws-checksums...
installing aws-c-event-stream...
installing aws-c-mqtt...
installing aws-c-s3...
installing aws-crt-cpp...
installing aws-sdk-cpp-core...
installing aws-sdk-cpp-iam...
installing aws-sdk-cpp-s3...
installing gflags...
installing google-glog...
installing c-ares...
installing re2...
installing grpc...
installing thrift...
Optional dependencies for thrift
qt5-base: TQTcpServer (Qt5) support
installing arrow...
installing python-pyarrow...
Optional dependencies for python-pyarrow
python-cffi: interact with C code
python-pandas: Pandas integration [installed]
python-fsspec: Filesystem Spec support [installed]
installing python-requests...
Optional dependencies for python-requests
python-chardet: alternative character encoding library
python-pysocks: SOCKS proxy support
installing python-xxhash...
installing python-datasets...
Optional dependencies for python-datasets
python-librosa: Audio datasets
python-soxr: Audio datasets
python-torchcodec: Audio datasets
python-pillow: Vision datasets
python-tensorflow: TensorFlow support
python-pytorch: PyTorch support
installing python-iniconfig...
installing python-pluggy...
installing python-pytest...
:: Running post-transaction hooks...
(1/1) Arming ConditionNeedsUpdate...
==> Retrieving sources...
-> Downloading tokenizers-0.23.1.tar.gz...
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0
100 1.53M 0 1.53M 0 0 2.28M 0 0
100 1.53M 0 1.53M 0 0 2.28M 0 0
100 1.53M 0 1.53M 0 0 2.28M 0 0
-> Downloading norvig-big.txt...
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 0
56 6.18M 56 3.48M 0 0 2.53M 0 00:02 00:01 00:01 3.32M
100 6.18M 100 6.18M 0 0 4.18M 0 00:01 00:01 3.32M
100 6.18M 100 6.18M 0 0 4.18M 0 00:01 00:01 3.32M
100 6.18M 100 6.18M 0 0 4.18M 0 00:01 00:01 3.32M
-> Downloading roberta.json...
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 0
100 84 100 84 0 0 596 0 0
100 84 100 84 0 0 595 0 0
0 0 0 0 0 0 0 0 0
100 1.29M 100 1.29M 0 0 2.27M 0 0
100 1.29M 100 1.29M 0 0 2.27M 0 0
100 1.29M 100 1.29M 0 0 2.27M 0 0
==> WARNING: Skipping verification of source file PGP signatures.
==> Validating source files with sha256sums...
tokenizers-0.23.1.tar.gz ... Passed
norvig-big.txt ... Passed
roberta.json ... Passed
==> Extracting sources...
-> Extracting tokenizers-0.23.1.tar.gz with bsdtar
==> Starting prepare()...
Updating crates.io index
Locking 22 packages to latest compatible versions
Updating autocfg v1.5.0 -> v1.5.1
Updating bumpalo v3.20.2 -> v3.20.3
Updating cc v1.2.61 -> v1.2.63
Updating compact_str v0.9.0 -> v0.9.1
Updating either v1.15.0 -> v1.16.0
Updating hashbrown v0.17.0 -> v0.17.1
Updating jiff v0.2.24 -> v0.2.28
Updating jiff-static v0.2.24 -> v0.2.28
Updating js-sys v0.3.95 -> v0.3.99
Updating log v0.4.29 -> v0.4.30
Updating memchr v2.8.0 -> v2.8.1
Updating mio v1.2.0 -> v1.2.1
Updating serde_json v1.0.149 -> v1.0.150
Updating shlex v1.3.0 -> v2.0.1
Updating tokio v1.52.1 -> v1.52.3
Updating unicode-segmentation v1.13.2 -> v1.13.3
Updating wasm-bindgen v0.2.118 -> v0.2.122
Updating wasm-bindgen-macro v0.2.118 -> v0.2.122
Updating wasm-bindgen-macro-support v0.2.118 -> v0.2.122
Updating wasm-bindgen-shared v0.2.118 -> v0.2.122
Updating zerocopy v0.8.48 -> v0.8.50
Updating zerocopy-derive v0.8.48 -> v0.8.50
note: pass `--verbose` to see 6 unchanged dependencies behind latest
Downloading crates ...
Downloaded rustc-hash v2.1.2
Downloaded futures-macro v0.3.32
Downloaded tokio-macros v2.7.0
Downloaded quote v1.0.45
Downloaded numpy v0.28.0
Downloaded futures-core v0.3.32
Downloaded unicode-ident v1.0.24
Downloaded futures-channel v0.3.32
Downloaded proc-macro2 v1.0.106
Downloaded futures-task v0.3.32
Downloaded heck v0.5.0
Downloaded anstyle v1.0.14
Downloaded env_filter v1.0.1
Downloaded colorchoice v1.0.5
Downloaded anstyle-query v1.1.5
Downloaded utf8parse v0.2.2
Downloaded num-complex v0.4.6
Downloaded matrixmultiply v0.3.10
Downloaded derive_builder_macro v0.20.2
Downloaded itoa v1.0.18
Downloaded is_terminal_polyfill v1.70.2
Downloaded rawpointer v0.2.1
Downloaded strsim v0.11.1
Downloaded anstyle-parse v1.0.0
Downloaded num-integer v0.1.46
Downloaded anstream v1.0.0
Downloaded cfg-if v1.0.4
Downloaded env_logger v0.11.10
Downloaded autocfg v1.5.1
Downloaded macro_rules_attribute v0.2.2
Downloaded macro_rules_attribute-proc_macro v0.2.2
Downloaded num-traits v0.2.19
Downloaded errno v0.3.14
Downloaded fnv v1.0.7
Downloaded rayon-cond v0.4.0
Downloaded monostate v0.1.18
Downloaded version_check v0.9.5
Downloaded castaway v0.2.4
Downloaded ndarray v0.17.2
Downloaded monostate-impl v0.1.18
Downloaded unit-prefix v0.5.2
Downloaded ident_case v1.0.1
Downloaded ndarray v0.16.1
Downloaded jiff v0.2.28
Downloaded slab v0.4.12
Downloaded pyo3-build-config v0.28.2
Downloaded rustversion v1.0.22
Downloaded darling v0.20.11
Downloaded console v0.16.3
Downloaded bitflags v2.11.1
Downloaded once_cell v1.21.4
Downloaded rand_chacha v0.9.0
Downloaded rand_core v0.9.5
Downloaded crossbeam-utils v0.8.21
Downloaded zmij v1.0.21
Downloaded thiserror-impl v2.0.18
Downloaded thiserror v2.0.18
Downloaded pin-project-lite v0.2.17
Downloaded signal-hook-registry v1.4.8
Downloaded pyo3-macros v0.28.2
Downloaded target-lexicon v0.13.5
Downloaded fastrand v2.4.1
Downloaded static_assertions v1.1.0
Downloaded dary_heap v0.3.9
Downloaded darling_macro v0.20.11
Downloaded derive_builder_core v0.20.2
Downloaded find-msvc-tools v0.1.9
Downloaded pkg-config v0.3.33
Downloaded onig v6.5.3
Downloaded paste v1.0.15
Downloaded ppv-lite86 v0.2.21
Downloaded either v1.16.0
Downloaded crossbeam-deque v0.8.6
Downloaded smallvec v1.15.1
Downloaded shlex v2.0.1
Downloaded ryu v1.0.23
Downloaded derive_builder v0.20.2
Downloaded crossbeam-epoch v0.9.18
Downloaded ahash v0.8.12
Downloaded tempfile v3.27.0
Downloaded pyo3-async-runtimes v0.28.0
Downloaded getrandom v0.4.2
Downloaded base64 v0.13.1
Downloaded getrandom v0.3.4
Downloaded daachorse v1.0.1
Downloaded log v0.4.30
Downloaded darling_core v0.20.11
Downloaded serde_core v1.0.228
Downloaded serde_derive v1.0.228
Downloaded rayon-core v1.13.0
Downloaded cc v1.2.63
Downloaded indicatif v0.18.4
Downloaded pyo3-macros-backend v0.28.2
Downloaded unicode_categories v0.1.1
Downloaded compact_str v0.9.1
Downloaded unicode-normalization-alignments v0.1.12
Downloaded pyo3-ffi v0.28.2
Downloaded minimal-lexical v0.2.1
Downloaded rand v0.9.4
Downloaded memchr v2.8.1
Downloaded serde v1.0.228
Downloaded nom v7.1.3
Downloaded unicode-segmentation v1.13.3
Downloaded mio v1.2.1
Downloaded itertools v0.14.0
Downloaded futures-util v0.3.32
Downloaded portable-atomic v1.13.1
Downloaded rayon v1.12.0
Downloaded esaxx-rs v0.1.10
Downloaded aho-corasick v1.1.4
Downloaded syn v2.0.117
Downloaded unicode-width v0.2.2
Downloaded regex v1.12.3
Downloaded zerocopy v0.8.50
Downloaded serde_json v1.0.150
Downloaded regex-syntax v0.8.10
Downloaded rustix v1.1.4
Downloaded spm_precompiled v0.1.4
Downloaded regex-automata v0.4.14
Downloaded onig_sys v69.9.3
Downloaded tokio v1.52.3
Downloaded libc v0.2.186
Downloaded pyo3 v0.28.2
Downloaded linux-raw-sys v0.12.1
==> Starting build()...
Compiling proc-macro2 v1.0.106
Compiling quote v1.0.45
Compiling unicode-ident v1.0.24
Compiling target-lexicon v0.13.5
Compiling libc v0.2.186
Compiling autocfg v1.5.1
Compiling serde_core v1.0.228
Compiling pyo3-build-config v0.28.2
Compiling syn v2.0.117
Compiling memchr v2.8.1
Compiling find-msvc-tools v0.1.9
Compiling shlex v2.0.1
Compiling cc v1.2.63
Compiling crossbeam-utils v0.8.21
Compiling num-traits v0.2.19
Compiling ident_case v1.0.1
Compiling getrandom v0.3.4
Compiling zerocopy v0.8.50
Compiling strsim v0.11.1
Compiling cfg-if v1.0.4
Compiling serde v1.0.228
Compiling fnv v1.0.7
Compiling darling_core v0.20.11
Compiling pyo3-macros-backend v0.28.2
Compiling pyo3-ffi v0.28.2
Compiling once_cell v1.21.4
Compiling crossbeam-epoch v0.9.18
Compiling serde_derive v1.0.228
Compiling aho-corasick v1.1.4
Compiling darling_macro v0.20.11
Compiling matrixmultiply v0.3.10
Compiling pkg-config v0.3.33
Compiling heck v0.5.0
Compiling rayon-core v1.13.0
Compiling regex-syntax v0.8.10
Compiling rustversion v1.0.22
Compiling onig_sys v69.9.3
Compiling darling v0.20.11
Compiling regex-automata v0.4.14
Compiling crossbeam-deque v0.8.6
Compiling pyo3 v0.28.2
Compiling version_check v0.9.5
Compiling either v1.16.0
Compiling zmij v1.0.21
Compiling portable-atomic v1.13.1
Compiling rawpointer v0.2.1
Compiling paste v1.0.15
Compiling ahash v0.8.12
Compiling pyo3-macros v0.28.2
Compiling regex v1.12.3
Compiling derive_builder_core v0.20.2
Compiling ppv-lite86 v0.2.21
Compiling rand_core v0.9.5
Compiling num-complex v0.4.6
Compiling num-integer v0.1.46
Compiling esaxx-rs v0.1.10
Compiling errno v0.3.14
Compiling futures-core v0.3.32
Compiling thiserror v2.0.18
Compiling log v0.4.30
Compiling unicode-width v0.2.2
Compiling pin-project-lite v0.2.17
Compiling utf8parse v0.2.2
Compiling serde_json v1.0.150
Compiling minimal-lexical v0.2.1
Compiling itoa v1.0.18
Compiling nom v7.1.3
Compiling console v0.16.3
Compiling anstyle-parse v1.0.0
Compiling signal-hook-registry v1.4.8
Compiling rand_chacha v0.9.0
Compiling castaway v0.2.4
Compiling derive_builder_macro v0.20.2
Compiling rayon v1.12.0
Compiling itertools v0.14.0
Compiling thiserror-impl v2.0.18
Compiling futures-macro v0.3.32
Compiling tokio-macros v2.7.0
Compiling monostate-impl v0.1.18
Compiling numpy v0.28.0
Compiling mio v1.2.1
Compiling anstyle v1.0.14
Compiling smallvec v1.15.1
Compiling colorchoice v1.0.5
Compiling static_assertions v1.1.0
Compiling macro_rules_attribute-proc_macro v0.2.2
Compiling unicode-segmentation v1.13.3
Compiling anstyle-query v1.1.5
Compiling ryu v1.0.23
Compiling base64 v0.13.1
Compiling slab v0.4.12
Compiling bitflags v2.11.1
Compiling futures-task v0.3.32
Compiling is_terminal_polyfill v1.70.2
Compiling unit-prefix v0.5.2
Compiling onig v6.5.3
Compiling indicatif v0.18.4
Compiling futures-util v0.3.32
Compiling anstream v1.0.0
Compiling spm_precompiled v0.1.4
Compiling compact_str v0.9.1
Compiling macro_rules_attribute v0.2.2
Compiling unicode-normalization-alignments v0.1.12
Compiling tokio v1.52.3
Compiling rayon-cond v0.4.0
Compiling monostate v0.1.18
Compiling derive_builder v0.20.2
Compiling rand v0.9.4
Compiling env_filter v1.0.1
Compiling futures-channel v0.3.32
Compiling ndarray v0.17.2
Compiling dary_heap v0.3.9
Compiling jiff v0.2.28
Compiling daachorse v1.0.1
Compiling rustc-hash v2.1.2
Compiling unicode_categories v0.1.1
Compiling tokenizers v0.23.1 (/builder/src/tokenizers-0.23.1/tokenizers)
Compiling env_logger v0.11.10
Compiling pyo3-async-runtimes v0.28.0
Compiling ndarray v0.16.1
Compiling tokenizers-python v0.23.1 (/builder/src/tokenizers-0.23.1/bindings/python)
Finished `release` profile [optimized] target(s) in 3m 09s
* Getting build dependencies for wheel...
* Building wheel...
Running `maturin pep517 build-wheel -i /usr/sbin/python --compatibility off`
Downloading crates ...
Downloaded equivalent v1.0.2
Downloaded anstyle-wincon v3.0.11
Downloaded anyhow v1.0.102
Downloaded once_cell_polyfill v1.70.2
Downloaded encode_unicode v1.0.0
Downloaded foldhash v0.1.5
Downloaded id-arena v2.3.0
Downloaded unicode-xid v0.2.6
Downloaded wasm-metadata v0.244.0
Downloaded wasm-bindgen v0.2.122
Downloaded windows-link v0.2.1
Downloaded wit-bindgen-rust-macro v0.51.0
Downloaded wasip3 v0.4.0+wasi-0.3.0-rc-2026-01-06
Downloaded wit-bindgen-core v0.51.0
Downloaded zerocopy-derive v0.8.50
Downloaded wasm-encoder v0.244.0
Downloaded wasm-bindgen-macro-support v0.2.122
Downloaded wit-parser v0.244.0
Downloaded wit-bindgen v0.57.1
Downloaded wasmparser v0.244.0
Downloaded wit-component v0.244.0
Downloaded wit-bindgen v0.51.0
Downloaded hashbrown v0.15.5
Downloaded hashbrown v0.17.1
Downloaded wasip2 v1.0.3+wasi-0.2.9
Downloaded js-sys v0.3.99
Downloaded wit-bindgen-rust v0.51.0
Downloaded web-time v1.1.0
Downloaded wasm-bindgen-shared v0.2.122
Downloaded wasm-bindgen-macro v0.2.122
Downloaded jiff-static v0.2.28
Downloaded indexmap v2.14.0
Downloaded r-efi v6.0.0
Downloaded prettyplease v0.2.37
Downloaded wasi v0.11.1+wasi-snapshot-preview1
Downloaded r-efi v5.3.0
Downloaded portable-atomic-util v0.2.7
Downloaded bumpalo v3.20.3
Downloaded semver v1.0.28
Downloaded leb128fmt v0.1.0
Downloaded windows-sys v0.61.2
🍹 Building a mixed python/rust project
🐍 Found CPython 3.14 at /usr/sbin/python
🔗 Found pyo3 bindings with abi3 support
📡 Using build options features, bindings from pyproject.toml
Compiling pyo3-build-config v0.28.2
Compiling pyo3-ffi v0.28.2
Compiling pyo3-macros-backend v0.28.2
Compiling pyo3 v0.28.2
Compiling numpy v0.28.0
Compiling pyo3-macros v0.28.2
Compiling pyo3-async-runtimes v0.28.0
Compiling tokenizers-python v0.23.1 (/builder/src/tokenizers-0.23.1/bindings/python)
Finished `release` profile [optimized] target(s) in 40.46s
📦 Built wheel for abi3 Python ≥ 3.10 to /builder/src/tokenizers-0.23.1/bindings/python/target/wheels/tokenizers-0.23.1-cp310-abi3-linux_x86_64.whl
/builder/src/tokenizers-0.23.1/bindings/python/target/wheels/tokenizers-0.23.1-cp310-abi3-linux_x86_64.whl
Successfully built tokenizers-0.23.1-cp310-abi3-linux_x86_64.whl
==> Starting check()...
============================= test session starts ==============================
platform linux -- Python 3.14.5, pytest-9.0.3, pluggy-1.6.0 -- /builder/src/tokenizers-0.23.1/bindings/python/test-env/bin/python
cachedir: .pytest_cache
rootdir: /builder/src/tokenizers-0.23.1/bindings/python
configfile: pytest.ini (WARNING: ignoring pytest config in setup.cfg!)
plugins: anyio-4.13.0
collecting ... collected 198 items / 1 skipped
tests/bindings/test_decoders.py::TestByteLevel::test_instantiate PASSED
tests/bindings/test_decoders.py::TestByteLevel::test_decoding PASSED
tests/bindings/test_decoders.py::TestByteLevel::test_manual_reload PASSED
tests/bindings/test_decoders.py::TestReplace::test_instantiate PASSED
tests/bindings/test_decoders.py::TestReplace::test_decoding PASSED
tests/bindings/test_decoders.py::TestWordPiece::test_instantiate PASSED
tests/bindings/test_decoders.py::TestWordPiece::test_decoding PASSED
tests/bindings/test_decoders.py::TestWordPiece::test_can_modify PASSED
tests/bindings/test_decoders.py::TestByteFallback::test_instantiate PASSED
tests/bindings/test_decoders.py::TestByteFallback::test_decoding PASSED
tests/bindings/test_decoders.py::TestFuse::test_instantiate PASSED
tests/bindings/test_decoders.py::TestFuse::test_decoding PASSED
tests/bindings/test_decoders.py::TestStrip::test_instantiate PASSED
tests/bindings/test_decoders.py::TestStrip::test_decoding PASSED
tests/bindings/test_decoders.py::TestMetaspace::test_instantiate PASSED
tests/bindings/test_decoders.py::TestMetaspace::test_decoding PASSED
tests/bindings/test_decoders.py::TestMetaspace::test_can_modify PASSED
tests/bindings/test_decoders.py::TestBPEDecoder::test_instantiate PASSED
tests/bindings/test_decoders.py::TestBPEDecoder::test_decoding PASSED
tests/bindings/test_decoders.py::TestBPEDecoder::test_can_modify PASSED
tests/bindings/test_decoders.py::TestCTCDecoder::test_instantiate PASSED
tests/bindings/test_decoders.py::TestCTCDecoder::test_decoding PASSED
tests/bindings/test_decoders.py::TestCTCDecoder::test_can_modify PASSED
tests/bindings/test_decoders.py::TestSequenceDecoder::test_instantiate PASSED
tests/bindings/test_decoders.py::TestSequenceDecoder::test_decoding PASSED
tests/bindings/test_encoding.py::TestEncoding::test_sequence_ids PASSED
tests/bindings/test_encoding.py::TestEncoding::test_n_sequences PASSED
tests/bindings/test_encoding.py::TestEncoding::test_word_to_tokens PASSED
tests/bindings/test_encoding.py::TestEncoding::test_word_to_chars PASSED
tests/bindings/test_encoding.py::TestEncoding::test_token_to_sequence PASSED
tests/bindings/test_encoding.py::TestEncoding::test_token_to_chars PASSED
tests/bindings/test_encoding.py::TestEncoding::test_token_to_word PASSED
tests/bindings/test_encoding.py::TestEncoding::test_char_to_token PASSED
tests/bindings/test_encoding.py::TestEncoding::test_char_to_word PASSED
tests/bindings/test_encoding.py::TestEncoding::test_truncation PASSED
tests/bindings/test_encoding.py::TestEncoding::test_invalid_truncate_direction PASSED
tests/bindings/test_models.py::TestBPE::test_can_modify PASSED
tests/bindings/test_models.py::TestBPE::test_dropout_zero PASSED
tests/bindings/test_models.py::TestUnigram::test_can_modify PASSED
tests/bindings/test_models.py::TestUnigram::test_alpha_zero PASSED
tests/bindings/test_models.py::TestWordPiece::test_instantiate PASSED
tests/bindings/test_models.py::TestWordPiece::test_can_modify PASSED
tests/bindings/test_models.py::TestWordLevel::test_instantiate PASSED
tests/bindings/test_models.py::TestWordLevel::test_can_modify PASSED
tests/bindings/test_normalizers.py::TestBertNormalizer::test_instantiate PASSED
tests/bindings/test_normalizers.py::TestBertNormalizer::test_strip_accents PASSED
tests/bindings/test_normalizers.py::TestBertNormalizer::test_handle_chinese_chars PASSED
tests/bindings/test_normalizers.py::TestBertNormalizer::test_clean_text PASSED
tests/bindings/test_normalizers.py::TestBertNormalizer::test_lowercase PASSED
tests/bindings/test_normalizers.py::TestBertNormalizer::test_can_modify PASSED
tests/bindings/test_normalizers.py::TestSequence::test_instantiate PASSED
tests/bindings/test_normalizers.py::TestSequence::test_can_make_sequences PASSED
tests/bindings/test_normalizers.py::TestSequence::test_set_item PASSED
tests/bindings/test_normalizers.py::TestSequence::test_item_getters_and_setters PASSED
tests/bindings/test_normalizers.py::TestLowercase::test_instantiate PASSED
tests/bindings/test_normalizers.py::TestLowercase::test_lowercase PASSED
tests/bindings/test_normalizers.py::TestStrip::test_instantiate PASSED
tests/bindings/test_normalizers.py::TestStrip::test_left_strip PASSED
tests/bindings/test_normalizers.py::TestStrip::test_right_strip PASSED
tests/bindings/test_normalizers.py::TestStrip::test_full_strip PASSED
tests/bindings/test_normalizers.py::TestStrip::test_can_modify PASSED
tests/bindings/test_normalizers.py::TestPrepend::test_instantiate PASSED
tests/bindings/test_normalizers.py::TestPrepend::test_prepend PASSED
tests/bindings/test_normalizers.py::TestPrepend::test_can_modify PASSED
tests/bindings/test_normalizers.py::TestCustomNormalizer::test_instantiate PASSED
tests/bindings/test_normalizers.py::TestCustomNormalizer::test_normalizer_interface PASSED
tests/bindings/test_pre_tokenizers.py::TestByteLevel::test_instantiate PASSED
tests/bindings/test_pre_tokenizers.py::TestByteLevel::test_has_alphabet PASSED
tests/bindings/test_pre_tokenizers.py::TestByteLevel::test_can_modify PASSED
tests/bindings/test_pre_tokenizers.py::TestByteLevel::test_manual_reload PASSED
tests/bindings/test_pre_tokenizers.py::TestSplit::test_instantiate PASSED
tests/bindings/test_pre_tokenizers.py::TestWhitespace::test_instantiate PASSED
tests/bindings/test_pre_tokenizers.py::TestWhitespaceSplit::test_instantiate PASSED
tests/bindings/test_pre_tokenizers.py::TestBertPreTokenizer::test_instantiate PASSED
tests/bindings/test_pre_tokenizers.py::TestMetaspace::test_instantiate PASSED
tests/bindings/test_pre_tokenizers.py::TestMetaspace::test_can_modify PASSED
tests/bindings/test_pre_tokenizers.py::TestCharDelimiterSplit::test_instantiate PASSED
tests/bindings/test_pre_tokenizers.py::TestCharDelimiterSplit::test_can_modify PASSED
tests/bindings/test_pre_tokenizers.py::TestPunctuation::test_instantiate PASSED
tests/bindings/test_pre_tokenizers.py::TestSequence::test_instantiate PASSED
tests/bindings/test_pre_tokenizers.py::TestSequence::test_bert_like PASSED
tests/bindings/test_pre_tokenizers.py::TestSequence::test_set_item PASSED
tests/bindings/test_pre_tokenizers.py::TestSequence::test_item_getters_and_setters PASSED
tests/bindings/test_pre_tokenizers.py::TestDigits::test_instantiate PASSED
tests/bindings/test_pre_tokenizers.py::TestDigits::test_can_modify PASSED
tests/bindings/test_pre_tokenizers.py::TestFixedLength::test_instantiate PASSED
tests/bindings/test_pre_tokenizers.py::TestFixedLength::test_pre_tokenize_str PASSED
tests/bindings/test_pre_tokenizers.py::TestUnicodeScripts::test_instantiate PASSED
tests/bindings/test_pre_tokenizers.py::TestCustomPreTokenizer::test_instantiate PASSED
tests/bindings/test_pre_tokenizers.py::TestCustomPreTokenizer::test_camel_case PASSED
tests/bindings/test_processors.py::TestBertProcessing::test_instantiate PASSED
tests/bindings/test_processors.py::TestBertProcessing::test_processing PASSED
tests/bindings/test_processors.py::TestRobertaProcessing::test_instantiate PASSED
tests/bindings/test_processors.py::TestRobertaProcessing::test_processing PASSED
tests/bindings/test_processors.py::TestByteLevelProcessing::test_instantiate PASSED
tests/bindings/test_processors.py::TestByteLevelProcessing::test_processing PASSED
tests/bindings/test_processors.py::TestByteLevelProcessing::test_manual_reload PASSED
tests/bindings/test_processors.py::TestTemplateProcessing::test_instantiate PASSED
tests/bindings/test_processors.py::TestTemplateProcessing::test_bert_parity PASSED
tests/bindings/test_processors.py::TestTemplateProcessing::test_roberta_parity PASSED
tests/bindings/test_processors.py::TestSequenceProcessing::test_sequence_processing PASSED
tests/bindings/test_processors.py::TestSequenceProcessing::test_post_process PASSED
tests/bindings/test_processors.py::TestSequenceProcessing::test_items ByteLevel(add_prefix_space=False, trim_offsets=False, use_regex=False)
ByteLevel(add_prefix_space=True, trim_offsets=True, use_regex=True)
PASSED
tests/bindings/test_tokenizer.py::TestAddedToken::test_instantiate_with_content_only PASSED
tests/bindings/test_tokenizer.py::TestAddedToken::test_can_set_rstrip PASSED
tests/bindings/test_tokenizer.py::TestAddedToken::test_can_set_lstrip PASSED
tests/bindings/test_tokenizer.py::TestAddedToken::test_can_set_single_world PASSED
tests/bindings/test_tokenizer.py::TestAddedToken::test_can_set_normalized PASSED
tests/bindings/test_tokenizer.py::TestTokenizer::test_has_expected_type_and_methods PASSED
tests/bindings/test_tokenizer.py::TestTokenizer::test_add_tokens PASSED
tests/bindings/test_tokenizer.py::TestTokenizer::test_add_tokens_with_normalizer PASSED
tests/bindings/test_tokenizer.py::TestTokenizer::test_normalizer_change_refreshes_added_tokens PASSED
tests/bindings/test_tokenizer.py::TestTokenizer::test_add_special_tokens PASSED
tests/bindings/test_tokenizer.py::TestTokenizer::test_encode PASSED
tests/bindings/test_tokenizer.py::TestTokenizer::test_encode_formats PASSED
tests/bindings/test_tokenizer.py::TestTokenizer::test_encode_add_special_tokens PASSED
tests/bindings/test_tokenizer.py::TestTokenizer::test_truncation PASSED
tests/bindings/test_tokenizer.py::TestTokenizer::test_padding PASSED
tests/bindings/test_tokenizer.py::TestTokenizer::test_decode PASSED
tests/bindings/test_tokenizer.py::TestTokenizer::test_decode_stream_copy_and_prefix_ids PASSED
tests/bindings/test_tokenizer.py::TestTokenizer::test_decode_stream_fallback Warning: You are sending unauthenticated requests to the HF Hub. Please set a HF_TOKEN to enable higher rate limits and faster downloads.
PASSED
tests/bindings/test_tokenizer.py::TestTokenizer::test_decode_skip_special_tokens PASSED
tests/bindings/test_tokenizer.py::TestTokenizer::test_decode_stream PASSED
tests/bindings/test_tokenizer.py::TestTokenizer::test_get_vocab PASSED
tests/bindings/test_tokenizer.py::TestTokenizer::test_get_vocab_size PASSED
tests/bindings/test_tokenizer.py::TestTokenizer::test_post_process PASSED
tests/bindings/test_tokenizer.py::TestTokenizer::test_multiprocessing_with_parallelism PASSED
tests/bindings/test_tokenizer.py::TestTokenizer::test_multithreaded_concurrency PASSED
tests/bindings/test_tokenizer.py::TestTokenizer::test_from_pretrained PASSED
tests/bindings/test_tokenizer.py::TestTokenizer::test_from_pretrained_revision PASSED
tests/bindings/test_tokenizer.py::TestTokenizer::test_unigram_byte_fallback PASSED
tests/bindings/test_tokenizer.py::TestTokenizer::test_encode_special_tokens PASSED
tests/bindings/test_tokenizer.py::TestTokenizer::test_splitting PASSED
tests/bindings/test_tokenizer.py::TestTokenizer::test_decode_special PASSED
tests/bindings/test_tokenizer.py::TestTokenizer::test_weakref_support PASSED
tests/bindings/test_tokenizer.py::TestTokenizer::test_weakref_with_multiple_references PASSED
tests/bindings/test_tokenizer.py::TestTokenizer::test_setting_to_none PASSED
tests/bindings/test_tokenizer.py::TestTokenizerRepr::test_repr PASSED
tests/bindings/test_tokenizer.py::TestTokenizerRepr::test_repr_complete PASSED
tests/bindings/test_tokenizer.py::TestAsyncTokenizer::test_basic_encoding FAILED
tests/bindings/test_tokenizer.py::TestAsyncTokenizer::test_encode FAILED
tests/bindings/test_tokenizer.py::TestAsyncTokenizer::test_with_special_tokens FAILED
tests/bindings/test_tokenizer.py::TestAsyncTokenizer::test_with_truncation_padding FAILED
tests/bindings/test_tokenizer.py::TestAsyncTokenizer::test_various_input_formats FAILED
tests/bindings/test_tokenizer.py::TestAsyncTokenizer::test_error_handling FAILED
tests/bindings/test_tokenizer.py::TestAsyncTokenizer::test_concurrency FAILED
tests/bindings/test_tokenizer.py::TestAsyncTokenizer::test_decode FAILED
tests/bindings/test_tokenizer.py::TestAsyncTokenizer::test_large_batch FAILED
tests/bindings/test_tokenizer.py::TestAsyncTokenizer::test_numpy_inputs FAILED
tests/bindings/test_tokenizer.py::TestAsyncTokenizer::test_async_methods_existence PASSED
tests/bindings/test_tokenizer.py::TestAsyncTokenizer::test_performance_comparison FAILED
tests/bindings/test_trainers.py::TestBpeTrainer::test_can_modify PASSED
tests/bindings/test_trainers.py::TestBpeTrainer::test_can_pickle PASSED
tests/bindings/test_trainers.py::TestWordPieceTrainer::test_can_modify PASSED
tests/bindings/test_trainers.py::TestWordPieceTrainer::test_can_pickle PASSED
tests/bindings/test_trainers.py::TestWordLevelTrainer::test_can_modify PASSED
tests/bindings/test_trainers.py::TestWordLevelTrainer::test_can_pickle PASSED
tests/bindings/test_trainers.py::TestUnigram::test_train PASSED
tests/bindings/test_trainers.py::TestUnigram::test_train_parallelism_with_custom_pretokenizer PASSED
tests/bindings/test_trainers.py::TestUnigram::test_can_pickle PASSED
tests/bindings/test_trainers.py::TestUnigram::test_train_with_special_tokens PASSED
tests/bindings/test_trainers.py::TestUnigram::test_cannot_train_different_model PASSED
tests/bindings/test_trainers.py::TestUnigram::test_can_modify PASSED
tests/bindings/test_trainers.py::TestUnigram::test_continuing_prefix_trainer_mismatch
PASSED
tests/documentation/test_pipeline.py::TestPipeline::test_pipeline PASSED
tests/documentation/test_pipeline.py::TestPipeline::test_bert_example PASSED
tests/documentation/test_quicktour.py::TestQuicktour::test_quicktour PASSED
tests/documentation/test_tutorial_train_from_iterators.py::TestTrainFromIterators::test_train_basic PASSED
tests/documentation/test_tutorial_train_from_iterators.py::TestTrainFromIterators::test_datasets FAILED
tests/documentation/test_tutorial_train_from_iterators.py::TestTrainFromIterators::test_gzip PASSED
tests/implementations/test_base_tokenizer.py::TestBaseTokenizer::test_get_set_components PASSED
tests/implementations/test_bert_wordpiece.py::TestBertWordPieceTokenizer::test_basic_encode PASSED
tests/implementations/test_bert_wordpiece.py::TestBertWordPieceTokenizer::test_multiprocessing_with_parallelism PASSED
tests/implementations/test_bert_wordpiece.py::TestBertWordPieceTokenizer::test_train_from_iterator PASSED
tests/implementations/test_byte_level_bpe.py::TestByteLevelBPE::test_basic_encode PASSED
tests/implementations/test_byte_level_bpe.py::TestByteLevelBPE::test_add_prefix_space PASSED
tests/implementations/test_byte_level_bpe.py::TestByteLevelBPE::test_lowerspace PASSED
tests/implementations/test_byte_level_bpe.py::TestByteLevelBPE::test_multiprocessing_with_parallelism PASSED
tests/implementations/test_byte_level_bpe.py::TestByteLevelBPE::test_train_from_iterator PASSED
tests/implementations/test_char_bpe.py::TestCharBPETokenizer::test_basic_encode PASSED
tests/implementations/test_char_bpe.py::TestCharBPETokenizer::test_lowercase PASSED
tests/implementations/test_char_bpe.py::TestCharBPETokenizer::test_decoding PASSED
tests/implementations/test_char_bpe.py::TestCharBPETokenizer::test_multiprocessing_with_parallelism PASSED
tests/implementations/test_char_bpe.py::TestCharBPETokenizer::test_train_from_iterator PASSED
tests/implementations/test_sentencepiece.py::TestSentencePieceBPE::test_train_from_iterator PASSED
tests/implementations/test_sentencepiece.py::TestSentencePieceUnigram::test_train PASSED
tests/implementations/test_sentencepiece.py::TestSentencePieceUnigram::test_train_with_unk_token PASSED
tests/implementations/test_sentencepiece.py::TestSentencePieceUnigram::test_train_from_iterator PASSED
tests/implementations/test_sentencepiece.py::TestSentencePieceUnigram::test_train_from_iterator_with_unk_token PASSED
tests/test_freethreaded.py::TestEncodeUnderConcurrentSetters::test_encode_while_swapping_post_processor PASSED
tests/test_freethreaded.py::TestEncodeUnderConcurrentSetters::test_encode_while_mutating_trainer_fields PASSED
tests/test_freethreaded.py::TestEncodeUnderConcurrentSetters::test_concurrent_setters_no_lock_poisoning PASSED
tests/test_freethreaded.py::TestFreeThreadedSpecific::test_gil_actually_disabled_on_import SKIPPED
tests/test_serialization.py::TestSerialization::test_full_serialization_albert PASSED
tests/test_serialization.py::TestSerialization::test_str_big PASSED
tests/test_serialization.py::TestSerialization::test_repr_str PASSED
tests/test_serialization.py::TestSerialization::test_repr_str_ellipsis PASSED
tests/test_serialization.py::TestFullDeserialization::test_full_deserialization_hub SKIPPED
=================================== FAILURES ===================================
____________________ TestAsyncTokenizer.test_basic_encoding ____________________
async def functions are not natively supported.
You need to install a suitable plugin for your async framework, for example:
- anyio
- pytest-asyncio
- pytest-tornasync
- pytest-trio
- pytest-twisted
________________________ TestAsyncTokenizer.test_encode ________________________
async def functions are not natively supported.
You need to install a suitable plugin for your async framework, for example:
- anyio
- pytest-asyncio
- pytest-tornasync
- pytest-trio
- pytest-twisted
_________________ TestAsyncTokenizer.test_with_special_tokens __________________
async def functions are not natively supported.
You need to install a suitable plugin for your async framework, for example:
- anyio
- pytest-asyncio
- pytest-tornasync
- pytest-trio
- pytest-twisted
_______________ TestAsyncTokenizer.test_with_truncation_padding ________________
async def functions are not natively supported.
You need to install a suitable plugin for your async framework, for example:
- anyio
- pytest-asyncio
- pytest-tornasync
- pytest-trio
- pytest-twisted
________________ TestAsyncTokenizer.test_various_input_formats _________________
async def functions are not natively supported.
You need to install a suitable plugin for your async framework, for example:
- anyio
- pytest-asyncio
- pytest-tornasync
- pytest-trio
- pytest-twisted
____________________ TestAsyncTokenizer.test_error_handling ____________________
async def functions are not natively supported.
You need to install a suitable plugin for your async framework, for example:
- anyio
- pytest-asyncio
- pytest-tornasync
- pytest-trio
- pytest-twisted
_____________________ TestAsyncTokenizer.test_concurrency ______________________
async def functions are not natively supported.
You need to install a suitable plugin for your async framework, for example:
- anyio
- pytest-asyncio
- pytest-tornasync
- pytest-trio
- pytest-twisted
________________________ TestAsyncTokenizer.test_decode ________________________
async def functions are not natively supported.
You need to install a suitable plugin for your async framework, for example:
- anyio
- pytest-asyncio
- pytest-tornasync
- pytest-trio
- pytest-twisted
_____________________ TestAsyncTokenizer.test_large_batch ______________________
async def functions are not natively supported.
You need to install a suitable plugin for your async framework, for example:
- anyio
- pytest-asyncio
- pytest-tornasync
- pytest-trio
- pytest-twisted
_____________________ TestAsyncTokenizer.test_numpy_inputs _____________________
async def functions are not natively supported.
You need to install a suitable plugin for your async framework, for example:
- anyio
- pytest-asyncio
- pytest-tornasync
- pytest-trio
- pytest-twisted
________________ TestAsyncTokenizer.test_performance_comparison ________________
async def functions are not natively supported.
You need to install a suitable plugin for your async framework, for example:
- anyio
- pytest-asyncio
- pytest-tornasync
- pytest-trio
- pytest-twisted
_____________________ TestTrainFromIterators.test_datasets _____________________
self = <tests.documentation.test_tutorial_train_from_iterators.TestTrainFromIterators object at 0x731572794190>
@pytest.mark.network
def test_datasets(self):
tokenizer, trainer = self.get_tokenizer_trainer()
# In order to keep tests fast, we only use the first 100 examples
os.environ["TOKENIZERS_PARALLELISM"] = "true"
> dataset = datasets.load_dataset("wikitext", "wikitext-103-raw-v1", split="train[0:100]")
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
tests/documentation/test_tutorial_train_from_iterators.py:71:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
/usr/lib/python3.14/site-packages/datasets/load.py:1688: in load_dataset
builder_instance = load_dataset_builder(
/usr/lib/python3.14/site-packages/datasets/load.py:1315: in load_dataset_builder
dataset_module = dataset_module_factory(
/usr/lib/python3.14/site-packages/datasets/load.py:1207: in dataset_module_factory
raise e1 from None
/usr/lib/python3.14/site-packages/datasets/load.py:1182: in dataset_module_factory
).get_module()
^^^^^^^^^^^^
/usr/lib/python3.14/site-packages/datasets/load.py:598: in get_module
standalone_yaml_path = cached_path(
/usr/lib/python3.14/site-packages/datasets/utils/file_utils.py:180: in cached_path
).resolve_path(url_or_filename)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
/usr/lib/python3.14/site-packages/huggingface_hub/hf_file_system.py:305: in resolve_path
parsed = parse_hf_uri(f"{constants.HF_PROTOCOL}{path}")
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
/usr/lib/python3.14/site-packages/huggingface_hub/utils/_hf_uris.py:240: in parse_hf_uri
return _parse_repo_body(location, type_, raw=raw)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
location = 'wikitext@b08601e04326c79dfdd32d625aee71d232d685c3/.huggingface.yaml'
type_ = 'dataset'
def _parse_repo_body(
location: str,
type_: constants.HfUriType,
*,
raw: str,
) -> HfUri:
"""Parse the body of a repo URI: '<repo_id>[@<revision>][/<path>]'."""
location = location.strip("/")
if not location:
raise HfUriError(uri=raw, msg="Missing repository id.")
# The '@' separates the repo_id from the revision, but only when it
# appears right after 'namespace/name' (at most one '/' before it).
# An '@' deeper in the path (e.g. in a filename like 'file@1.txt') is literal.
at_idx = location.find("@")
revision: str | None
if at_idx == -1 or location[:at_idx].count("/") > 1:
# No '@' at all, or the '@' is past the repo_id portion (in a filename).
revision = None
parts = location.split("/", 2)
if len(parts) < 2:
raise HfUriError(uri=raw, msg=f"Repository id must be 'namespace/name', got '{location}'. ")
repo_id = f"{parts[0]}/{parts[1]}"
path_in_repo = parts[2] if len(parts) > 2 else ""
else:
repo_id = location[:at_idx]
rev_and_path = location[at_idx + 1 :]
if not repo_id:
raise HfUriError(uri=raw, msg="Missing repository id before '@'.")
if repo_id.count("/") != 1:
> raise HfUriError(uri=raw, msg=f"Repository id must be 'namespace/name', got '{repo_id}'.")
E huggingface_hub.errors.HfUriError: Invalid HF URI 'hf://datasets/wikitext@b08601e04326c79dfdd32d625aee71d232d685c3/.huggingface.yaml'. Repository id must be 'namespace/name', got 'wikitext'.
/usr/lib/python3.14/site-packages/huggingface_hub/utils/_hf_uris.py:417: HfUriError
=============================== warnings summary ===============================
tests/bindings/test_tokenizer.py:895
/builder/src/tokenizers-0.23.1/bindings/python/tests/bindings/test_tokenizer.py:895: PytestUnknownMarkWarning: Unknown pytest.mark.asyncio - is this a typo? You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/stable/how-to/mark.html
@pytest.mark.asyncio
tests/bindings/test_tokenizer.py:907
/builder/src/tokenizers-0.23.1/bindings/python/tests/bindings/test_tokenizer.py:907: PytestUnknownMarkWarning: Unknown pytest.mark.asyncio - is this a typo? You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/stable/how-to/mark.html
@pytest.mark.asyncio
tests/bindings/test_tokenizer.py:917
/builder/src/tokenizers-0.23.1/bindings/python/tests/bindings/test_tokenizer.py:917: PytestUnknownMarkWarning: Unknown pytest.mark.asyncio - is this a typo? You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/stable/how-to/mark.html
@pytest.mark.asyncio
tests/bindings/test_tokenizer.py:936
/builder/src/tokenizers-0.23.1/bindings/python/tests/bindings/test_tokenizer.py:936: PytestUnknownMarkWarning: Unknown pytest.mark.asyncio - is this a typo? You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/stable/how-to/mark.html
@pytest.mark.asyncio
tests/bindings/test_tokenizer.py:948
/builder/src/tokenizers-0.23.1/bindings/python/tests/bindings/test_tokenizer.py:948: PytestUnknownMarkWarning: Unknown pytest.mark.asyncio - is this a typo? You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/stable/how-to/mark.html
@pytest.mark.asyncio
tests/bindings/test_tokenizer.py:963
/builder/src/tokenizers-0.23.1/bindings/python/tests/bindings/test_tokenizer.py:963: PytestUnknownMarkWarning: Unknown pytest.mark.asyncio - is this a typo? You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/stable/how-to/mark.html
@pytest.mark.asyncio
tests/bindings/test_tokenizer.py:980
/builder/src/tokenizers-0.23.1/bindings/python/tests/bindings/test_tokenizer.py:980: PytestUnknownMarkWarning: Unknown pytest.mark.asyncio - is this a typo? You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/stable/how-to/mark.html
@pytest.mark.asyncio
tests/bindings/test_tokenizer.py:1000
/builder/src/tokenizers-0.23.1/bindings/python/tests/bindings/test_tokenizer.py:1000: PytestUnknownMarkWarning: Unknown pytest.mark.asyncio - is this a typo? You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/stable/how-to/mark.html
@pytest.mark.asyncio
tests/bindings/test_tokenizer.py:1015
/builder/src/tokenizers-0.23.1/bindings/python/tests/bindings/test_tokenizer.py:1015: PytestUnknownMarkWarning: Unknown pytest.mark.asyncio - is this a typo? You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/stable/how-to/mark.html
@pytest.mark.asyncio
tests/bindings/test_tokenizer.py:1028
/builder/src/tokenizers-0.23.1/bindings/python/tests/bindings/test_tokenizer.py:1028: PytestUnknownMarkWarning: Unknown pytest.mark.asyncio - is this a typo? You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/stable/how-to/mark.html
@pytest.mark.asyncio
tests/bindings/test_tokenizer.py:1046
/builder/src/tokenizers-0.23.1/bindings/python/tests/bindings/test_tokenizer.py:1046: PytestUnknownMarkWarning: Unknown pytest.mark.asyncio - is this a typo? You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/stable/how-to/mark.html
@pytest.mark.asyncio
tests/test_freethreaded.py:32
/builder/src/tokenizers-0.23.1/bindings/python/tests/test_freethreaded.py:32: PytestUnknownMarkWarning: Unknown pytest.mark.timeout - is this a typo? You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/stable/how-to/mark.html
pytestmark = pytest.mark.timeout(60) # any of these hanging means a deadlock
tests/bindings/test_tokenizer.py::TestTokenizer::test_decode_skip_special_tokens
tests/bindings/test_tokenizer.py::TestAsyncTokenizer::test_basic_encoding
/usr/lib/python3.14/site-packages/huggingface_hub/file_download.py:1855: DeprecationWarning: hf_xet.download_files() is deprecated. Use XetSession().new_file_download_group().start_download_file() instead.
xet_get(
tests/bindings/test_tokenizer.py::TestTokenizer::test_multiprocessing_with_parallelism
tests/bindings/test_tokenizer.py::TestTokenizer::test_multiprocessing_with_parallelism
tests/implementations/test_bert_wordpiece.py::TestBertWordPieceTokenizer::test_multiprocessing_with_parallelism
tests/implementations/test_bert_wordpiece.py::TestBertWordPieceTokenizer::test_multiprocessing_with_parallelism
tests/implementations/test_byte_level_bpe.py::TestByteLevelBPE::test_multiprocessing_with_parallelism
tests/implementations/test_byte_level_bpe.py::TestByteLevelBPE::test_multiprocessing_with_parallelism
tests/implementations/test_char_bpe.py::TestCharBPETokenizer::test_multiprocessing_with_parallelism
tests/implementations/test_char_bpe.py::TestCharBPETokenizer::test_multiprocessing_with_parallelism
/usr/lib/python3.14/multiprocessing/popen_fork.py:76: DeprecationWarning: This process (pid=4915) is multi-threaded, use of fork() may lead to deadlocks in the child.
self.pid = os.fork()
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
=========================== short test summary info ============================
FAILED tests/bindings/test_tokenizer.py::TestAsyncTokenizer::test_basic_encoding
FAILED tests/bindings/test_tokenizer.py::TestAsyncTokenizer::test_encode - Fa...
FAILED tests/bindings/test_tokenizer.py::TestAsyncTokenizer::test_with_special_tokens
FAILED tests/bindings/test_tokenizer.py::TestAsyncTokenizer::test_with_truncation_padding
FAILED tests/bindings/test_tokenizer.py::TestAsyncTokenizer::test_various_input_formats
FAILED tests/bindings/test_tokenizer.py::TestAsyncTokenizer::test_error_handling
FAILED tests/bindings/test_tokenizer.py::TestAsyncTokenizer::test_concurrency
FAILED tests/bindings/test_tokenizer.py::TestAsyncTokenizer::test_decode - Fa...
FAILED tests/bindings/test_tokenizer.py::TestAsyncTokenizer::test_large_batch
FAILED tests/bindings/test_tokenizer.py::TestAsyncTokenizer::test_numpy_inputs
FAILED tests/bindings/test_tokenizer.py::TestAsyncTokenizer::test_performance_comparison
FAILED tests/documentation/test_tutorial_train_from_iterators.py::TestTrainFromIterators::test_datasets
=========== 12 failed, 184 passed, 3 skipped, 22 warnings in 35.98s ============
Updating crates.io index
Downloading crates ...
Downloaded slab v0.4.12
Downloaded signal-hook-registry v1.4.8
Downloaded anstyle v1.0.14
Downloaded static_assertions v1.1.0
Downloaded smallvec v1.15.1
Downloaded strsim v0.11.1
Downloaded env_filter v1.0.1
Downloaded anstyle-query v1.1.5
Downloaded tokio-macros v2.7.0
Downloaded version_check v0.9.5
Downloaded unit-prefix v0.5.2
Downloaded env_logger v0.11.10
Downloaded target-lexicon v0.13.5
Downloaded zmij v1.0.21
Downloaded unicode-segmentation v1.13.3
Downloaded unicode_categories v0.1.1
Downloaded jiff v0.2.28
Downloaded unicode-normalization-alignments v0.1.12
Downloaded macro_rules_attribute-proc_macro v0.2.2
Downloaded colorchoice v1.0.5
Downloaded pyo3-macros v0.28.2
Downloaded rawpointer v0.2.1
Downloaded rustc-hash v2.1.2
Downloaded rand_chacha v0.9.0
Downloaded pkg-config v0.3.33
Downloaded ryu v1.0.23
Downloaded zerocopy v0.8.50
Downloaded serde_derive v1.0.228
Downloaded syn v2.0.117
Downloaded rayon-core v1.13.0
Downloaded mio v1.2.1
Downloaded rand v0.9.4
Downloaded itertools v0.14.0
Downloaded regex v1.12.3
Downloaded rayon v1.12.0
Downloaded portable-atomic v1.13.1
Downloaded ndarray v0.17.2
Downloaded memchr v2.8.1
Downloaded compact_str v0.9.1
Downloaded spm_precompiled v0.1.4
Downloaded ndarray v0.16.1
Downloaded pyo3-macros-backend v0.28.2
Downloaded nom v7.1.3
Downloaded regex-syntax v0.8.10
Downloaded pyo3-ffi v0.28.2
Downloaded proc-macro2 v1.0.106
Downloaded log v0.4.30
Downloaded rustix v1.1.4
Downloaded getrandom v0.4.2
Downloaded base64 v0.13.1
Downloaded numpy v0.28.0
Downloaded matrixmultiply v0.3.10
Downloaded getrandom v0.3.4
Downloaded console v0.16.3
Downloaded bitflags v2.11.1
Downloaded unicode-width v0.2.2
Downloaded serde_json v1.0.150
Downloaded serde_core v1.0.228
Downloaded regex-automata v0.4.14
Downloaded serde v1.0.228
Downloaded rand_core v0.9.5
Downloaded tokio v1.52.3
Downloaded onig_sys v69.9.3
Downloaded quote v1.0.45
Downloaded libc v0.2.186
Downloaded pyo3-build-config v0.28.2
Downloaded ppv-lite86 v0.2.21
Downloaded num-complex v0.4.6
Downloaded indicatif v0.18.4
Downloaded pin-project-lite v0.2.17
Downloaded paste v1.0.15
Downloaded fastrand v2.4.1
Downloaded onig v6.5.3
Downloaded once_cell v1.21.4
Downloaded num-integer v0.1.46
Downloaded macro_rules_attribute v0.2.2
Downloaded is_terminal_polyfill v1.70.2
Downloaded crossbeam-deque v0.8.6
Downloaded ident_case v1.0.1
Downloaded castaway v0.2.4
Downloaded itoa v1.0.18
Downloaded futures-task v0.3.32
Downloaded futures-macro v0.3.32
Downloaded fnv v1.0.7
Downloaded find-msvc-tools v0.1.9
Downloaded pyo3 v0.28.2
Downloaded rustversion v1.0.22
Downloaded rayon-cond v0.4.0
Downloaded pyo3-async-runtimes v0.28.0
Downloaded num-traits v0.2.19
Downloaded monostate-impl v0.1.18
Downloaded monostate v0.1.18
Downloaded heck v0.5.0
Downloaded futures-util v0.3.32
Downloaded futures-channel v0.3.32
Downloaded futures-core v0.3.32
Downloaded minimal-lexical v0.2.1
Downloaded unicode-ident v1.0.24
Downloaded thiserror-impl v2.0.18
Downloaded tempfile v3.27.0
Downloaded thiserror v2.0.18
Downloaded esaxx-rs v0.1.10
Downloaded either v1.16.0
Downloaded derive_builder v0.20.2
Downloaded daachorse v1.0.1
Downloaded cc v1.2.63
Downloaded dary_heap v0.3.9
Downloaded darling_core v0.20.11
Downloaded ahash v0.8.12
Downloaded derive_builder_core v0.20.2
Downloaded darling_macro v0.20.11
Downloaded darling v0.20.11
Downloaded crossbeam-utils v0.8.21
Downloaded crossbeam-epoch v0.9.18
Downloaded cfg-if v1.0.4
Downloaded autocfg v1.5.1
Downloaded aho-corasick v1.1.4
Downloaded utf8parse v0.2.2
Downloaded errno v0.3.14
Downloaded derive_builder_macro v0.20.2
Downloaded shlex v2.0.1
Downloaded anstyle-parse v1.0.0
Downloaded anstream v1.0.0
Downloaded linux-raw-sys v0.12.1
Compiling proc-macro2 v1.0.106
Compiling unicode-ident v1.0.24
Compiling quote v1.0.45
Compiling libc v0.2.186
Compiling target-lexicon v0.13.5
Compiling autocfg v1.5.1
Compiling serde_core v1.0.228
Compiling pyo3-build-config v0.28.2
Compiling syn v2.0.117
Compiling cfg-if v1.0.4
Compiling shlex v2.0.1
Compiling memchr v2.8.1
Compiling find-msvc-tools v0.1.9
Compiling cc v1.2.63
Compiling crossbeam-utils v0.8.21
Compiling num-traits v0.2.19
Compiling once_cell v1.21.4
Compiling zerocopy v0.8.50
Compiling serde v1.0.228
Compiling fnv v1.0.7
Compiling ident_case v1.0.1
Compiling strsim v0.11.1
Compiling getrandom v0.3.4
Compiling pyo3-ffi v0.28.2
Compiling pyo3-macros-backend v0.28.2
Compiling crossbeam-epoch v0.9.18
Compiling darling_core v0.20.11
Compiling aho-corasick v1.1.4
Compiling serde_derive v1.0.228
Compiling darling_macro v0.20.11
Compiling matrixmultiply v0.3.10
Compiling pkg-config v0.3.33
Compiling rustversion v1.0.22
Compiling regex-syntax v0.8.10
Compiling heck v0.5.0
Compiling rayon-core v1.13.0
Compiling onig_sys v69.9.3
Compiling darling v0.20.11
Compiling crossbeam-deque v0.8.6
Compiling pyo3 v0.28.2
Compiling either v1.16.0
Compiling bitflags v2.11.1
Compiling rawpointer v0.2.1
Compiling portable-atomic v1.13.1
Compiling regex-automata v0.4.14
Compiling paste v1.0.15
Compiling version_check v0.9.5
Compiling zmij v1.0.21
Compiling ahash v0.8.12
Compiling derive_builder_core v0.20.2
Compiling regex v1.12.3
Compiling ppv-lite86 v0.2.21
Compiling rand_core v0.9.5
Compiling pyo3-macros v0.28.2
Compiling num-integer v0.1.46
Compiling num-complex v0.4.6
Compiling esaxx-rs v0.1.10
Compiling errno v0.3.14
Compiling futures-core v0.3.32
Compiling thiserror v2.0.18
Compiling unicode-width v0.2.2
Compiling minimal-lexical v0.2.1
Compiling rustix v1.1.4
Compiling getrandom v0.4.2
Compiling utf8parse v0.2.2
Compiling serde_json v1.0.150
Compiling itoa v1.0.18
Compiling pin-project-lite v0.2.17
Compiling log v0.4.30
Compiling anstyle-parse v1.0.0
Compiling nom v7.1.3
Compiling console v0.16.3
Compiling signal-hook-registry v1.4.8
Compiling rand_chacha v0.9.0
Compiling castaway v0.2.4
Compiling derive_builder_macro v0.20.2
Compiling rayon v1.12.0
Compiling itertools v0.14.0
Compiling monostate-impl v0.1.18
Compiling futures-macro v0.3.32
Compiling tokio-macros v2.7.0
Compiling thiserror-impl v2.0.18
Compiling numpy v0.28.0
Compiling mio v1.2.1
Compiling is_terminal_polyfill v1.70.2
Compiling base64 v0.13.1
Compiling unit-prefix v0.5.2
Compiling colorchoice v1.0.5
Compiling slab v0.4.12
Compiling static_assertions v1.1.0
Compiling smallvec v1.15.1
Compiling anstyle-query v1.1.5
Compiling futures-task v0.3.32
Compiling unicode-segmentation v1.13.3
Compiling linux-raw-sys v0.12.1
Compiling anstyle v1.0.14
Compiling macro_rules_attribute-proc_macro v0.2.2
Compiling ryu v1.0.23
Compiling compact_str v0.9.1
Compiling spm_precompiled v0.1.4
Compiling macro_rules_attribute v0.2.2
Compiling anstream v1.0.0
Compiling futures-util v0.3.32
Compiling unicode-normalization-alignments v0.1.12
Compiling onig v6.5.3
Compiling indicatif v0.18.4
Compiling tokio v1.52.3
Compiling rayon-cond v0.4.0
Compiling monostate v0.1.18
Compiling derive_builder v0.20.2
Compiling rand v0.9.4
Compiling env_filter v1.0.1
Compiling futures-channel v0.3.32
Compiling ndarray v0.17.2
Compiling dary_heap v0.3.9
Compiling jiff v0.2.28
Compiling daachorse v1.0.1
Compiling fastrand v2.4.1
Compiling rustc-hash v2.1.2
Compiling unicode_categories v0.1.1
Compiling tokenizers v0.23.1 (/builder/src/tokenizers-0.23.1/tokenizers)
Compiling tempfile v3.27.0
Compiling pyo3-async-runtimes v0.28.0
Compiling env_logger v0.11.10
Compiling ndarray v0.16.1
Compiling tokenizers-python v0.23.1 (/builder/src/tokenizers-0.23.1/bindings/python)
Finished `test` profile [unoptimized + debuginfo] target(s) in 51.48s
Running unittests src/lib.rs (target/debug/deps/tokenizers-d0ff840e11124795)
running 20 tests
test models::test::serialize ... ok
test normalizers::test::deserialize_sequence ... ok
test models::test::get_subtype ... ok
test decoders::test::serialize ... ok
test decoders::test::get_subtype ... ok
test normalizers::test::get_subtype ... ok
test normalizers::test::serialize ... ok
test pre_tokenizers::test::get_subtype ... ok
test pre_tokenizers::test::serialize ... ok
test processors::test::get_subtype ... ok
test processors::test::serialize ... ok
test tokenizer::test::serde_pyo3 ... ok
test utils::serde_pyo3::test_basic ... ok
test utils::serde_pyo3::test_enum ... ok
test trainers::tests::get_subtype ... ok
test utils::serde_pyo3::test_enum_untagged ... ok
test utils::serde_pyo3::test_flatten ... ok
test utils::serde_pyo3::test_struct ... ok
test utils::serde_pyo3::test_struct_tagged ... ok
test tokenizer::test::serialize ... ok
test result: ok. 20 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.01s
==> Entering fakeroot environment...
==> Starting package()...
==> Tidying install...
-> Removing libtool files...
-> Removing static library files...
-> Purging unwanted files...
-> Stripping unneeded symbols from binaries and libraries...
-> Compressing man and info pages...
==> Checking for packaging issues...
==> WARNING: Package contains reference to $srcdir
usr/lib/python3.14/site-packages/tokenizers/tokenizers.abi3.so
usr/lib/python3.14/site-packages/tokenizers-0.23.1.dist-info/sboms/tokenizers-python.cyclonedx.json
==> Creating package "python-tokenizers"...
-> Generating .PKGINFO file...
-> Generating .BUILDINFO file...
-> Generating .MTREE file...
-> Compressing package...
==> Leaving fakeroot environment.
==> Finished making: python-tokenizers 0.23.1-1 (Tue Jun 2 07:12:48 2026)