拥抱脸
从Hugging Face扫描数据集
所有支持云端的扫描功能及其read_对应功能都透明地支持从Hugging Face进行扫描：
| Scan | Read |
|---|---|
| scan_parquet | read_parquet |
| scan_csv | read_csv |
| scan_ndjson | read_ndjson |
| scan_ipc | read_ipc |
路径格式
要从 Hugging Face 扫描，可以将 hf:// 路径传递给扫描函数。hf:// 路径格式定义为 hf://BUCKET/REPOSITORY@REVISION/PATH，其中：
BUCKET是datasets或spaces中的一个REPOSITORY是仓库的位置，通常格式为username/repo_name。也可以通过附加@branch来指定分支。REVISION是要使用的分支（或提交）的名称。这是可选的，如果未提供，则默认为main。PATH是一个文件或目录路径，或者是从仓库根目录开始的通配符模式。
示例 hf:// 路径：
| Path | Path components |
|---|---|
| hf://datasets/nameexhaustion/polars-docs/iris.csv | Bucket: datasets Repository: nameexhaustion/polars-docs Branch: main Path: iris.csv Web URL |
| hf://datasets/nameexhaustion/polars-docs@foods/*.csv | Bucket: datasets Repository: nameexhaustion/polars-docs Branch: foods Path: *.csv Web URL |
| hf://datasets/nameexhaustion/polars-docs/hive_dates/ | Bucket: datasets Repository: nameexhaustion/polars-docs Branch: main Path: hive_dates/ Web URL |
| hf://spaces/nameexhaustion/polars-docs/orders.feather | Bucket: spaces Repository: nameexhaustion/polars-docs Branch: main Path: orders.feather Web URL |
认证
可以将 Hugging Face API 密钥传递给 Polars，以使用以下任一方法访问私有位置：
- 在
storage_options中传递一个token给扫描函数，例如scan_parquet(..., storage_options={'token': ''}) - 设置
HF_TOKEN环境变量，例如export HF_TOKEN=
示例
CSV
print(pl.scan_csv("hf://datasets/nameexhaustion/polars-docs/iris.csv").collect())
shape: (150, 5)
┌──────────────┬─────────────┬──────────────┬─────────────┬───────────┐
│ sepal_length ┆ sepal_width ┆ petal_length ┆ petal_width ┆ species │
│ --- ┆ --- ┆ --- ┆ --- ┆ --- │
│ f64 ┆ f64 ┆ f64 ┆ f64 ┆ str │
╞══════════════╪═════════════╪══════════════╪═════════════╪═══════════╡
│ 5.1 ┆ 3.5 ┆ 1.4 ┆ 0.2 ┆ setosa │
│ 4.9 ┆ 3.0 ┆ 1.4 ┆ 0.2 ┆ setosa │
│ 4.7 ┆ 3.2 ┆ 1.3 ┆ 0.2 ┆ setosa │
│ 4.6 ┆ 3.1 ┆ 1.5 ┆ 0.2 ┆ setosa │
│ 5.0 ┆ 3.6 ┆ 1.4 ┆ 0.2 ┆ setosa │
│ … ┆ … ┆ … ┆ … ┆ … │
│ 6.7 ┆ 3.0 ┆ 5.2 ┆ 2.3 ┆ virginica │
│ 6.3 ┆ 2.5 ┆ 5.0 ┆ 1.9 ┆ virginica │
│ 6.5 ┆ 3.0 ┆ 5.2 ┆ 2.0 ┆ virginica │
│ 6.2 ┆ 3.4 ┆ 5.4 ┆ 2.3 ┆ virginica │
│ 5.9 ┆ 3.0 ┆ 5.1 ┆ 1.8 ┆ virginica │
└──────────────┴─────────────┴──────────────┴─────────────┴───────────┘
查看此文件位于 https://huggingface.co/datasets/nameexhaustion/polars-docs/blob/main/iris.csv
NDJSON
print(pl.scan_ndjson("hf://datasets/nameexhaustion/polars-docs/iris.jsonl").collect())
shape: (150, 5)
┌──────────────┬─────────────┬──────────────┬─────────────┬───────────┐
│ sepal_length ┆ sepal_width ┆ petal_length ┆ petal_width ┆ species │
│ --- ┆ --- ┆ --- ┆ --- ┆ --- │
│ f64 ┆ f64 ┆ f64 ┆ f64 ┆ str │
╞══════════════╪═════════════╪══════════════╪═════════════╪═══════════╡
│ 5.1 ┆ 3.5 ┆ 1.4 ┆ 0.2 ┆ setosa │
│ 4.9 ┆ 3.0 ┆ 1.4 ┆ 0.2 ┆ setosa │
│ 4.7 ┆ 3.2 ┆ 1.3 ┆ 0.2 ┆ setosa │
│ 4.6 ┆ 3.1 ┆ 1.5 ┆ 0.2 ┆ setosa │
│ 5.0 ┆ 3.6 ┆ 1.4 ┆ 0.2 ┆ setosa │
│ … ┆ … ┆ … ┆ … ┆ … │
│ 6.7 ┆ 3.0 ┆ 5.2 ┆ 2.3 ┆ virginica │
│ 6.3 ┆ 2.5 ┆ 5.0 ┆ 1.9 ┆ virginica │
│ 6.5 ┆ 3.0 ┆ 5.2 ┆ 2.0 ┆ virginica │
│ 6.2 ┆ 3.4 ┆ 5.4 ┆ 2.3 ┆ virginica │
│ 5.9 ┆ 3.0 ┆ 5.1 ┆ 1.8 ┆ virginica │
└──────────────┴─────────────┴──────────────┴─────────────┴───────────┘
查看此文件位于 https://huggingface.co/datasets/nameexhaustion/polars-docs/blob/main/iris.jsonl
Parquet
print(
"""\
形状: (4, 3)
┌────────────┬────────────────────────────┬─────┐
│ 日期1 ┆ 日期2 ┆ x │
│ --- ┆ --- ┆ --- │
│ 日期 ┆ 日期时间[微秒] ┆ i32 │
╞════════════╪════════════════════════════╪═════╡
│ 2024-01-01 ┆ 2023-01-01 00:00:00 ┆ 1 │
│ 2024-02-01 ┆ 2023-02-01 00:00:00 ┆ 2 │
│ 2024-03-01 ┆ 空值 ┆ 3 │
│ 空值 ┆ 2023-03-01 01:01:01.000001 ┆ 4 │
└────────────┴────────────────────────────┴─────┘
"""
)
shape: (4, 3)
┌────────────┬────────────────────────────┬─────┐
│ date1 ┆ date2 ┆ x │
│ --- ┆ --- ┆ --- │
│ date ┆ datetime[μs] ┆ i32 │
╞════════════╪════════════════════════════╪═════╡
│ 2024-01-01 ┆ 2023-01-01 00:00:00 ┆ 1 │
│ 2024-02-01 ┆ 2023-02-01 00:00:00 ┆ 2 │
│ 2024-03-01 ┆ null ┆ 3 │
│ null ┆ 2023-03-01 01:01:01.000001 ┆ 4 │
└────────────┴────────────────────────────┴─────┘
请查看此文件夹 https://huggingface.co/datasets/nameexhaustion/polars-docs/tree/main/hive_dates/
IPC
print(pl.scan_ipc("hf://spaces/nameexhaustion/polars-docs/orders.feather").collect())
shape: (10, 9)
┌────────────┬───────────┬───────────────┬──────────────┬───┬─────────────────┬─────────────────┬────────────────┬─────────────────────────┐
│ o_orderkey ┆ o_custkey ┆ o_orderstatus ┆ o_totalprice ┆ … ┆ o_orderpriority ┆ o_clerk ┆ o_shippriority ┆ o_comment │
│ --- ┆ --- ┆ --- ┆ --- ┆ ┆ --- ┆ --- ┆ --- ┆ --- │
│ i64 ┆ i64 ┆ str ┆ f64 ┆ ┆ str ┆ str ┆ i64 ┆ str │
╞════════════╪═══════════╪═══════════════╪══════════════╪═══╪═════════════════╪═════════════════╪════════════════╪═════════════════════════╡
│ 1 ┆ 36901 ┆ O ┆ 173665.47 ┆ … ┆ 5-LOW ┆ Clerk#000000951 ┆ 0 ┆ nstructions sleep │
│ ┆ ┆ ┆ ┆ ┆ ┆ ┆ ┆ furiously am… │
│ 2 ┆ 78002 ┆ O ┆ 46929.18 ┆ … ┆ 1-URGENT ┆ Clerk#000000880 ┆ 0 ┆ foxes. pending accounts │
│ ┆ ┆ ┆ ┆ ┆ ┆ ┆ ┆ at th… │
│ 3 ┆ 123314 ┆ F ┆ 193846.25 ┆ … ┆ 5-LOW ┆ Clerk#000000955 ┆ 0 ┆ sly final accounts │
│ ┆ ┆ ┆ ┆ ┆ ┆ ┆ ┆ boost. care… │
│ 4 ┆ 136777 ┆ O ┆ 32151.78 ┆ … ┆ 5-LOW ┆ Clerk#000000124 ┆ 0 ┆ sits. slyly regular │
│ ┆ ┆ ┆ ┆ ┆ ┆ ┆ ┆ warthogs c… │
│ 5 ┆ 44485 ┆ F ┆ 144659.2 ┆ … ┆ 5-LOW ┆ Clerk#000000925 ┆ 0 ┆ quickly. bold deposits │
│ ┆ ┆ ┆ ┆ ┆ ┆ ┆ ┆ sleep s… │
│ 6 ┆ 55624 ┆ F ┆ 58749.59 ┆ … ┆ 4-NOT SPECIFIED ┆ Clerk#000000058 ┆ 0 ┆ ggle. special, final │
│ ┆ ┆ ┆ ┆ ┆ ┆ ┆ ┆ requests … │
│ 7 ┆ 39136 ┆ O ┆ 252004.18 ┆ … ┆ 2-HIGH ┆ Clerk#000000470 ┆ 0 ┆ ly special requests │
│ 32 ┆ 130057 ┆ O ┆ 208660.75 ┆ … ┆ 2-HIGH ┆ Clerk#000000616 ┆ 0 ┆ ise blithely bold, │
│ ┆ ┆ ┆ ┆ ┆ ┆ ┆ ┆ regular req… │
│ 33 ┆ 66958 ┆ F ┆ 163243.98 ┆ … ┆ 3-MEDIUM ┆ Clerk#000000409 ┆ 0 ┆ uriously. furiously │
│ ┆ ┆ ┆ ┆ ┆ ┆ ┆ ┆ final requ… │
│ 34 ┆ 61001 ┆ O ┆ 58949.67 ┆ … ┆ 3-MEDIUM ┆ Clerk#000000223 ┆ 0 ┆ ly final packages. │
│ ┆ ┆ ┆ ┆ ┆ ┆ ┆ ┆ fluffily fi… │
└────────────┴───────────┴───────────────┴──────────────┴───┴─────────────────┴─────────────────┴────────────────┴─────────────────────────┘
查看此文件位于 https://huggingface.co/spaces/nameexhaustion/polars-docs/blob/main/orders.feather