这篇文章记录 huggingface 常用的功能
datasets
| func_name | usage explaination |
|---|---|
| ds.map | Apply a function to all the examples in the table (individually or in batches) and update the table. If your function returns a column that already exists, then it overwrites it. |
| ds.save_to_disk(“path/to/dataset/dir”) | save datasets(after preprocessing) to local directory |
| ds.load_from_disk(“path/to/dataset/dir”) | load |
dataset.map(... load_from_cache_file=False)