Skip to content

Commit bbb3e10

Browse files
DzmitryPihulskiDzmitryPihulski
authored
Merge pull request #33 from LLMSQL/28-create-examples-of-usage
examples added; vllm inference corrected; version bumped;
2 parents 2825af1 + b46657e commit bbb3e10

File tree

5 files changed

+739
-9
lines changed

5 files changed

+739
-9
lines changed

‎examples/inference_transformers.ipynb‎

Lines changed: 90 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,90 @@
1+
{
2+
"cells": [
3+
{
4+
"cell_type": "code",
5+
"execution_count": null,
6+
"id": "e71979fc",
7+
"metadata": {},
8+
"outputs": [
9+
{
10+
"name": "stderr",
11+
"output_type": "stream",
12+
"text": [
13+
"/workspace/.venv/lib/python3.10/site-packages/tqdm/auto.py:21: TqdmWarning: IProgress not found. Please update jupyter and ipywidgets. See https://ipywidgets.readthedocs.io/en/stable/user_install.html\n",
14+
" from .autonotebook import tqdm as notebook_tqdm\n",
15+
"2025-10-15 13:47:35,036 [INFO] llmsql-bench: Loading model from: EleutherAI/pythia-14m\n",
16+
"`torch_dtype` is deprecated! Use `dtype` instead!\n",
17+
"2025-10-15 13:47:36,581 [INFO] accelerate.utils.modeling: Based on the current allocation process, no modules could be assigned to the following devices due to insufficient memory:\n",
18+
" - 0: 25755648 bytes required\n",
19+
"These minimum requirements are specific to this allocation attempt and may vary. Consider increasing the available memory for these devices to at least the specified minimum, or adjusting the model config.\n",
20+
"2025-10-15 13:47:39,399 [INFO] llmsql-bench: Writing results to test_outputs.jsonl\n",
21+
"2025-10-15 13:47:39,400 [INFO] llmsql-bench: Using 5-shot prompt builder: build_prompt_5shot\n",
22+
"Generating: 0%| | 0/17 [00:00<?, ?it/s]Asking to truncate to max_length but no maximum length is provided and the model has no predefined maximum length. Default to no truncation.\n",
23+
"The following generation flags are not valid and may be ignored: ['temperature']. Set `TRANSFORMERS_VERBOSITY=info` for more details.\n",
24+
"A decoder-only architecture is being used, but right-padding was detected! For correct generation results, please set `padding_side='left'` when initializing the tokenizer.\n",
25+
"2025-10-15 13:48:01,043 [INFO] llmsql-bench: Saved batch 1: 5000/80330\n",
26+
"Generating: 6%|▌ | 1/17 [00:21<05:46, 21.64s/it]A decoder-only architecture is being used, but right-padding was detected! For correct generation results, please set `padding_side='left'` when initializing the tokenizer.\n",
27+
"2025-10-15 13:48:22,872 [INFO] llmsql-bench: Saved batch 2: 10000/80330\n",
28+
"Generating: 12%|█▏ | 2/17 [00:43<05:26, 21.75s/it]A decoder-only architecture is being used, but right-padding was detected! For correct generation results, please set `padding_side='left'` when initializing the tokenizer.\n",
29+
"2025-10-15 13:48:43,885 [INFO] llmsql-bench: Saved batch 3: 15000/80330\n",
30+
"Generating: 18%|█▊ | 3/17 [01:04<04:59, 21.41s/it]A decoder-only architecture is being used, but right-padding was detected! For correct generation results, please set `padding_side='left'` when initializing the tokenizer.\n",
31+
"2025-10-15 13:49:05,981 [INFO] llmsql-bench: Saved batch 4: 20000/80330\n",
32+
"Generating: 24%|██▎ | 4/17 [01:26<04:41, 21.68s/it]A decoder-only architecture is being used, but right-padding was detected! For correct generation results, please set `padding_side='left'` when initializing the tokenizer.\n",
33+
"2025-10-15 13:49:27,091 [INFO] llmsql-bench: Saved batch 5: 25000/80330\n",
34+
"Generating: 29%|██▉ | 5/17 [01:47<04:17, 21.48s/it]A decoder-only architecture is being used, but right-padding was detected! For correct generation results, please set `padding_side='left'` when initializing the tokenizer.\n",
35+
"2025-10-15 13:49:49,392 [INFO] llmsql-bench: Saved batch 6: 30000/80330\n",
36+
"Generating: 35%|███▌ | 6/17 [02:09<03:59, 21.76s/it]A decoder-only architecture is being used, but right-padding was detected! For correct generation results, please set `padding_side='left'` when initializing the tokenizer.\n",
37+
"2025-10-15 13:50:11,577 [INFO] llmsql-bench: Saved batch 7: 35000/80330\n",
38+
"Generating: 41%|████ | 7/17 [02:32<03:38, 21.90s/it]A decoder-only architecture is being used, but right-padding was detected! For correct generation results, please set `padding_side='left'` when initializing the tokenizer.\n",
39+
"2025-10-15 13:50:32,938 [INFO] llmsql-bench: Saved batch 8: 40000/80330\n",
40+
"Generating: 47%|████▋ | 8/17 [02:53<03:15, 21.73s/it]A decoder-only architecture is being used, but right-padding was detected! For correct generation results, please set `padding_side='left'` when initializing the tokenizer.\n",
41+
"2025-10-15 13:50:58,184 [INFO] llmsql-bench: Saved batch 9: 45000/80330\n",
42+
"Generating: 53%|█████▎ | 9/17 [03:18<03:02, 22.83s/it]A decoder-only architecture is being used, but right-padding was detected! For correct generation results, please set `padding_side='left'` when initializing the tokenizer.\n",
43+
"2025-10-15 13:51:22,430 [INFO] llmsql-bench: Saved batch 10: 50000/80330\n",
44+
"Generating: 59%|█████▉ | 10/17 [03:43<02:42, 23.26s/it]A decoder-only architecture is being used, but right-padding was detected! For correct generation results, please set `padding_side='left'` when initializing the tokenizer.\n",
45+
"2025-10-15 13:51:48,417 [INFO] llmsql-bench: Saved batch 11: 55000/80330\n",
46+
"Generating: 65%|██████▍ | 11/17 [04:09<02:24, 24.10s/it]A decoder-only architecture is being used, but right-padding was detected! For correct generation results, please set `padding_side='left'` when initializing the tokenizer.\n",
47+
"2025-10-15 13:52:13,406 [INFO] llmsql-bench: Saved batch 12: 60000/80330\n",
48+
"Generating: 71%|███████ | 12/17 [04:34<02:01, 24.37s/it]A decoder-only architecture is being used, but right-padding was detected! For correct generation results, please set `padding_side='left'` when initializing the tokenizer.\n",
49+
"2025-10-15 13:52:33,864 [INFO] llmsql-bench: Saved batch 13: 65000/80330\n",
50+
"Generating: 76%|███████▋ | 13/17 [04:54<01:32, 23.18s/it]A decoder-only architecture is being used, but right-padding was detected! For correct generation results, please set `padding_side='left'` when initializing the tokenizer.\n",
51+
"2025-10-15 13:52:55,954 [INFO] llmsql-bench: Saved batch 14: 70000/80330\n",
52+
"Generating: 82%|████████▏ | 14/17 [05:16<01:08, 22.85s/it]A decoder-only architecture is being used, but right-padding was detected! For correct generation results, please set `padding_side='left'` when initializing the tokenizer.\n",
53+
"2025-10-15 13:53:17,756 [INFO] llmsql-bench: Saved batch 15: 75000/80330\n",
54+
"Generating: 88%|████████▊ | 15/17 [05:38<00:45, 22.54s/it]A decoder-only architecture is being used, but right-padding was detected! For correct generation results, please set `padding_side='left'` when initializing the tokenizer.\n",
55+
"2025-10-15 13:53:39,280 [INFO] llmsql-bench: Saved batch 16: 80000/80330\n",
56+
"Generating: 94%|█████████▍| 16/17 [05:59<00:22, 22.23s/it]A decoder-only architecture is being used, but right-padding was detected! For correct generation results, please set `padding_side='left'` when initializing the tokenizer.\n",
57+
"2025-10-15 13:53:41,689 [INFO] llmsql-bench: Saved batch 17: 80330/80330\n",
58+
"Generating: 100%|██████████| 17/17 [06:02<00:00, 21.31s/it]\n",
59+
"2025-10-15 13:53:41,690 [INFO] llmsql-bench: Generation complete — total: 80330\n"
60+
]
61+
}
62+
],
63+
"source": [
64+
"from llmsql import inference_transformers\n",
65+
"results = inference_transformers(model_or_model_name_or_path=\"EleutherAI/pythia-14m\", output_file=\"test_output.jsonl\", batch_size=5000, do_sample=False)"
66+
]
67+
}
68+
],
69+
"metadata": {
70+
"kernelspec": {
71+
"display_name": "llmsql-benchmark-3.11",
72+
"language": "python",
73+
"name": "python3"
74+
},
75+
"language_info": {
76+
"codemirror_mode": {
77+
"name": "ipython",
78+
"version": 3
79+
},
80+
"file_extension": ".py",
81+
"mimetype": "text/x-python",
82+
"name": "python",
83+
"nbconvert_exporter": "python",
84+
"pygments_lexer": "ipython3",
85+
"version": "3.11.13"
86+
}
87+
},
88+
"nbformat": 4,
89+
"nbformat_minor": 5
90+
}

0 commit comments

Comments
 (0)