refactor: pipeline training with tf.data#1

Open

xxxyj wants to merge 5 commits intomainfrom

codex/optimize-lstm-training-pipeline

Owner

xxxyj commented Aug 17, 2025

Summary

replace Python generator with tf.data pipeline in training script
add prefetching for better GPU/CPU overlap
feed model using tf.data datasets for training and validation

Testing

python -m py_compile MarcDecoderWin/train.py
pytest (no tests found)

https://chatgpt.com/codex/tasks/task_b_68a1d8da03448320bf4bb52b2f346332


          refactor: pipeline training with tf.data

75f7b4b

xxxyj added the codex label

— with

ChatGPT Codex Connector

xxxyj added 4 commits

August 17, 2025 15:45


          perf: create tf.data pipeline from arrays

e837d47


          fix: pad datasets for concat

a5a6eca


          perf: cache dataset and prefetch to gpu

0eb6abe


          fix: save weights without unsupported options

b95489e

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels