[Example] Refactor and Polish Cifar10-DeepSpeed Code Example.

[keli-wen]

Description

I'm grateful to the DeepSpeed Team for their detailed examples, which are invaluable for learners like me ✨.

As a beginner, I attempted to learn from the DeepSpeed official tutorial/example, similar to how I approached learning PyTorch's DDP with a minimum code example like Example.

Upon comprehensive review, I believe the Cifar10 may be the most suitable code example for me to gain practical experience with DeepSpeed.

However, I found some potential issues after going through it. The original example indeed illustrates how to train and test cifar10 using DeepSpeed, but:

• The code contains some unused or meaningless variables (such as with_cuda in argparse section) and comments (Sudden appearance of 2 passes in comments). Even some temporarily commented statements for debugging purposes.
• The entire code lacks structure and is more akin to a jupyter notebook rather than an official example. I believe it should align more with the style of gan_deepspeed_train.py for clarity.
• Some exploratory content is indeed interesting (such as showcasing images), but it can be confusing when approached with the aim of learning DeepSpeed best practices.

Therefore, I believe that the Cifar10-deepspeed code example could benefit from some additional polishing and refactoring.

Proposal

I have proposed some changes to the example, based on my understanding and experience. These changes are open for discussion and I welcome feedback for further refinement. The aim of these modifications is to enhance the following aspects:

• Clearer code structure with functions like get_ds_config, test, and main. This allows us to quickly target our goals by collapsing irrelevant sections of the code while reading.
• The example should minimize confusing variables/code/comments. I chose to remove completely unused variables and meaningless comments, including those interesting but not quite fitting visualizations and step-by-step debugging codes.
• Clearer output, especially in a distributed environment where output is inherently messy. I propose some logical modifications to enhance the clarity of the output, aiming for a more user-friendly experience.
• I attempted to format the entire function using some linters and simple comment formatting rules.

Log

To verify correctness, here are the logs I obtained after running it on my own server:

$ bash run_ds.sh
...
Accuracy of the network on the 10000 test images:  59 %
Accuracy of plane :  64 %
Accuracy of   car :  73 %
Accuracy of  bird :  56 %
Accuracy of   cat :  42 %
Accuracy of  deer :  48 %
Accuracy of   dog :  48 %
Accuracy of  frog :  59 %
Accuracy of horse :  62 %
Accuracy of  ship :  72 %
Accuracy of truck :  62 %
[2024-01-05 09:10:30,930] [INFO] [launch.py:347:main] Process 2124952 exits successfully.
[2024-01-05 09:10:30,931] [INFO] [launch.py:347:main] Process 2124957 exits successfully.
[2024-01-05 09:10:30,931] [INFO] [launch.py:347:main] Process 2124949 exits successfully.
[2024-01-05 09:10:30,932] [INFO] [launch.py:347:main] Process 2124937 exits successfully.

[keli-wen]

@microsoft-github-policy-service agree

[keli-wen]

Hi DeepSpeed Team.

Could you please advise on any further steps needed to progress with the merge of my PR.

[loadams]

Hi DeepSpeed Team.

Could you please advise on any further steps needed to progress with the merge of my PR.

Hi @keli-wen - we will prioritize looking at this PR and reviewing it soon, thanks for your contributions

[conglongli]

Tested. LGTM except two typos.