Add option to return SUCCEED when training is completed with some failed job tasks

[charliechen211]

App will be stopped when chief worker (or some certain jobtype), and continue training when other job task fails. It looks reasonable.

But in this circumstance, training is finished and return FAILED on YARN.

I think it's a contradiction.
If we will continue training after one worker fails, and training is finally completed, why should it return FAILED?
Users would feel confused, and don't know what they should do next. Rerun training for more accurate model, or use this model with a FAILED status?

They could config #412 if they needs fail fast, while return SUCCEED when training is completed if they needs training to continue.

I believe it is better to add an extra option for whether return SUCCEED or FAILED when training is completed with some failed job tasks, which user can specify on a per-job basis.

[charliechen211]

@oliverhu

[oliverhu]

Agreed, but I am refrained from making the APIs over complicated, any thought on how the configurations would look like?

[charliechen211]

@oliverhu
emmm...I made a PR, in which the configuration naming is not very good.
https://github.com/linkedin/TonY/pull/421/files

I also agreed that it's better to simplify the APIs, or we can cancel this configuration and all completed application with failure tasks will return SUCCEEDED, which is more reasonable.

[charliechen211]

CI failed with following logs:
Too long with no output (exceeded 10m0s): context deadline exceeded

Link: https://circleci.com/gh/morenn520/TonY/2?utm_campaign=vcs-integration-link&utm_medium=referral&utm_source=github-build-link

Maybe test has been run too long?

[oliverhu]

Not sure, will take a look tmr

[oliverhu]

Assign to Ankit to follow up on this issue