how to run it on yarn

[tonywang-sh]

I didn't find submarine usage guide for running on yarn. Whether its ver 0.7 does support to run task on yarn? Any guide documents? Thanks.

[cdmikechen]

@tonywang-sh
Hi~ You can try 0.6.0. Here is the document https://submarine.apache.org/docs/0.6.0/adminDocs/yarn/

[tonywang-sh]

From v0.6.0 yarn document, there is the cmd line, "java org.apache.submarine.client.cli.Cli job run --name tf-job-001 \ ....". Where is the 'java org.apache.submarine.client.cli.Cli' implementation? I didn't find its package with Cli interface. Thanks.

[cdmikechen]

@tonywang-sh
It is in submarine-client, and you need to change tag to release-0.6.0

[tonywang-sh]

Does release v0.7 or later version support to run on yarn?

[tonywang-sh]

In addition, Does release-0.6.0 support hadoop 3.2.2? It reported errors when compiling the source codes of release-0.6.0 as the below.

[cdmikechen]

@tonywang-sh
Can you show your build command and mvn/java version?
There is a profile hadoop-3.2 in pom.xml, hadoop 3.2 should be compiled if it is configured correctly.

[tonywang-sh]

release-0.6.0 showed the errors with hadoop 3.2 as the below.

[cdmikechen]

@tonywang-sh
Can you show your build command and mvn/java version?
I run mvn clean install -DskipTests -Phadoop-3.2 and build succeeded.

[tonywang-sh]

I tried the cmd line, mvn clean install -DskipTests -Phadoop-3.2.2, but it failed. Another issue, release-0.7.0 or later versions don't have java code about yarn, how to run release-0.7.0 or later version on yarn?

[cdmikechen]

@tonywang-sh
You can not use hadoop-3.2.2, because there is no hadoop-3.2.2 profile in pom.xml. If you want to use hadoop 3.2.2, I think you need to change hadoop.version to 3.2.2.
From 0.7.0, we only support k8s, so that you may use 0.6.0 for yarn as latest version.

[tonywang-sh]

Thanks for your reply. because submarine v0.6.0 on yarn needs TonY framework, how to set application conf for TonY to meet kerberos verification of hadoop? I didn't find TonY conf setting from the website, https://submarine.apache.org/docs/0.6.0/userDocs/yarn/YARNRuntimeGuide/.

[cdmikechen]

@tonywang-sh
I have not tried kerberos, but in my experience, if you have logged into kerberos, hadoop will recognize kerberos authentication information by default.
If you encounter problems when trying kerberos you can leave a comment here and I or someone else may be able to help you out.