|
1 | | -# goyarn |
2 | | -use goyarn as native go clients for Apache Hadoop YARN. |
| 1 | +<h1 align="center"> |
| 2 | + <p align="center">Koordinator goyarn</p> |
| 3 | + <a href="https://koordinator.sh"><img src="https://github.com/koordinator-sh/koordinator/raw/main/docs/images/koordinator-logo.jpeg" alt="Koordinator"></a> |
| 4 | +</h1> |
3 | 5 |
|
4 | | -It includes an early version of Hadoop IPC client and requisite YARN client libraries to implement YARN applications completely in go (both YARN application-client and application-master). |
| 6 | +[](https://opensource.org/licenses/Apache-2.0) |
| 7 | +[](https://github.com/koordinator-sh/goyarn/releases/latest) |
| 8 | +[](https://github.com/koordinator-sh/goyarn/actions/workflows/ci.yaml) |
| 9 | +[](https://goreportcard.com/report/github.com/koordinator-sh/goyarn) |
| 10 | +[](https://codecov.io/github/koordinator-sh/goyarn) |
| 11 | +[](CONTRIBUTING.md) |
| 12 | +[](https://join.slack.com/t/koordinator-sh/shared_invite/zt-1756qoub4-Cn4~esfdlfAPsD7cwO2NzA) |
5 | 13 |
|
6 | | -Koordinator extends `github.com/hortonworks/gohadoop` by implementing resource manager administration service and other clients. |
7 | 14 |
|
8 | | -# Notes: |
9 | | -Set HADOOP_CONF_DIR environment variable, and ensure the conf directory contains both *-default.xml and *-site.xml files. |
10 | | -rm_update_node_resource.go is an example go YARN rpc client of rm-admin: call update node resource to do the updates. |
| 15 | +English | [简体中文](./README-zh_CN.md) |
| 16 | +## Introduction |
11 | 17 |
|
12 | | -# Run rm_update_node_resource example |
13 | | -Change the `host` and `port` to target node id. |
| 18 | +Koordinator has supported hybrid orchestration workloads on Kubernetes, so that batch jobs can use the requested but unused resource |
| 19 | +as `koord-batch` priority and `BE` QoS class to improve the cluster utilization. However, there still lots of applications |
| 20 | +running beyond K8s such as Apache Haddop YARN. As a resource management platform in BigData ecosystem, YARN has supported |
| 21 | +numbers of computing engines including MapReduce, Spark, Flink, Presto, etc. |
14 | 22 |
|
15 | | -Execute command: |
| 23 | +In order to extend the co-location scenario of Koordinator, now the community has provided Hadoop YARN extended suits |
| 24 | +in BigData ecosystem, supporting running Hadoop YARN jobs by `koord-batch` resources with other K8s pods. The Koordiantor |
| 25 | +YARN suits has following characters: |
16 | 26 |
|
17 | | -```shell script |
18 | | -$ HADOOP_CONF_DIR=conf go run pkg/yarn/client/examples/rm_update_node_resource.go |
19 | | -``` |
| 27 | +- Open-Source native: implement against open-sourced version of Hadoop YARN; so there is no hack inside YARN modules. |
| 28 | +- Unifed resource priority and QoS strategy: the suits aims to the `koord-batch` priority of Koordinator, and also managed by QoS strategies of koordlet. |
| 29 | +- Resource sharing on node level: node resources of `koord-batch` priority can be requested by tasks of YARN or `Batch` pods both. |
| 30 | +- Adaptive for multiple environments: the suits can be run under any environment, including public cloud or IDC. |
20 | 31 |
|
21 | | -# Run yarn-operator |
22 | | -Install `koordinator` according to [doc](https://koordinator.sh/docs/installation/). |
| 32 | +## Quick Start |
23 | 33 |
|
24 | | -Add annotation to node with YARN node ID |
25 | | -```shell script |
26 | | -kubectl annotate node --overwrite ${k8s.node.name} node.yarn.koordinator.sh=${yarn.node.id} |
27 | | -``` |
| 34 | +You can view the full documentation from the [Koordinator website](https://koordinator.sh/docs). |
28 | 35 |
|
29 | | -Change the `yarn.resourcemanager.admin.address` in config/manager/configmap.yaml |
| 36 | +- Install or upgrade Koordinator with [the latest version](https://koordinator.sh/docs/installation). |
| 37 | +- Referring to [best practices](https://koordinator.sh/docs/next/best-practices/colocation-of-hadoop-yarn), there will be |
| 38 | + detailed instructions for running Hadoop YARN jobs with Koordinator batch resources in K8s. |
30 | 39 |
|
31 | | -Execute command: |
32 | | -```shell script |
33 | | -$ kubectl apply -f config/manager/ |
34 | | -``` |
| 40 | +## Code of conduct |
| 41 | + |
| 42 | +The Koordinator community is guided by our [Code of Conduct](https://github.com/koordinator-sh/koordinator/CODE_OF_CONDUCT.md), |
| 43 | +which we encourage everybody to read before participating. |
| 44 | + |
| 45 | +In the interest of fostering an open and welcoming environment, we as contributors and maintainers pledge to making |
| 46 | +participation in our project and our community a harassment-free experience for everyone, regardless of age, body size, |
| 47 | +disability, ethnicity, level of experience, education, socio-economic status, |
| 48 | +nationality, personal appearance, race, religion, or sexual identity and orientation. |
| 49 | + |
| 50 | +## Contributing |
| 51 | + |
| 52 | +You are warmly welcome to hack on Koordinator. We have prepared a detailed guide [CONTRIBUTING.md](https://github.com/koordinator-sh/koordinator/ONTRIBUTING.md). |
| 53 | + |
| 54 | +## Community |
| 55 | + |
| 56 | +The [koordinator-sh/community repository](https://github.com/koordinator-sh/community) hosts all information about |
| 57 | +the community, membership and how to become them, developing inspection, who to contact about what, etc. |
| 58 | + |
| 59 | +We encourage all contributors to become members. We aim to grow an active, healthy community of contributors, reviewers, |
| 60 | +and code owners. Learn more about requirements and responsibilities of membership in |
| 61 | +the [community membership](https://github.com/koordinator-sh/community/blob/main/community-membership.md) page. |
| 62 | + |
| 63 | +Active communication channels: |
| 64 | + |
| 65 | +- Bi-weekly Community Meeting (APAC, *Chinese*): |
| 66 | + - Tuesday 19:30 GMT+8 (Asia/Shanghai) |
| 67 | + - [Meeting Link(DingTalk)](https://meeting.dingtalk.com/j/cgTTojEI8Zy) |
| 68 | + - [Notes and agenda](https://shimo.im/docs/m4kMLdgO1LIma9qD) |
| 69 | +- Slack(English): [koordinator channel](https://kubernetes.slack.com/channels/koordinator) in Kubernetes workspace |
| 70 | +- DingTalk(Chinese): Search Group ID `33383887` or scan the following QR Code |
| 71 | + |
| 72 | +<div> |
| 73 | + <img src="https://github.com/koordinator-sh/koordinator/raw/main/docs/images/dingtalk.png" width="300" alt="Dingtalk QRCode"> |
| 74 | +</div> |
| 75 | + |
| 76 | +## License |
| 77 | + |
| 78 | +Koordinator is licensed under the Apache License, Version 2.0. See [LICENSE](./LICENSE) for the full license text. |
| 79 | +<!-- |
| 80 | +
|
| 81 | +## Star History |
| 82 | +
|
| 83 | +[](https://star-history.com/#koordinator-sh/koordinator&Date) |
| 84 | +--> |
0 commit comments