Skip to content

Commit 7b48712

Browse files
troiantroian
committed
refactor(operator): feature discovery
refs akash-network/support#141 Signed-off-by: Artur Troian <troian.ap@gmail.com>
1 parent af85c28 commit 7b48712

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

80 files changed

+6216
-2138
lines changed

‎.github/workflows/integration-tests.yaml

Lines changed: 54 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -16,27 +16,68 @@ jobs:
1616
KIND_NAME: kube
1717
runs-on: ubuntu-latest
1818
steps:
19-
- uses: actions/checkout@v3
19+
- run: echo "GOPATH=$GITHUB_WORKSPACE/go" >> $GITHUB_ENV
20+
- run: mkdir -p ${{ env.GOPATH }}/{src,bin,pkg}
21+
- uses: actions/checkout@v4
2022
with:
2123
fetch-depth: 0
24+
set-safe-directory: true
25+
path: go/src/github.com/akash-network/provider
2226
- name: Detect required Go version
27+
working-directory: "${{ env.GOPATH }}/src/github.com/akash-network/provider"
2328
run: |
2429
toolchain=$(./script/tools.sh gotoolchain | sed 's/go*//')
2530
echo "GOVERSION=${toolchain}" >> $GITHUB_ENV
2631
- uses: actions/setup-go@v4
2732
with:
2833
go-version: "${{ env.GOVERSION }}"
29-
- name: Setup direnv
30-
uses: HatsuneMiku3939/direnv-action@v1
34+
- name: Install direnv
35+
run: curl -sfL https://direnv.net/install.sh | bash
36+
- name: Hook direnv to bash
37+
run: echo 'eval "$(direnv hook bash)"' >> $HOME/.bashrc
38+
- name: Direnv allow
39+
run: direnv allow ${{ env.GOPATH }}/src/github.com/akash-network/provider
40+
- name: Checkout akash-api
41+
working-directory: ${{ env.GOPATH }}/src/github.com/akash-network/provider
42+
run: |
43+
AKASH_API_VERSION=$(go list -mod=readonly -m -f '{{ .Version }}' github.com/akash-network/akash-api)
44+
echo "AKASH_API_VERSION=$AKASH_API_VERSION" >> "$GITHUB_ENV"
45+
- uses: actions/checkout@v4
46+
with:
47+
repository: "akash-network/akash-api"
48+
fetch-depth: 0
49+
set-safe-directory: true
50+
path: go/src/github.com/akash-network/akash-api
51+
ref: ${{ env.AKASH_API_VERSION }}
52+
- name: Setup akash-api
53+
run: |
54+
direnv allow ${{ env.GOPATH }}/src/github.com/akash-network/akash-api
55+
cd ${{ env.GOPATH }}/src/github.com/akash-network/akash-api
56+
cat "$GITHUB_ENV" > "$GITHUB_WORKSPACE/env.backup"
57+
direnv export gha >> "$GITHUB_ENV"
58+
- name: Add akash-api to go.work
59+
working-directory: ${{ env.GOPATH }}/src/github.com/akash-network/akash-api
60+
run: |
61+
make modvendor
62+
printf "use (\n\t.\n)\n" > ../provider/go.work
63+
printf "\ngo ${{ env.GOVERSION }}\n" >> ../provider/go.work
64+
printf "\nreplace (\n\tgithub.com/akash-network/akash-api => ../akash-api\n)\n" >> ../provider/go.work
65+
cat "$GITHUB_WORKSPACE/env.backup" > "$GITHUB_ENV"
66+
- name: Setup direnv for provider
67+
run: direnv export gha >> "$GITHUB_ENV"
68+
working-directory: ${{ env.GOPATH }}/src/github.com/akash-network/provider
3169
- name: Fetch kind version from go modules
70+
working-directory: ${{ env.GOPATH }}/src/github.com/akash-network/provider
3271
run: echo "KIND_VERSION=$(go list -mod=readonly -m -f '{{ .Version }}' sigs.k8s.io/kind)" >> $GITHUB_ENV
3372
- name: Set up QEMU
3473
uses: docker/setup-qemu-action@v2
3574
- name: Set up Docker Buildx
3675
uses: docker/setup-buildx-action@v2
3776
- name: Go mod tidy
77+
working-directory: ${{ env.GOPATH }}/src/github.com/akash-network/provider
3878
run: go mod tidy
3979
- name: Make node scripts executable
80+
working-directory: ${{ env.GOPATH }}/src/github.com/akash-network/provider
4081
run: make chmod-akash-scripts
4182
- uses: helm/kind-action@v1
4283
with:
@@ -45,18 +86,28 @@ jobs:
4586
cluster_name: "${{ env.KIND_NAME }}"
4687
config: ./_run/kube/kind-config.yaml
4788
- name: Configure Kind cluster
89+
working-directory: ${{ env.GOPATH }}/src/github.com/akash-network/provider
4890
run: KUSTOMIZE_INSTALLS=akash-operator-inventory make -s -C _run/kube kube-cluster-setup-e2e-ci
4991
- name: k8s-ingress
92+
working-directory: ${{ env.GOPATH }}/src/github.com/akash-network/provider
5093
run: make -s -C _run/kube kind-k8s-ip
5194
- name: Kube Environment
95+
working-directory: ${{ env.GOPATH }}/src/github.com/akash-network/provider
5296
run: |
5397
kubectl config view
5498
kubectl cluster-info
5599
kubectl get pods,ingress,svc -A
100+
__pod=$(kubectl -n akash-services get pods -l akash.network/component=operator -l akash.network/component=inventory -l app.kubernetes.io/name=operator-inventory-node --no-headers -o custom-columns=":metadata.name")
101+
make -s -C _run/kube kube-deployment-rollout-operator-inventory
102+
kubectl -n akash-services port-forward --address 0.0.0.0 pod/${__pod} 8444:8081 &
103+
./script/inventory-test.sh --host=localhost:8444 --mode=plaintext akash.inventory.v1.NodeRPC/QueryNode
56104
- name: Run E2E Tests
105+
working-directory: ${{ env.GOPATH }}/src/github.com/akash-network/provider
57106
run: make test-e2e-integration
58107
- name: Run K8s Tests
108+
working-directory: ${{ env.GOPATH }}/src/github.com/akash-network/provider
59109
run: make test-k8s-integration
60110
- name: Post-Run Kube Environment
111+
working-directory: ${{ env.GOPATH }}/src/github.com/akash-network/provider
61112
run: |
62113
kubectl get ns,pods,ingress,svc -A

‎_docs/development-environment.md

Lines changed: 252 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,252 @@
1+
# Setting-up development environment
2+
3+
**Warning**
4+
All links to the `provider` repo are referencing to the `gpu` branch. As soon as `gpu` is merged into `main` all links need update.
5+
6+
This page covers setting up development environment for both [node](https://github.com/akash-network/node) and [provider](https://github.com/akash-network/provider) repositories.
7+
The provider repo elected as placeholder for all the scripts as it depends on the `node` repo (Better explanation?)
8+
Should you already know what this guide is all about - feel free to explore [examples](#how-to-use-runbook)
9+
10+
## Code
11+
12+
Checkout code if not done so into place of your convenience.
13+
For this example, repositories will be located in `~/go/src/github.com/akash-network`
14+
15+
Checkout below assumes `git` is set to use SSH connection to GitHub
16+
17+
```shell
18+
cd ~/go/src/github.com/akash-network # all commands below assume this as current directory
19+
git clone git@github.com:akash-netowrk/node
20+
git clone git@github.com:akash-netowrk/provider
21+
```
22+
23+
## Requirements
24+
25+
- `Go` must be installed. Both projects are keeping up-to-date with major version on development branches.
26+
Both repositories are using the latest version of the Go, however only minor that has to always match.
27+
28+
### Install tools
29+
30+
Run following script to install all system-wide tools.
31+
Currently supported host platforms:
32+
33+
- MacOS
34+
- Debian based OS
35+
PRs with another hosts are welcome (except Windows)
36+
37+
```shell
38+
./provider/script/install_dev_dependencies.sh
39+
```
40+
41+
## How it works
42+
43+
### General behaviour
44+
45+
All examples are located within [_run](https://github.com/akash-network/provider/blob/gpu/_run) directory.
46+
[Commands](#commands) are implemented as `make` targets.
47+
48+
There are three ways we use to set up the k8s cluster.
49+
50+
- kind
51+
- minukube
52+
- ssh
53+
54+
Both `kind` and `minikube` are e2e, i.e. the configuration is capable of spinning up cluster and the local host, whereas `ssh` expects cluster to be configured before use.
55+
56+
### Runbook
57+
58+
There are four configuration variants, each presented as directory within [_run](https://github.com/akash-network/provider/blob/gpu/_run).
59+
60+
- `kube` - uses `kind` to set up local cluster. It is widely used by e2e testing of the provider. Provider and the node run as host services. All operators run as kubernetes deployments.
61+
- `single` - uses `kind` to set up local cluster. Main difference is both node and provider (and all operators) are running within k8s cluster as deployments. (at some point we will merge `single`
62+
with `kube` and call it `kind`)
63+
- `minikube` - not in use for now
64+
- `ssh` - expects cluster to be up and running. mainly used to test sophisticated features like `GPU` or `IP leases`
65+
66+
The only difference between environments above is how they set up. Once running, all commands are the same.
67+
68+
Running through the entire runbook requires multiples terminals.
69+
Each command is marked __t1__-__t3__ to indicate a suggested terminal number.
70+
71+
If at any point something goes wrong and cluster needs to be run from the beginning:
72+
73+
```shell
74+
cd _run/<kube|single|ssh>
75+
make kube-cluster-delete
76+
make clean
77+
```
78+
79+
### Kustomize
80+
81+
TBD
82+
83+
#### Parameters
84+
85+
| Name | Default value | Effective on target(s) | Notes |
86+
|:---------------------|:------------------------------------------------------------------------------:|------------------------------------------------------------------------------------------------------------------------------------------|-------|
87+
| `SKIP_BUILD` | `false` | |
88+
| `DSEQ` | `1` | `deployment-*`<br/>`lease-*`<br/>`bid-*`<br/>`send-manifest` |
89+
| `OSEQ` | `1` | `deployment-*`<br/>`lease-*`<br/>`bid-*`<br/>`send-manifest` |
90+
| `GSEQ` | `1` | `deployment-*`<br/>`lease-*`<br/>`bid-*`<br/>`send-manifest` |
91+
| `KUSTOMIZE_INSTALLS` | Depends on runbook<br/>Refer to each runbook's `Makefile` to see default value | `kustomize-init`<br/>`kustomize-templates`<br/>`kustomize-set-images`<br/>`kustomize-configure-services`<br/>`kustomize-deploy-services` | |
92+
93+
##### Keys
94+
95+
Each configuration creates four [keys](https://github.com/akash-network/provider/blob/gpu/_run/common.mk#L40..L43):
96+
They keys are assigned to the targets and under normal circumstances there is no need to alter it. However, it can be done with setting `KEY_NAME`:
97+
98+
```shell
99+
# create provider from **provider** key
100+
make provider-create
101+
102+
# create provider from custom key
103+
KEY_NAME=other make provider-create
104+
```
105+
106+
#### How to use runbook
107+
108+
##### Kube
109+
110+
This runbook requires three terminals
111+
112+
1. Open runbook
113+
114+
__all three terminals__
115+
```shell
116+
cd _run/kube
117+
```
118+
119+
2. Create and provision local kind cluster.
120+
121+
__t1 run__
122+
```shell
123+
make kube-cluster-setup
124+
```
125+
3. Start akash node
126+
127+
__t2 run__
128+
```shell
129+
make node-run
130+
```
131+
4. Create provider
132+
133+
__t1 run__
134+
```shell
135+
make provider-create
136+
```
137+
138+
5. Start the provider
139+
140+
__t3 run__
141+
```shell
142+
make provider-create
143+
```
144+
145+
6. Start the provider
146+
147+
__t1 run__
148+
```shell
149+
make provider-create
150+
```
151+
152+
7. __t1__ Create a deployment. Check that the deployment was created. Take note of the `dseq` - deployment sequence:
153+
154+
```shell
155+
make deployment-create
156+
```
157+
158+
```shell
159+
make query-deployments
160+
```
161+
162+
After a short time, you should see an order created for this deployment with the following command:
163+
164+
```shell
165+
make query-orders
166+
```
167+
168+
The Provider Services Daemon should see this order and bid on it.
169+
170+
```shell
171+
make query-bids
172+
```
173+
174+
8. __t1 When a bid has been created, you may create a lease__
175+
176+
To create a lease, run
177+
178+
```shell
179+
make lease-create
180+
```
181+
182+
You can see the lease with:
183+
184+
```shell
185+
make query-leases
186+
```
187+
188+
You should now see "pending" inventory in the provider status:
189+
190+
```shell
191+
make provider-status
192+
```
193+
194+
9. __t1 Distribute Manifest__
195+
196+
Now that you have a lease with a provider, you need to send your
197+
workload configuration to that provider by sending it the manifest:
198+
199+
```shell
200+
make send-manifest
201+
```
202+
203+
You can check the status of your deployment with:
204+
205+
```shell
206+
make provider-lease-status
207+
```
208+
209+
You can reach your app with the following (Note: `Host:` header tomfoolery abound)
210+
211+
```shell
212+
make provider-lease-ping
213+
```
214+
215+
10. __t1 Get service status__
216+
217+
```sh
218+
make provider-lease-status
219+
```
220+
221+
Fetch logs from deployed service (all pods)
222+
223+
```sh
224+
make provider-lease-logs
225+
```
226+
227+
##### Kube for e2e tests
228+
229+
This runbook requires two terminal
230+
231+
1. Open runbook
232+
233+
__t1__
234+
```shell
235+
cd _run/kube
236+
```
237+
238+
2. Create and provision local kind cluster for e2e testing.
239+
240+
__t1 run__
241+
```shell
242+
make kube-cluster-setup-e2e
243+
244+
3. Run e2e tests
245+
246+
```shell
247+
make test-e2e-intergration
248+
```
249+
250+
##### Single
251+
252+
##### SSH

0 commit comments

Comments
 (0)