feat: introduce democracy (#965)

* feature: democracy draft

* fix: basic democracy setup

* fix: cargo fmt

* feat: setup maintenance manager origin

* feat: root or tech comm cna set collators

* feat: root or tech commm can set identities

* fix: remove democracy benchmarks

* fix: pallet-presence

* fix: externalDefaultOrigin

* fix: all tech comm can suspend/resume xcm

* test(governance): first batch

* revert(democracy): launch-config

* refactor(governance): address suggestions

* refactor(governance): switch to frame's democracy pallet

* chore: update deps

* chore: fix deps

* chore: fix xcm mod

* feature(governance): add new associated types from 42 v

* feat(governance): add ranked collective

* refactor(governance): split codebase to entities

* feature(governance): pallets config & refactor

* feat(governance): fix tracks and mapping

* feature(governance): tune types for maintenance&xcm

* feat(governance): rename pallet name to integrate with apps ui

* feature(governance): remove unused origins

* feature(governance): rename track

* feature(governance): change rank mapper & democracy track id

* feat(governance): configuration impl

* refactor: remove old gov tests, add new test-cases + minor fix

* feature(governance): aded events, config impl, bencmarks

* feature(governance): set correct weight to `set_governance_arg`

* refactor(governance): `ClassToRankMapper`

* Update runtime/common/config/pallets/governance/types.rs

Co-authored-by: Daniel Shiposha <mrshiposha@gmail.com>

* Update pallets/configuration/src/lib.rs

Co-authored-by: Daniel Shiposha <mrshiposha@gmail.com>

* Update runtime/common/config/pallets/governance/council_collective.rs

Co-authored-by: Daniel Shiposha <mrshiposha@gmail.com>

* Update runtime/common/config/pallets/governance/council_collective.rs

Co-authored-by: Daniel Shiposha <mrshiposha@gmail.com>

* Update runtime/common/config/pallets/governance/council_collective.rs

Co-authored-by: Daniel Shiposha <mrshiposha@gmail.com>

* Update runtime/common/config/pallets/governance/technical_committee.rs

Co-authored-by: Daniel Shiposha <mrshiposha@gmail.com>

* Update runtime/common/config/pallets/governance/technical_committee.rs

Co-authored-by: Daniel Shiposha <mrshiposha@gmail.com>

* Update runtime/common/config/pallets/governance/technical_committee.rs

Co-authored-by: Daniel Shiposha <mrshiposha@gmail.com>

* refactor(governance): code organization

* fix: format

* feature(governance): added new types

* refactor: gov origins, minor fixes

* chore: regenerate types

* chore: update Cargo.lock

* fix: governance playground fixes

* fix: cargo fmt

* Initialize Council and Technical Committee tests

* refactor: governance params + test timings

* feat(tests): add time-to-block constants

* refactor(tests): event helpers

* feat(tests): add fellowship helpers

* fix(tests): xcm tests use new event helpers

* test: fellowship basic tests

* feat: add fellowship timing MIN_ENACTMENT_PERIOD

* refactor(tests): instroduce governance test infrastructure

* fix: cargo fmt

* fix: yarn lint

* feat: add moreThanHalfCouncil

* fix: council test

* test: fellowship

* test(council): added impl for 14 cases + helpers + refac

* test(council): prime member impl + Closed events parser

* test(council): tests + helpers

* test(goverment): refactor test organization

* test(governance): comm tests

* fix: quartz test-env feature

* feat: add gov workflow, add describeGov

* test: democracy

* fix: optimize techcomm imports

* fix: exclude unique from gov workflow

* Update tests/src/governance/council.test.ts

Co-authored-by: Daniel Shiposha <mrshiposha@gmail.com>

* fix(test governance): origin for council members

* No more state in governance test utils

* fix typo

* fix(test gov): dummy prop impl

* test(governance): refactor & remove unused

* refactor(gobv test): initCouncil

* chore(governance): after rebase fixes

* fix: fmt

* fix: fmt & types

* chore: polkadot types

* chore(gov): fix gov workflow

* change api port to rpc for 0.9.43 polkadot

* test(gov): fix waiter for block

Added fn for hard reset some of pallets

* test(gov): fix waiter for block

Added hard reset for some of the pallets

* Update runtime/common/config/pallets/governance/fellowship.rs

Co-authored-by: Yaroslav Bolyukin <iam@lach.pw>

* add var wasm_name for goverments workflow for running sapphire

* fix(runtime): spces -> tabs

* add var wasm_name for goverments workflow for running sapphire

* refactor(governance):

- Removed debug test info
- Added new test for current Add\Remove assoc. type
- Changed gov settings
-Added helper `getMembers` for RankedCollective group

* refactor(governacne): nitpicks

* refactor(governance):construct runtime.

Fixed condition for skipping unique scheduler tests.

Added full name codec dep .

* fix(gov): added `origins` to pallets list

---------

Co-authored-by: Daniel Shiposha <dshiposha@usetech.com>
Co-authored-by: Fahrrader <fhakimov@usetech.com>
Co-authored-by: Daniel Shiposha <mrshiposha@gmail.com>
Co-authored-by: Andy Smith <andreyk@unique.network>
Co-authored-by: Konstantin Astakhov <kastakhov@usetech.com>
Co-authored-by: Yaroslav Bolyukin <iam@lach.pw>

.docker/docker-compose.gov.j2

@@ -0,0 +1,24 @@
+version: "3.5"
+
+services:
+  node-dev:
+    build:
+      args:
+        - "RUST_TOOLCHAIN={{ RUST_TOOLCHAIN }}"
+        - "NETWORK={{ NETWORK }}"
+        - "WASM_NAME={{ WASM_NAME }}"
+      context: ../
+      dockerfile: .docker/Dockerfile-chain-dev
+    image: node-dev
+    container_name: node-dev
+    expose:
+      - 9944
+      - 9933
+    ports:
+      - 127.0.0.1:9944:9944
+      - 127.0.0.1:9933:9933
+    logging:
+      options:
+        max-size: "1m"
+        max-file: "3"
+    command: cargo run --release --features={{ NETWORK }}-runtime,{{ WASM_NAME }}-runtime/test-env -- --dev -linfo --rpc-cors=all --unsafe-rpc-external

.github/workflows/ci-develop.yml

@@ -61,7 +61,12 @@ jobs:
     uses: ./.github/workflows/node-only-update.yml
     secrets: inherit
 
+  gov:
+    if: ${{ (github.event.pull_request.draft == false && contains( github.event.pull_request.labels.*.name, 'CI-gov')) }}       # Conditional check for draft & labels per job.
+    uses: ./.github/workflows/gov.yml
+    secrets: inherit # pass all secrets from initial workflow to nested
+
   codestyle:
     if: github.event.pull_request.draft == false                                                                             # Conditional check for draft per job.
     uses: ./.github/workflows/codestyle.yml
-    secrets: inherit
+    secrets: inherit

.github/workflows/gov.yml

@@ -0,0 +1,118 @@
+# Governance tests in --dev mode with test-env feature enabled to reduce gov timings
+name: governance tests
+
+# Triger: only call from main workflow(re-usable workflows)
+on:
+  workflow_call:
+
+# A workflow run is made up of one or more jobs that can run sequentially or in parallel
+jobs:
+  prepare-execution-marix:
+    name: Prepare execution matrix
+
+    runs-on: self-hosted-ci
+    outputs:
+      matrix: ${{ steps.create_matrix.outputs.matrix }}
+
+    steps:
+      - name: Clean Workspace
+        uses: AutoModality/action-clean@v1.1.0
+
+      # Checks-out your repository under $GITHUB_WORKSPACE, so your job can access it
+      - uses: actions/checkout@v3.1.0
+        with:
+          ref: ${{ github.head_ref }} #Checking out head commit
+
+      - name: Read .env file
+        uses: xom9ikk/dotenv@v2
+
+      - name: Create Execution matrix
+        uses: CertainLach/create-matrix-action@v4
+        id: create_matrix
+        with:
+          matrix: |
+            network {quartz}, wasm_name {quartz}
+            network {opal}, wasm_name {opal}
+            network {sapphire}, wasm_name {quartz}
+
+  dev_build_int_tests:
+    needs: prepare-execution-marix
+    # The type of runner that the job will run on
+    runs-on: [self-hosted-ci, medium]
+    timeout-minutes: 1380
+
+    name: ${{ matrix.network }}
+    strategy:
+      matrix:
+        include: ${{fromJson(needs.prepare-execution-marix.outputs.matrix)}}
+
+    continue-on-error: true #Do not stop testing of matrix runs failed.  As it decided during PR review - it required 50/50& Let's check it with false.
+
+    steps:
+      - name: Clean Workspace
+        uses: AutoModality/action-clean@v1.1.0
+
+      # Checks-out your repository under $GITHUB_WORKSPACE, so your job can access it
+      - uses: actions/checkout@v3.1.0
+        with:
+          ref: ${{ github.head_ref }} #Checking out head commit
+
+      - name: Read .env file
+        uses: xom9ikk/dotenv@v2
+
+      - name: Generate ENV related extend file for docker-compose
+        uses: cuchi/jinja2-action@v1.2.0
+        with:
+          template: .docker/docker-compose.gov.j2
+          output_file: .docker/docker-compose.${{ matrix.network }}.yml
+          variables: |
+            RUST_TOOLCHAIN=${{ env.RUST_TOOLCHAIN }}
+            NETWORK=${{ matrix.network }}
+            WASM_NAME=${{ matrix.wasm_name }}
+
+      - name: Show build configuration
+        run: cat .docker/docker-compose.${{ matrix.network }}.yml
+
+      - name: Build the stack
+        run: docker-compose -f ".docker/docker-compose.${{ matrix.network }}.yml" up -d --build --remove-orphans
+
+      - uses: actions/setup-node@v3.5.1
+        with:
+          node-version: 16
+
+      - name: Run tests
+        working-directory: tests
+        run: |
+          yarn install
+          yarn add mochawesome
+          ./scripts/wait_for_first_block.sh
+          echo "Ready to start tests"
+          yarn polkadot-types
+          NOW=$(date +%s) && yarn testGovernance --reporter mochawesome --reporter-options reportFilename=test-${NOW}
+        env:
+          RPC_URL: http://127.0.0.1:9944/
+
+      - name: Test Report
+        uses: phoenix-actions/test-reporting@v10
+        id: test-report
+        if: success() || failure() # run this step even if previous step failed
+        with:
+          name: int test results - ${{ matrix.network }} # Name of the check run which will be created
+          path: tests/mochawesome-report/test-*.json # Path to test results
+          reporter: mochawesome-json
+          fail-on-error: 'false'
+
+      - name: Read output variables
+        run: |
+          echo "url is ${{ steps.test-report.outputs.runHtmlUrl }}"
+
+      - name: Stop running containers
+        if: always() # run this step always
+        run: docker-compose -f ".docker/docker-compose.${{ matrix.network }}.yml" down
+
+      - name: Remove builder cache
+        if: always() # run this step always
+        run: |
+          docker builder prune -f -a
+          docker system prune -f
+          docker image prune -f -a