diff --git a/dev/apidocs/apidocs/index.html b/dev/apidocs/apidocs/index.html index 489273c398..65b4cb63cc 100644 --- a/dev/apidocs/apidocs/index.html +++ b/dev/apidocs/apidocs/index.html @@ -1,2 +1,2 @@ -API Documentation · metal-stack
Edit on GitHub
+API Documentation · metal-stack
Edit on GitHub
diff --git a/dev/apidocs/metal-api/index.html b/dev/apidocs/metal-api/index.html index 3b3f2f8ffb..7c4b8386cd 100644 --- a/dev/apidocs/metal-api/index.html +++ b/dev/apidocs/metal-api/index.html @@ -14,7 +14,7 @@ - + diff --git a/dev/development/client_libraries/index.html b/dev/development/client_libraries/index.html index 7e1a67b68a..c8b5a14f96 100644 --- a/dev/development/client_libraries/index.html +++ b/dev/development/client_libraries/index.html @@ -1,2 +1,2 @@ -Client Libraries · metal-stack
Edit on GitHub
+Client Libraries · metal-stack
Edit on GitHub
diff --git a/dev/development/contributing/index.html b/dev/development/contributing/index.html index a549bac3c0..d4abb630dd 100644 --- a/dev/development/contributing/index.html +++ b/dev/development/contributing/index.html @@ -1,2 +1,2 @@ -Contributing · metal-stack
Edit on GitHub

Contributing

This document describes the way we want to contribute code to the projects of metal-stack, which are hosted on github.com/metal-stack.

The document is meant to be understood as a general guideline for contributions, but not as burden to be placed on a developer. Use your best judgment when contributing code. Try to be as clean and precise as possible when writing code and try to make your code as maintainable and understandable as possible for other people.

Even if it should go without saying, we live an open culture of discussion, in which everybody is welcome to participate. We treat every contribution with respect and objectiveness with the general aim to write software of quality.

If you want, feel free to propose changes to this document in a pull request.

How Can I Contribute?

Open a Github issue in the project you would like to contribute. Within the issue, your idea can be discussed. It is also possible to directly create a pull request when the set of changes is relatively small.

Pull Requests

The process described here has several goals:

  • Maintain quality
  • Enable a sustainable system to review contributions
  • Enable documented and reproducible addition of contributions
  1. Create a meaningful issue describing the WHY? of your contribution
  2. Create a repository fork within the context of that issue.
  3. Create a Draft Pull Request to the master branch of the target repository.
  4. Develop, document and test your contribution (try not to solve more than one issue in a single pull request)
  5. Ask for merging your contribution by removing the draft marker
  6. If code owners are defined, try to assign the request to a code owner

General Objectives

This section contains language-agnostic topics that all metal-stack projects are trying to follow.

Code Ownership

The code base is owned by the entire team and every member is allowed to contribute changes to any of the projects. This is considered as collective code ownership[1].

As a matter of fact, there are persons in a project, which already have experience with the sources. These are defined directly in the repository's CODEOWNERS file. If you want to merge changes into the master branch, it is advisable to include code owners into the proecess of discussion and merging.

Microservices

One major ambition of metal-stack is to follow the idea of microservices. This way, we want to achieve that we can

  • adapt to changes faster than with monolithic architectures,
  • be free of restrictions due to certain choices of technology,
  • leverage powerful traits of cloud infrastructures (e.g. high-scalability, high-availability, ...).

Programming Languages

We are generally open to write code in any language that fits best to the function of the software. However, we encourage golang to be the main language of metal-stack as we think that it makes development faster when not establishing too many different languages in our architecture. Reason for this is that we are striving for consistent behavior of the microservices, similar to what has been described for the Twelve-Factor App (see https://12factor.net/). We help enforcing unified behavior by allowing a small layer of shared code for every programming language. We will refer to this shared code as "libraries" for the rest of this document.

Artifacts

Artifacts are always produced by a CI process (Github Actions).

Docker images are published on the Github Container Registry of the metal-stack organization.

Binary artifacts or OS images can be uploaded to images.metal-stack.io if necessary.

When building Docker images, please consider our build tool docker-make or the specific docker-make action respectively.

APIs

We are currently making use of Swagger when we exposing traditional REST APIs for end-users. This helps us with being technology-agnostic as we can generate clients in almost any language using go-swagger. Swagger additionally simplifies the documentation of our APIs.

Most APIs though are not required to be user-facing but are of technical nature. These are preferred to be implemented using grpc.

Versioning

Artifacts are versioned by tagging the respective repository with a tag starting with the letter v. After the letter, there stands a valid semantic version.

Documentation

In order to make it easier for others to understand a project, we document general information and usage instructions in a README.md in any project.

In addition to that, we document a microservice in the docs repository. The documentation should contain the reasoning why this service exists and why it was being implemented the way it was being implemented. The aim of this procedure is to reduce the time for contributors to comprehend architectural decisions that were made during the process of writing the software and to clarify the general purpose of this service in the entire context of the software.

Guidelines

This chapter describes general guidelines on how to develop and contribute code for a certain programming language.

Golang

Development follows the official guide to:

Development Decisions

  • Dependency Management by using Go modules
  • Build and Test Automation by using GNU Make.
  • End-user APIs should consider using go-swagger and Go-Restful Technical APIs should consider using grpc

Libraries

metal-stack maintains several libraries that you should utilize in your project in order unify common behavior. Some of these projects are:

Error Handling with Generated Swagger Clients

From the server-side you should ensure that you are returning the common error json struct in case of an error as defined in the metal-lib/httperrors. Ensure you are using go-restful >= v2.9.1 and go-restful-openapi >= v0.13.1 (allows default responses with error codes other than 200).

Documentation

We want to share knowledge and keep things simple. If things cannot kept simple we want enable everybody to understand them by:

  • Document in short sentences[4].
  • Do not explain the HOW (this is already documented by your code and documenting the obvious is considered a defect).
  • Explain the WHY. Add a "to" in your documentation line to force yourself to explain the reasonning (e.g. "<THE WHAT> to <THE TO>").

Python

Development follows the official guide to:

  • Style Guide for Python Code (PEP 8)[5]
    • The use of an IDE like PyCharm helps to write compliant code easily
  • Consider setuptools for packaging
  • If you want to add a Python microservice to the mix, consider pyinstaller on Alpine to achieve small image sizes
+Contributing · metal-stack
Edit on GitHub

Contributing

This document describes the way we want to contribute code to the projects of metal-stack, which are hosted on github.com/metal-stack.

The document is meant to be understood as a general guideline for contributions, but not as burden to be placed on a developer. Use your best judgment when contributing code. Try to be as clean and precise as possible when writing code and try to make your code as maintainable and understandable as possible for other people.

Even if it should go without saying, we live an open culture of discussion, in which everybody is welcome to participate. We treat every contribution with respect and objectiveness with the general aim to write software of quality.

If you want, feel free to propose changes to this document in a pull request.

How Can I Contribute?

Open a Github issue in the project you would like to contribute. Within the issue, your idea can be discussed. It is also possible to directly create a pull request when the set of changes is relatively small.

Pull Requests

The process described here has several goals:

  • Maintain quality
  • Enable a sustainable system to review contributions
  • Enable documented and reproducible addition of contributions
  1. Create a meaningful issue describing the WHY? of your contribution
  2. Create a repository fork within the context of that issue.
  3. Create a Draft Pull Request to the master branch of the target repository.
  4. Develop, document and test your contribution (try not to solve more than one issue in a single pull request)
  5. Ask for merging your contribution by removing the draft marker
  6. If code owners are defined, try to assign the request to a code owner

General Objectives

This section contains language-agnostic topics that all metal-stack projects are trying to follow.

Code Ownership

The code base is owned by the entire team and every member is allowed to contribute changes to any of the projects. This is considered as collective code ownership[1].

As a matter of fact, there are persons in a project, which already have experience with the sources. These are defined directly in the repository's CODEOWNERS file. If you want to merge changes into the master branch, it is advisable to include code owners into the proecess of discussion and merging.

Microservices

One major ambition of metal-stack is to follow the idea of microservices. This way, we want to achieve that we can

  • adapt to changes faster than with monolithic architectures,
  • be free of restrictions due to certain choices of technology,
  • leverage powerful traits of cloud infrastructures (e.g. high-scalability, high-availability, ...).

Programming Languages

We are generally open to write code in any language that fits best to the function of the software. However, we encourage golang to be the main language of metal-stack as we think that it makes development faster when not establishing too many different languages in our architecture. Reason for this is that we are striving for consistent behavior of the microservices, similar to what has been described for the Twelve-Factor App (see https://12factor.net/). We help enforcing unified behavior by allowing a small layer of shared code for every programming language. We will refer to this shared code as "libraries" for the rest of this document.

Artifacts

Artifacts are always produced by a CI process (Github Actions).

Docker images are published on the Github Container Registry of the metal-stack organization.

Binary artifacts or OS images can be uploaded to images.metal-stack.io if necessary.

When building Docker images, please consider our build tool docker-make or the specific docker-make action respectively.

APIs

We are currently making use of Swagger when we exposing traditional REST APIs for end-users. This helps us with being technology-agnostic as we can generate clients in almost any language using go-swagger. Swagger additionally simplifies the documentation of our APIs.

Most APIs though are not required to be user-facing but are of technical nature. These are preferred to be implemented using grpc.

Versioning

Artifacts are versioned by tagging the respective repository with a tag starting with the letter v. After the letter, there stands a valid semantic version.

Documentation

In order to make it easier for others to understand a project, we document general information and usage instructions in a README.md in any project.

In addition to that, we document a microservice in the docs repository. The documentation should contain the reasoning why this service exists and why it was being implemented the way it was being implemented. The aim of this procedure is to reduce the time for contributors to comprehend architectural decisions that were made during the process of writing the software and to clarify the general purpose of this service in the entire context of the software.

Guidelines

This chapter describes general guidelines on how to develop and contribute code for a certain programming language.

Golang

Development follows the official guide to:

Development Decisions

  • Dependency Management by using Go modules
  • Build and Test Automation by using GNU Make.
  • End-user APIs should consider using go-swagger and Go-Restful Technical APIs should consider using grpc

Libraries

metal-stack maintains several libraries that you should utilize in your project in order unify common behavior. Some of these projects are:

Error Handling with Generated Swagger Clients

From the server-side you should ensure that you are returning the common error json struct in case of an error as defined in the metal-lib/httperrors. Ensure you are using go-restful >= v2.9.1 and go-restful-openapi >= v0.13.1 (allows default responses with error codes other than 200).

Documentation

We want to share knowledge and keep things simple. If things cannot kept simple we want enable everybody to understand them by:

  • Document in short sentences[4].
  • Do not explain the HOW (this is already documented by your code and documenting the obvious is considered a defect).
  • Explain the WHY. Add a "to" in your documentation line to force yourself to explain the reasonning (e.g. "<THE WHAT> to <THE TO>").

Python

Development follows the official guide to:

  • Style Guide for Python Code (PEP 8)[5]
    • The use of an IDE like PyCharm helps to write compliant code easily
  • Consider setuptools for packaging
  • If you want to add a Python microservice to the mix, consider pyinstaller on Alpine to achieve small image sizes
diff --git a/dev/development/proposals/MEP1/README/index.html b/dev/development/proposals/MEP1/README/index.html index cce0168ac1..c5c641675a 100644 --- a/dev/development/proposals/MEP1/README/index.html +++ b/dev/development/proposals/MEP1/README/index.html @@ -1,2 +1,2 @@ -Distributed Metal Control Plane · metal-stack
Edit on GitHub

Distributed Metal Control Plane

Problem Statement

We face the situation that we argue for running bare metal on premise because this way the customers can control where and how their software and data are processed and stored. On the other hand, we have currently decided that our metal-api control plane components run on a kubernetes cluster (in our case on a cluster provided by one of the available hyperscalers).

Running the control plane on Kubernetes has the following benefits:

  • Ease of deployment
  • Get most, if not all, of the required infrastructure services like (probably incomplete):
    • IPs
    • DNS
    • L7-Loadbalancing
    • Storage
    • S3 Backup
    • High Availability

Using a kubernetes as a service offering from one of the hyperscalers, enables us to focus on using kubernetes instead of maintaining it as well.

Goal

It would be much saner if metal-stack has no, or only minimal dependencies to external services. Imagine a metal-stack deployment in a plant, it would be optimal if we only have to deliver a single rack with servers and networking gear installed and wired, plug that rack to the power supply and a internet uplink and its ready to go.

Have a second plant which you want to be part of all your plants? Just tell both that they are part of something bigger and metal-api knows of two partitions.

Possible Solutions

We can think of two different solutions to this vision:

  1. Keep the central control plane approach and require some sort of kubernetes deployment accessible from the internet. This has the downside that the user must, provide a managed kubernetes deployment in his own datacenter or uses a hyperscaler. Still not optimal.
  2. Install the metal-api and all its dependencies in every partition, replicate or shard the databases to every connected partition, make them know each other. Connect the partitions over the internet with some sort of vpn to make the services visible to each other.

As we can see, the first approach does not really address the problem, therefore i will describe solution #2 in more details.

Central/Current setup

Stateful services

Every distributed system suffer from handling state in a scalable, fast and correct way. To start how to cope with the state, we first must identify which state can be seen as partition local only and which state must be synchronous for read, and synchronous for writes across partitions.

Affected states:

  • masterdata: e.g. tenant and project must be present in every partition, but these are entities which are read often but updates are rare. A write can therefore be visible with a decent delay in a distinct partition with no consequences.
  • ipam: the prefixes and ip´s allocated from machines. These entities are also read often and rare updates. But we must differentiate between dirty reads for different types. A machine network is partition local, ips acquired from such a network must by synchronous in the same partition. Ips acquired from global networks such as internet must by synchronous for all partitions, as otherwise a internet ip could be acquired twice.
  • vrf ids: they must only be unique in one partition
  • image and size configurations: read often, written seldom, so no high requirements on the storage of these entities.
  • images: os images are already replicated from a central s3 storage to a per partition s3 service. metal-hammer kernel and initrd are small and pull always from the central s3, can be done similar to os images.
  • machine and machine allocation: must be only synchronous in the partition
  • switch: must be only synchronous in the partition
  • nsq messages: do not need to cross partition boundaries. No need to keep the messages persistent, even the opposite is true, we don't want to have the messages persist for a longer period.

Now we can see that the most critical state to held and synchronize are the IPAM data, because these entities must be guaranteed to be synchronously updated, while being updated frequently.

Datastores:

We use three different types of datastores to persist the states of the metal application.

  • rethinkdb is the main datastore for almost all entities managed by metal-api
  • postgresql is used for masterdata and ipam data.
  • nsq uses disk and memory tho store the messages.

Stateless services

These are the easy part, all of our services which are stateless can be scaled up and down without any impact on functionality. Even the stateful services like masterdata and metal-api rely fully on the underlying datastore and can therefore also be scaled up and down to meet scalability requirements.

Albeit, most of these services need to be placed behind a loadbalancer which does the L4/L7 balancing across the started/available replicas of the service for the clients talking to it. This is actually provided by kubernetes with either service type loadbalancer or type clusterip.

One exception is the metal-console service which must have the partition in it´s dns name now, because there is no direct network connectivity between the management networks of the partitions. See "Network Setup)

Distributed setup

State

In order to replicate certain data which must be available across all partitions we can use on of the existing open source databases which enable such kind of setup. There are a few avaible out there, the following uncomplete list will highlight the pro´s and cons of each.

  • RethinkDB

    We already store most of our data in RethinkDB and it gives already the ability to synchronize the data in a distributed manner with different guarantees for consistency and latency. This is described here: Scaling, Sharding and replication. But because rethinkdb has a rough history and unsure future with the last release took more than a year, we in the team already thought that we eventually must move away from rethinkdb in the future.

  • Postgresql

    Postgres does not have a multi datacenter with replication in both directions, it just can make the remote instance store the same data.

  • CockroachDB

    Is a Postgresql compatible database enginge on the wire. CockroachDB gives you both, ACID and geo replication with writes allowed from all connected members. It is even possible to configure Follow the Workload and Geo Partitioning and Replication.

If we migrate all metal-api entities to be stored the same way we store masterdata, we could use cockroachdb to store all metal entities in one ore more databases spread across all partitions and still ensure consistency and high availability.

A simple setup how this would look like is shown here.

Simple CockroachDB setup

go-ipam was modified in a example PR here: https://github.com/metal-pod/go-ipam/pull/17

API Access

In order to make the metal-api accessible for api users like cloud-api or metalctl as easy at it is today, some effort has to be taken. One possible approach would be to use a external loadbalancer which spread the requests evenly to all metal-api endpoints in all partitions. Because all data are accessible from all partitions, a api request going to partition A with a request to create a machine in partition B, will still work. If on the other hand partition B is not in a connected state because the interconnection between both partitions is broken, then of course the request will fail.

IMPORTANT The NSQ Message to inform metal-core must end in the correct partition

To provide such a external loadbalancer we have several opportunities:

  • Cloudflare or comparable CDN service.
  • BGP Anycast from every partition

Another setup would place a small gateway behind the metal-api address, which forwards to the metal-api in the partition where the request must be executed. This gateway, metal-api-router must inspect the payload, extract the desired partition, and forward the request without any modifications to the metal-api endpoint in this partition. This can be done for all requests, or if we want to optimize, only for write accesses.

Network setup

In order to have the impact to the overall security concept as minimal as possible i would not modify the current network setup. The only modifications which has to be made are:

  • Allow https ingress traffic to all metal-api instances.
  • Allow ssh ingress traffic to all metal-console instances.
  • Allow CockroachDB Replication between all partitions.
  • No NSQ traffic from outside required anymore, except we cant solve the topic above.

A simple setup how this would look like is shown here, this does not work though because of the forementioned NSQ issue.

API and Console Access

Therefore we need the metal-api-router:

Working API and Console Access

Deployment

The deployment of our components will substantially differ in a partition compared to a the deployment we have actually. Deploying it in kubernetes in the partition would be very difficult to achieve because we have no sane way to deploy kubernetes on physical machines without a underlying API. I would therefore suggest to deploy our components in the same way we do that for the services running on the management server. Use systemd to start docker containers.

Deployment

+Distributed Metal Control Plane · metal-stack
Edit on GitHub

Distributed Metal Control Plane

Problem Statement

We face the situation that we argue for running bare metal on premise because this way the customers can control where and how their software and data are processed and stored. On the other hand, we have currently decided that our metal-api control plane components run on a kubernetes cluster (in our case on a cluster provided by one of the available hyperscalers).

Running the control plane on Kubernetes has the following benefits:

  • Ease of deployment
  • Get most, if not all, of the required infrastructure services like (probably incomplete):
    • IPs
    • DNS
    • L7-Loadbalancing
    • Storage
    • S3 Backup
    • High Availability

Using a kubernetes as a service offering from one of the hyperscalers, enables us to focus on using kubernetes instead of maintaining it as well.

Goal

It would be much saner if metal-stack has no, or only minimal dependencies to external services. Imagine a metal-stack deployment in a plant, it would be optimal if we only have to deliver a single rack with servers and networking gear installed and wired, plug that rack to the power supply and a internet uplink and its ready to go.

Have a second plant which you want to be part of all your plants? Just tell both that they are part of something bigger and metal-api knows of two partitions.

Possible Solutions

We can think of two different solutions to this vision:

  1. Keep the central control plane approach and require some sort of kubernetes deployment accessible from the internet. This has the downside that the user must, provide a managed kubernetes deployment in his own datacenter or uses a hyperscaler. Still not optimal.
  2. Install the metal-api and all its dependencies in every partition, replicate or shard the databases to every connected partition, make them know each other. Connect the partitions over the internet with some sort of vpn to make the services visible to each other.

As we can see, the first approach does not really address the problem, therefore i will describe solution #2 in more details.

Central/Current setup

Stateful services

Every distributed system suffer from handling state in a scalable, fast and correct way. To start how to cope with the state, we first must identify which state can be seen as partition local only and which state must be synchronous for read, and synchronous for writes across partitions.

Affected states:

  • masterdata: e.g. tenant and project must be present in every partition, but these are entities which are read often but updates are rare. A write can therefore be visible with a decent delay in a distinct partition with no consequences.
  • ipam: the prefixes and ip´s allocated from machines. These entities are also read often and rare updates. But we must differentiate between dirty reads for different types. A machine network is partition local, ips acquired from such a network must by synchronous in the same partition. Ips acquired from global networks such as internet must by synchronous for all partitions, as otherwise a internet ip could be acquired twice.
  • vrf ids: they must only be unique in one partition
  • image and size configurations: read often, written seldom, so no high requirements on the storage of these entities.
  • images: os images are already replicated from a central s3 storage to a per partition s3 service. metal-hammer kernel and initrd are small and pull always from the central s3, can be done similar to os images.
  • machine and machine allocation: must be only synchronous in the partition
  • switch: must be only synchronous in the partition
  • nsq messages: do not need to cross partition boundaries. No need to keep the messages persistent, even the opposite is true, we don't want to have the messages persist for a longer period.

Now we can see that the most critical state to held and synchronize are the IPAM data, because these entities must be guaranteed to be synchronously updated, while being updated frequently.

Datastores:

We use three different types of datastores to persist the states of the metal application.

  • rethinkdb is the main datastore for almost all entities managed by metal-api
  • postgresql is used for masterdata and ipam data.
  • nsq uses disk and memory tho store the messages.

Stateless services

These are the easy part, all of our services which are stateless can be scaled up and down without any impact on functionality. Even the stateful services like masterdata and metal-api rely fully on the underlying datastore and can therefore also be scaled up and down to meet scalability requirements.

Albeit, most of these services need to be placed behind a loadbalancer which does the L4/L7 balancing across the started/available replicas of the service for the clients talking to it. This is actually provided by kubernetes with either service type loadbalancer or type clusterip.

One exception is the metal-console service which must have the partition in it´s dns name now, because there is no direct network connectivity between the management networks of the partitions. See "Network Setup)

Distributed setup

State

In order to replicate certain data which must be available across all partitions we can use on of the existing open source databases which enable such kind of setup. There are a few avaible out there, the following uncomplete list will highlight the pro´s and cons of each.

  • RethinkDB

    We already store most of our data in RethinkDB and it gives already the ability to synchronize the data in a distributed manner with different guarantees for consistency and latency. This is described here: Scaling, Sharding and replication. But because rethinkdb has a rough history and unsure future with the last release took more than a year, we in the team already thought that we eventually must move away from rethinkdb in the future.

  • Postgresql

    Postgres does not have a multi datacenter with replication in both directions, it just can make the remote instance store the same data.

  • CockroachDB

    Is a Postgresql compatible database enginge on the wire. CockroachDB gives you both, ACID and geo replication with writes allowed from all connected members. It is even possible to configure Follow the Workload and Geo Partitioning and Replication.

If we migrate all metal-api entities to be stored the same way we store masterdata, we could use cockroachdb to store all metal entities in one ore more databases spread across all partitions and still ensure consistency and high availability.

A simple setup how this would look like is shown here.

Simple CockroachDB setup

go-ipam was modified in a example PR here: https://github.com/metal-pod/go-ipam/pull/17

API Access

In order to make the metal-api accessible for api users like cloud-api or metalctl as easy at it is today, some effort has to be taken. One possible approach would be to use a external loadbalancer which spread the requests evenly to all metal-api endpoints in all partitions. Because all data are accessible from all partitions, a api request going to partition A with a request to create a machine in partition B, will still work. If on the other hand partition B is not in a connected state because the interconnection between both partitions is broken, then of course the request will fail.

IMPORTANT The NSQ Message to inform metal-core must end in the correct partition

To provide such a external loadbalancer we have several opportunities:

  • Cloudflare or comparable CDN service.
  • BGP Anycast from every partition

Another setup would place a small gateway behind the metal-api address, which forwards to the metal-api in the partition where the request must be executed. This gateway, metal-api-router must inspect the payload, extract the desired partition, and forward the request without any modifications to the metal-api endpoint in this partition. This can be done for all requests, or if we want to optimize, only for write accesses.

Network setup

In order to have the impact to the overall security concept as minimal as possible i would not modify the current network setup. The only modifications which has to be made are:

  • Allow https ingress traffic to all metal-api instances.
  • Allow ssh ingress traffic to all metal-console instances.
  • Allow CockroachDB Replication between all partitions.
  • No NSQ traffic from outside required anymore, except we cant solve the topic above.

A simple setup how this would look like is shown here, this does not work though because of the forementioned NSQ issue.

API and Console Access

Therefore we need the metal-api-router:

Working API and Console Access

Deployment

The deployment of our components will substantially differ in a partition compared to a the deployment we have actually. Deploying it in kubernetes in the partition would be very difficult to achieve because we have no sane way to deploy kubernetes on physical machines without a underlying API. I would therefore suggest to deploy our components in the same way we do that for the services running on the management server. Use systemd to start docker containers.

Deployment

diff --git a/dev/development/proposals/MEP11/README/index.html b/dev/development/proposals/MEP11/README/index.html index f1af19dda0..1b36e4e04c 100644 --- a/dev/development/proposals/MEP11/README/index.html +++ b/dev/development/proposals/MEP11/README/index.html @@ -1,2 +1,2 @@ -Auditing of metal-stack resources · metal-stack
Edit on GitHub

Auditing of metal-stack resources

Currently no logs of the ownership of resources like machines, networks, ips and volumes are generated or kept. Though due to legal requirements data centers are required to keep track of this ownership over time to prevent liability issues when opening the platform for external users.

In this proposal we want to introduce a flexible and low-maintenance approach for auditing on top of Meilisearch.

Overview

In general our auditing logs will be collected by a request interceptor or middleware. Every request and response will be processed and eventually logged to Meilisearch. Meilisearch will be configured to regularly create chunks of the auditing logs. These finished chunks will be backed up to a S3 compatible storage with a read-only option enabled.

Of course sensitve data like session keys or passwords will be redacted before logging. We want to track relevant requests and responses. If auditing the request fails, the request itself will be aborted and will not be processed further. The requests and responses that will be audited will be annotated with a correlation id.

Transferring the meilisearch auditing data chunks to the S3 compatible storage will be done by a sidecar cronjob that is executed periodically. To avoid data manipulation the S3 compatible storage will be configured to be read-only.

Whitelisting

To reduce the amount of unnecessary logs we want to introduce a whitelist of resources and operations on those that should be logged. Other requests will be passed directly to the next middleware or web service without any further processing.

As we are only interested in mutating endpoints, we ignore all GET requests. The whitelist includes all POST, PUT, PATCH and DELETE endpoints of the HTTP middleware except for the following (non-manipulating) route suffixes:

  • /find
  • /notify
  • /try and /match
  • /capacity
  • /from-hardware

Regarding GRPC audit trails, they are not so interesting because only internal clients are using this API. However, we can log the trails of the Boot service, which can be interesting to revise the machine lifecycle.

Chunking in Meilisearch

We want our data to be chunked in Meilisearch. To accomplish this, we rotate the index identifier on a scheduled basis. The index identifiers will be derived from the current date and time.

To keep things simple, we only support hourly, daily and monthly rotation. The eventually prefixed index names will only include relevant parts of date and time like 2021-01, 2021-01-01 or 2021-01-01_13.

The metal-api will only write to the current index and switches to the new index on rotation. The metal-api will never read or update data in any indices.

Moving chunks to S3 compatible storage

As Meilisearch will be filled with data over time, we want to move completed chunks to a S3 compatible storage. This will be done by a sidecar cronjob that is executed periodically. Note that the periods of the index rotation and the cronjob execution don't have to match.

When the backup process gets started, it initiates a Meilisearch dump of the whole database across all indices. Once the returned task is finished, the dump must be copied from a Meilisearch volume to the S3 compatible storage. After a successful copy, the dump can be deleted.

Now we want to remove all indices from Meilisearch, except the most recent one. For this, we get all indices, sort them and delete each index except the most recent one to avoid data loss.

For the actual implementation, we can build upon backup-restore-sidecar. But due to the index rotation and the fact, that older indices need to be deleted, this probably does not fit into the mentioned sidecar.

S3 compatible storage

The dumps of chunks should automatically deleted after a certain amount of time, once we are either no longer allowed or required to keep them. The default retention time will be 6 months. Ideally already uploaded chunks should be read-only to prevent data manipulation.

A candidate for the S3 compatible storage is Google Cloud Storage, which allows to configure automatic expiration of objects through a lifecycle rule.

Affected components

  • metal-api grpc server needs an auditing interceptor
  • metal-api web server needs an auditing filter chain / middleware
  • metal-api needs new command line arguments to configure the auditing
  • mini-lab needs a Meilisearch instance
  • mini-lab may need a local S3 compatible storage
  • we need a sidecar to implement the backup to S3 compatible storage
  • Consider auditing of volume allocations and freeings outside of metal-stack

Alternatives considered

Instead of using Meilisearch we investigated using an immutable database like immudb. But immudb does not support chunking of data and due to its immutable nature, we will never be able to free up space of expired data. Even if we are legally allowed or required to delete data, we will not be able to do so with immudb.

In another variant of the Meilisearch approach the metal-api would also be responsible for copying chunks to the S3 compatible storage and deleting old indices. But separating the concerns allows completely different implementations for every deployment stage.

+Auditing of metal-stack resources · metal-stack
Edit on GitHub

Auditing of metal-stack resources

Currently no logs of the ownership of resources like machines, networks, ips and volumes are generated or kept. Though due to legal requirements data centers are required to keep track of this ownership over time to prevent liability issues when opening the platform for external users.

In this proposal we want to introduce a flexible and low-maintenance approach for auditing on top of Meilisearch.

Overview

In general our auditing logs will be collected by a request interceptor or middleware. Every request and response will be processed and eventually logged to Meilisearch. Meilisearch will be configured to regularly create chunks of the auditing logs. These finished chunks will be backed up to a S3 compatible storage with a read-only option enabled.

Of course sensitve data like session keys or passwords will be redacted before logging. We want to track relevant requests and responses. If auditing the request fails, the request itself will be aborted and will not be processed further. The requests and responses that will be audited will be annotated with a correlation id.

Transferring the meilisearch auditing data chunks to the S3 compatible storage will be done by a sidecar cronjob that is executed periodically. To avoid data manipulation the S3 compatible storage will be configured to be read-only.

Whitelisting

To reduce the amount of unnecessary logs we want to introduce a whitelist of resources and operations on those that should be logged. Other requests will be passed directly to the next middleware or web service without any further processing.

As we are only interested in mutating endpoints, we ignore all GET requests. The whitelist includes all POST, PUT, PATCH and DELETE endpoints of the HTTP middleware except for the following (non-manipulating) route suffixes:

  • /find
  • /notify
  • /try and /match
  • /capacity
  • /from-hardware

Regarding GRPC audit trails, they are not so interesting because only internal clients are using this API. However, we can log the trails of the Boot service, which can be interesting to revise the machine lifecycle.

Chunking in Meilisearch

We want our data to be chunked in Meilisearch. To accomplish this, we rotate the index identifier on a scheduled basis. The index identifiers will be derived from the current date and time.

To keep things simple, we only support hourly, daily and monthly rotation. The eventually prefixed index names will only include relevant parts of date and time like 2021-01, 2021-01-01 or 2021-01-01_13.

The metal-api will only write to the current index and switches to the new index on rotation. The metal-api will never read or update data in any indices.

Moving chunks to S3 compatible storage

As Meilisearch will be filled with data over time, we want to move completed chunks to a S3 compatible storage. This will be done by a sidecar cronjob that is executed periodically. Note that the periods of the index rotation and the cronjob execution don't have to match.

When the backup process gets started, it initiates a Meilisearch dump of the whole database across all indices. Once the returned task is finished, the dump must be copied from a Meilisearch volume to the S3 compatible storage. After a successful copy, the dump can be deleted.

Now we want to remove all indices from Meilisearch, except the most recent one. For this, we get all indices, sort them and delete each index except the most recent one to avoid data loss.

For the actual implementation, we can build upon backup-restore-sidecar. But due to the index rotation and the fact, that older indices need to be deleted, this probably does not fit into the mentioned sidecar.

S3 compatible storage

The dumps of chunks should automatically deleted after a certain amount of time, once we are either no longer allowed or required to keep them. The default retention time will be 6 months. Ideally already uploaded chunks should be read-only to prevent data manipulation.

A candidate for the S3 compatible storage is Google Cloud Storage, which allows to configure automatic expiration of objects through a lifecycle rule.

Affected components

  • metal-api grpc server needs an auditing interceptor
  • metal-api web server needs an auditing filter chain / middleware
  • metal-api needs new command line arguments to configure the auditing
  • mini-lab needs a Meilisearch instance
  • mini-lab may need a local S3 compatible storage
  • we need a sidecar to implement the backup to S3 compatible storage
  • Consider auditing of volume allocations and freeings outside of metal-stack

Alternatives considered

Instead of using Meilisearch we investigated using an immutable database like immudb. But immudb does not support chunking of data and due to its immutable nature, we will never be able to free up space of expired data. Even if we are legally allowed or required to delete data, we will not be able to do so with immudb.

In another variant of the Meilisearch approach the metal-api would also be responsible for copying chunks to the S3 compatible storage and deleting old indices. But separating the concerns allows completely different implementations for every deployment stage.

diff --git a/dev/development/proposals/MEP12/README/index.html b/dev/development/proposals/MEP12/README/index.html index a96abd9d59..82db7db8eb 100644 --- a/dev/development/proposals/MEP12/README/index.html +++ b/dev/development/proposals/MEP12/README/index.html @@ -4,4 +4,4 @@ type MachineAllocation struct { // existing fields are omitted for readability PlacementTags []string `json:"tags" description:"by default machines are spread across the racks inside a partition for every project. if placement tags are provided, the machine candidate has an additional anti-affinity to other machines having the same tags"` -} +} diff --git a/dev/development/proposals/MEP12/partitioning/index.html b/dev/development/proposals/MEP12/partitioning/index.html index a60b87b9c0..b60f41ab9e 100644 --- a/dev/development/proposals/MEP12/partitioning/index.html +++ b/dev/development/proposals/MEP12/partitioning/index.html @@ -1,2 +1,2 @@ -Multi-Partition-Layout · metal-stack
Edit on GitHub

marp: true theme: metal-stack paginate: true footer: Gerrit Schwerthelm – x-cellent technologies GmbH — metal-stack Training backgroundImage: url("https://metal-stack.io/images/shape/banner.png") –- <!– _class: cover lead –>

h:200px

<!– _class: cover lead –>

Multi-Partition-Layout

<!– _class: lead _backgroundColor: #1f1f1f _backgroundImage: _footer: "" –> bg contain

<!– _class: lead _backgroundColor: #1f1f1f _backgroundImage: _footer: "" –> bg contain

<style>section { font-size: 30px; }</style>

Multi-Partition-Layout Properties

  • Fully independent locations with own storage and own node networks
  • Clusters can only be created independent in every location
    • Failover mechanism for deployed applications requires duplicated deployments, which can serve indepedently
    • Failover through BGP
  • If cluster nodes are spread across partitions (not implemented yet), nodes will not be able to reach each other
    • Would require an overlay network for inter-node-communication

<!– _class: cover lead –>

Single-Partition-Layout

<!– _class: lead _backgroundColor: #1f1f1f _backgroundImage: _footer: "" –> bg contain

<style>section { font-size: 30px; }</style>

Single-Partition-Layout Properties

  • Multiple groups of racks at multiple locations but connected to same CLOS topology
  • All racks can connect to the same storage network
  • Nodes in private networks can communicate
  • When creating a cluster, nodes will be randomly spread across the racks
    • Possible improvement of this situation, see MEP-12: Rack Spreading

MEP-12: Rack Spreading

  • Instead of selecting a machine from a machine pool randomly
  • Get all existing machines in the same project and count to which rack they belong
  • Place machine on the rack with the least amount of machines already allocated
  • Best effort only
+Multi-Partition-Layout · metal-stack
Edit on GitHub

marp: true theme: metal-stack paginate: true footer: Gerrit Schwerthelm – x-cellent technologies GmbH — metal-stack Training backgroundImage: url("https://metal-stack.io/images/shape/banner.png") –- <!– _class: cover lead –>

h:200px

<!– _class: cover lead –>

Multi-Partition-Layout

<!– _class: lead _backgroundColor: #1f1f1f _backgroundImage: _footer: "" –> bg contain

<!– _class: lead _backgroundColor: #1f1f1f _backgroundImage: _footer: "" –> bg contain

<style>section { font-size: 30px; }</style>

Multi-Partition-Layout Properties

  • Fully independent locations with own storage and own node networks
  • Clusters can only be created independent in every location
    • Failover mechanism for deployed applications requires duplicated deployments, which can serve indepedently
    • Failover through BGP
  • If cluster nodes are spread across partitions (not implemented yet), nodes will not be able to reach each other
    • Would require an overlay network for inter-node-communication

<!– _class: cover lead –>

Single-Partition-Layout

<!– _class: lead _backgroundColor: #1f1f1f _backgroundImage: _footer: "" –> bg contain

<style>section { font-size: 30px; }</style>

Single-Partition-Layout Properties

  • Multiple groups of racks at multiple locations but connected to same CLOS topology
  • All racks can connect to the same storage network
  • Nodes in private networks can communicate
  • When creating a cluster, nodes will be randomly spread across the racks
    • Possible improvement of this situation, see MEP-12: Rack Spreading

MEP-12: Rack Spreading

  • Instead of selecting a machine from a machine pool randomly
  • Get all existing machines in the same project and count to which rack they belong
  • Place machine on the rack with the least amount of machines already allocated
  • Best effort only
diff --git a/dev/development/proposals/MEP2/README/index.html b/dev/development/proposals/MEP2/README/index.html index 4d61d9c2a6..0cf02b0784 100644 --- a/dev/development/proposals/MEP2/README/index.html +++ b/dev/development/proposals/MEP2/README/index.html @@ -1,2 +1,2 @@ -Two Factor Authentication · metal-stack
Edit on GitHub
+Two Factor Authentication · metal-stack
Edit on GitHub
diff --git a/dev/development/proposals/MEP3/README/index.html b/dev/development/proposals/MEP3/README/index.html index 85cb786632..1acac90003 100644 --- a/dev/development/proposals/MEP3/README/index.html +++ b/dev/development/proposals/MEP3/README/index.html @@ -1,2 +1,2 @@ -Machine Re-Installation · metal-stack
Edit on GitHub

Machine Re-Installation

In the current metal-api only machine installations are possible, performing a machine upgrade is only possible by creating a new machine and delete the old one. This has the drawback that in case a lot of data is stored on the local disks, a full restore of the original data must be performed.

To prevent this, we will introduce a new metal-api endpoint to reinstall the machine with a new image, without actually deleting the data stored on the additional hard disks.

Storage is a difficult task to get right and reliable. A short analysis of our different storage requirements lead to 3 different scenarios.

  • Storage for the etcd pvs in the seed cluster of every partition. This is the most important storage in our setup because these etcd pods serve as configuration backend for all customer kubernetes clusters. If they fail, the cluster is down. However gardener deploys a backup and restore sidecar into the etcd pod of every customer kubernetes control plane, and if this sidecar detects a corrupt or missing etcd database file(s) it starts automatic restore from the configured backup location. This will take some minutes. If for example a node dies, and gardener creates a new node instead, the csi-lvm created pv is not present on that node. Kubernetes will not schedule the missing etcd pod on this node because it has a local PV configured and is therefore tainted to run only on that node. To let kubernetes create that pod anyhow, someone has to either remove the taint, or delete the pod. If this is done, the pod starts and the restore of the etcd data can start as well. You can see this is a bit too complicated and will take the customer cluster down for a while (not measured yet but in the range of 5-10 minutes).
  • Storage in customer clusters. This was not promised in 2020. We have a intermediate solution with the provisioning of csi-lvm by default into all customer clusters. Albeit this is only local storage and will get deleted if a node dies.
  • S3 Storage. We have two possibilities to cope with storage:
    • In place update of the OS with a daemonset This will be fast and simple, but might fail because the packages being installed are broken right now, or a filesystem gets full, or any other failure you can think of during a os update. Another drawback is that metal-api does not reflect the updated os image.
    • metal-api get a machine reinstall endpoint With this approach we leverage from existing and already proven mechanisms. Reinstall must keep all data except the sata-dom. Gardener currently is not able to do an update with this approach because it can only do rolling updates. Therefore a additional osupdatestrategy has to be implemented for metal and other providers in gardener to be able to leverage the metal reinstall on the same machineID approach.

If reinstall is implemented, we should focus on the same technology for all scenarios and put ceph via rook.io into the kubernetes clusters as additional StorageClass. It has to be checked whether to use the raw disk or a PV as the underlay block device where ceph stores its data.

API and behavior

The API will get an new endpoint "reinstall" this endpoint takes two arguments:

  • machineID
  • image

No other aspects of the machine can be modified during the re-installation. All data stored in the existing allocation will be preserved, only the image will be modified. Once this endpoint was called, the machine will get a reboot signal with the boot order set to PXE instead of HDD and the network interfaces on the leaf are set to PXE as well. Then the normal installation process starts:

  • unchanged: PXE boot with metal-hammer
  • changed: metal-hammer first checks with the machineID in the metal-api (through metal-core) if there is already a allocation present
  • changed: if a allocation is present and the allocation has set reinstall: true, wipe disk is only executed for the root disk, all other disks are untouched.
  • unchanged: the specified image is downloaded and burned, /install.sh is executed
  • unchanged: successful installation is reported back, network is set the the vrf, boot order is set to HDD.
  • unchanged: distribution kernel is booted via kexec

We can see that the allocation requires one additional parameter: reinstall and metal-hammer must check for already existing allocation at an earlier stage.

Components which requires modifications (first guess):

  • metal-hammer:
    • check for allocation present earlier
    • evaluation of reinstall flag set
    • wipe of disks depends on that flag
    • Bonus: move configuration of disk layout and primary disk detection algorithm (PDDA) from metal-hammer into metal-api. metal-api MUST reject reinstallation if the disk found by PDDA does not have the /etc/metal directory!
  • metal-core:
    • probably nothing
  • metal-api:
    • new endpoint /machine/reinstall
    • add Reinstall bool to data model of allocation
    • make sure to reset Reinstall after reinstallation to prevent endless reinstallation loop
  • metalctl:
    • implement reinstall
  • metal-go:
    • implement reinstall
  • gardener (longterm):
    • add the OSUpgradeStrategy reinstall
+Machine Re-Installation · metal-stack
Edit on GitHub

Machine Re-Installation

In the current metal-api only machine installations are possible, performing a machine upgrade is only possible by creating a new machine and delete the old one. This has the drawback that in case a lot of data is stored on the local disks, a full restore of the original data must be performed.

To prevent this, we will introduce a new metal-api endpoint to reinstall the machine with a new image, without actually deleting the data stored on the additional hard disks.

Storage is a difficult task to get right and reliable. A short analysis of our different storage requirements lead to 3 different scenarios.

  • Storage for the etcd pvs in the seed cluster of every partition. This is the most important storage in our setup because these etcd pods serve as configuration backend for all customer kubernetes clusters. If they fail, the cluster is down. However gardener deploys a backup and restore sidecar into the etcd pod of every customer kubernetes control plane, and if this sidecar detects a corrupt or missing etcd database file(s) it starts automatic restore from the configured backup location. This will take some minutes. If for example a node dies, and gardener creates a new node instead, the csi-lvm created pv is not present on that node. Kubernetes will not schedule the missing etcd pod on this node because it has a local PV configured and is therefore tainted to run only on that node. To let kubernetes create that pod anyhow, someone has to either remove the taint, or delete the pod. If this is done, the pod starts and the restore of the etcd data can start as well. You can see this is a bit too complicated and will take the customer cluster down for a while (not measured yet but in the range of 5-10 minutes).
  • Storage in customer clusters. This was not promised in 2020. We have a intermediate solution with the provisioning of csi-lvm by default into all customer clusters. Albeit this is only local storage and will get deleted if a node dies.
  • S3 Storage. We have two possibilities to cope with storage:
    • In place update of the OS with a daemonset This will be fast and simple, but might fail because the packages being installed are broken right now, or a filesystem gets full, or any other failure you can think of during a os update. Another drawback is that metal-api does not reflect the updated os image.
    • metal-api get a machine reinstall endpoint With this approach we leverage from existing and already proven mechanisms. Reinstall must keep all data except the sata-dom. Gardener currently is not able to do an update with this approach because it can only do rolling updates. Therefore a additional osupdatestrategy has to be implemented for metal and other providers in gardener to be able to leverage the metal reinstall on the same machineID approach.

If reinstall is implemented, we should focus on the same technology for all scenarios and put ceph via rook.io into the kubernetes clusters as additional StorageClass. It has to be checked whether to use the raw disk or a PV as the underlay block device where ceph stores its data.

API and behavior

The API will get an new endpoint "reinstall" this endpoint takes two arguments:

  • machineID
  • image

No other aspects of the machine can be modified during the re-installation. All data stored in the existing allocation will be preserved, only the image will be modified. Once this endpoint was called, the machine will get a reboot signal with the boot order set to PXE instead of HDD and the network interfaces on the leaf are set to PXE as well. Then the normal installation process starts:

  • unchanged: PXE boot with metal-hammer
  • changed: metal-hammer first checks with the machineID in the metal-api (through metal-core) if there is already a allocation present
  • changed: if a allocation is present and the allocation has set reinstall: true, wipe disk is only executed for the root disk, all other disks are untouched.
  • unchanged: the specified image is downloaded and burned, /install.sh is executed
  • unchanged: successful installation is reported back, network is set the the vrf, boot order is set to HDD.
  • unchanged: distribution kernel is booted via kexec

We can see that the allocation requires one additional parameter: reinstall and metal-hammer must check for already existing allocation at an earlier stage.

Components which requires modifications (first guess):

  • metal-hammer:
    • check for allocation present earlier
    • evaluation of reinstall flag set
    • wipe of disks depends on that flag
    • Bonus: move configuration of disk layout and primary disk detection algorithm (PDDA) from metal-hammer into metal-api. metal-api MUST reject reinstallation if the disk found by PDDA does not have the /etc/metal directory!
  • metal-core:
    • probably nothing
  • metal-api:
    • new endpoint /machine/reinstall
    • add Reinstall bool to data model of allocation
    • make sure to reset Reinstall after reinstallation to prevent endless reinstallation loop
  • metalctl:
    • implement reinstall
  • metal-go:
    • implement reinstall
  • gardener (longterm):
    • add the OSUpgradeStrategy reinstall
diff --git a/dev/development/proposals/MEP4/README/index.html b/dev/development/proposals/MEP4/README/index.html index 2601e7b502..57c7eb015a 100644 --- a/dev/development/proposals/MEP4/README/index.html +++ b/dev/development/proposals/MEP4/README/index.html @@ -1,2 +1,2 @@ -Multi-tenancy for the metal-api · metal-stack
Edit on GitHub

Multi-tenancy for the metal-api

In the past we decided to treat the metal-api as a "low-level API", i.e. the API does not know anything about projects and tenants. A user with editor access can for example assign machines to every project he desires, he can see all the machines available and control them. Even though we always wanted to keep open the possibility to just offer bare metal machines to the end-user, the ultimate objective has always been to create an API for Kubernetes clusters. Hence, we tried to keep the metal-api code base as small as possible and we added resource scoping to a "higher-level API", the cloud-api, a component that is not open-source. From there, a user would be able to only see his own clusters and IP addresses.

The implication is that the metal-api has no multi-tenancy without another layer on top of it that implements resource scoping. One can say that we treat clusters as first-class citizens. In regard of clusters we fulfill the objective that we had from the very beginning: provide a multi-tenant API for Kubernetes clusters to the end-users.

However, as time passed by, things changed: The metal-stack is becoming an open-source product and we already have promising adopters of our product, who are willing to contribute to metal-stack. This is a serious chance of making our product better and more successful. It turns out that the decision we made was sufficient for us, but for others it is not.

Why adopters need multi-tenancy in the metal-api

Not every adopter will be interested in the cloud-api

For example, users who want to combine the Metal Stack with Gardener, may not want to hide all of the Gardener's functionality behind the cloud-api in the way we do. They want to use the much more powerful Gardener Dashboard instead. The Gardener itself does not need the cloud-api either. It is a cluster-api by itself. It only needs to utilize our "low-level API" and actually expects this API to have multi-tenancy as otherwise every logged in user can create / destroy clusters in every existing project from the Gardener dashboard.

This makes obvious that, with our decision, we placed an unnecessary obstacle in the way of our adopters: They now need to implement an own layer between the Gardener and the metal-api to provide multi-tenancy. From the Gardener-perspective we strongly differ from other cloud providers in this aspect and it is a matter of time when this will become an issue. When we encourage adopters to implement such interfaces on their own we also partly lose control of our product, we increase divergence.

We cannot claim that Metal Stack is a multi-tenant solution on our website

As the cloud-api is not part of the Metal Stack, the promise of multi-tenancy is only true for our network layer. Without the cloud-api to enable multi-tenancy, the network isolation is currently useless for end-users. Users of the Metal Stack can not self-manage machines, networks and ips without compromising the environment and thus, there is no self-service. We lose a valuable selling point when adopters can not immediately make use of our leading edge network isolation where we put so much effort to.

Open partitions for third-party usage

If a third-party uses Gardener and our metal-api had multi-tenancy, we would be able to allow a third-party to create clusters with workers in our own partitions. At the moment, this is not possible because the Gardener needs to know the HMAC secrets to create worker nodes, which would compromise our environment. If a thirdy-party knows our HMAC we lose control over the machines of our own tenants.

We do not actually want to open-source the cloud-api

One could think about solving the multi-tenancy issue by adding machine endpoints to the cloud-api. Gardener would then not consume the metal-api anymore but only the cloud-api.

This approach would not be ideal. We only want to offer a minimum viable product to adopters. The Gardener does not need a cluster-api as provided by the cloud-api. We want to treat additions on top of the basic stack as enterprise products.

The cloud-api contains billing endpoints, which are a perfect example for an optional addition of the Metal Stack. For basic usage of the Metal Stack a user does not need billing. Still, billing functionality can be interesting for some enterprises, who are like us, selling the infrastructure to third-parties.

Increased security for provider admins

Multi-tenancy in the metal-api also has the potential to limit the damage that a provider administrator can cause by mistake. If an administrator has to acquire project permissions on machine-level we can effectively reduce the damage he can make to this single project.

Another example would be the automatic provisioning of a Gitlab CI runner used for integration testing (a use case that we have where we do not require the cloud-api). This can easily be done in automated manner with Ansible and the Metal dynamic inventory + modules. However, with Ansible, mistakes in the automation can be made very quickly and if Ansible would only see machines of a dedicated project, this would also reduce damage it can make.

It is likely that there are more similar use-cases like that to come (maybe even for the storage solution?).

Also the surface for our Gardener components (metal-ccm, gardener-extension-provider-metal, machine-controller-manager) would be reduced to project scopes.

Conclusion

For these reasons the decision we made is very likely to have a negative impact on the adoption-rate of the Metal Stack and we should think about treating machines, networks and ips as first-class citizens as well. This makes us closer to the offer of hyperscalers. As mentioned in the beginning, all the time we tried to keep the possibility open to just offer bare metal machines. Let's continue with decision by adding multi-tenancy to the metal-api.

Required actions

Resource scoping

Just as implemented by the cloud-api, resource scoping needs to be added to almost every endpoint of the metal-api:

  • Machines / Firewalls
    • A user should only be able to view machines / firewalls of the projects he has at least view access to
    • A user should only be able to create and destroy machines / firewalls for projects he has at least editor access to Provider-tenants with at least view access can additionally view machines which have no project assignments Provider-tenants with at least editor access can additionally allocate / reserve machines which have no project assignments
  • Networks
    • A user should only be able to view networks of the projects he has at least view access to
    • A user should only be able to allocate networks of projects he has at least editor access to
    • A user should only be able to free networks assigned to projects he has at least editor access to Provider-tenants with at least view access can additionally view networks which have no project assignments Provider-tenants with at least editor access can additionally edit networks which have no project assignments Provider-tenants with at least admin access can additionally create or remove networks which have no project assignments
  • IPs
    • A user should only be able to view ips of the projects he has at least view access to
    • A user should only be able to allocate ips in networks of projects he has at least editor access to
    • A user should only be able to free ips assigned to projects he has at least editor access to
  • Projects
    • A logged in user is able to create projects when he has the permission to create projects
    • A user should only be able to view projects where he has at least view access to
    • A user should only be able to delete projects where he has admin access to
  • Partitions / Images
    • Only provider-admin users can add, delete, update
    • All logged in users can view
  • IPMI
    • Only provider-tenants can view machine IPMI data
  • Endpoints for internal use
    • Should only be accessible with HMAC auth and the HMAC secrets are only known by components of the Metal Stack (mainly for communication between partition and control plane), never for third-party usage

For all of this we need enhance the database queries with a filter for projects that a user has access to. As we already use a client to the masterdata-api in the metal-api, we can extract project memberships of a logged in user from there.

More permissions

We do not only need kaas-... permissions in the LDAP but also maas-. This way we can differentiate between permissions for the cloud-api and permissions for the metal-api.

Service account tokens / technical users

We need to provide the possibility for users to obtain access tokens to use for technical purposes (CI, third-party tooling like Gardener, ...).

We do not have this functionality yet, but it would also become a necessity for the cloud-api at some point in the future.

Cloud API

  • Project creation and deletion again have to be moved back into the metal-api, this also frees adopters from the need to write an own API in order to manage projects- The cloud-api will (again) only proxy project endpoints through to the metal-api
  • Do not point the secret bindings to a the shared provider secret in a partition. Create an individual provider-secret for the logged in tenant. The Gardener needs to use this tenant-specific provider secret to talk to the metal-api, do not give the Gardener HMAC access anymore.
  • The provider secret partition mapping can be removed from the cloud-api config and from the deployment
+Multi-tenancy for the metal-api · metal-stack
Edit on GitHub

Multi-tenancy for the metal-api

In the past we decided to treat the metal-api as a "low-level API", i.e. the API does not know anything about projects and tenants. A user with editor access can for example assign machines to every project he desires, he can see all the machines available and control them. Even though we always wanted to keep open the possibility to just offer bare metal machines to the end-user, the ultimate objective has always been to create an API for Kubernetes clusters. Hence, we tried to keep the metal-api code base as small as possible and we added resource scoping to a "higher-level API", the cloud-api, a component that is not open-source. From there, a user would be able to only see his own clusters and IP addresses.

The implication is that the metal-api has no multi-tenancy without another layer on top of it that implements resource scoping. One can say that we treat clusters as first-class citizens. In regard of clusters we fulfill the objective that we had from the very beginning: provide a multi-tenant API for Kubernetes clusters to the end-users.

However, as time passed by, things changed: The metal-stack is becoming an open-source product and we already have promising adopters of our product, who are willing to contribute to metal-stack. This is a serious chance of making our product better and more successful. It turns out that the decision we made was sufficient for us, but for others it is not.

Why adopters need multi-tenancy in the metal-api

Not every adopter will be interested in the cloud-api

For example, users who want to combine the Metal Stack with Gardener, may not want to hide all of the Gardener's functionality behind the cloud-api in the way we do. They want to use the much more powerful Gardener Dashboard instead. The Gardener itself does not need the cloud-api either. It is a cluster-api by itself. It only needs to utilize our "low-level API" and actually expects this API to have multi-tenancy as otherwise every logged in user can create / destroy clusters in every existing project from the Gardener dashboard.

This makes obvious that, with our decision, we placed an unnecessary obstacle in the way of our adopters: They now need to implement an own layer between the Gardener and the metal-api to provide multi-tenancy. From the Gardener-perspective we strongly differ from other cloud providers in this aspect and it is a matter of time when this will become an issue. When we encourage adopters to implement such interfaces on their own we also partly lose control of our product, we increase divergence.

We cannot claim that Metal Stack is a multi-tenant solution on our website

As the cloud-api is not part of the Metal Stack, the promise of multi-tenancy is only true for our network layer. Without the cloud-api to enable multi-tenancy, the network isolation is currently useless for end-users. Users of the Metal Stack can not self-manage machines, networks and ips without compromising the environment and thus, there is no self-service. We lose a valuable selling point when adopters can not immediately make use of our leading edge network isolation where we put so much effort to.

Open partitions for third-party usage

If a third-party uses Gardener and our metal-api had multi-tenancy, we would be able to allow a third-party to create clusters with workers in our own partitions. At the moment, this is not possible because the Gardener needs to know the HMAC secrets to create worker nodes, which would compromise our environment. If a thirdy-party knows our HMAC we lose control over the machines of our own tenants.

We do not actually want to open-source the cloud-api

One could think about solving the multi-tenancy issue by adding machine endpoints to the cloud-api. Gardener would then not consume the metal-api anymore but only the cloud-api.

This approach would not be ideal. We only want to offer a minimum viable product to adopters. The Gardener does not need a cluster-api as provided by the cloud-api. We want to treat additions on top of the basic stack as enterprise products.

The cloud-api contains billing endpoints, which are a perfect example for an optional addition of the Metal Stack. For basic usage of the Metal Stack a user does not need billing. Still, billing functionality can be interesting for some enterprises, who are like us, selling the infrastructure to third-parties.

Increased security for provider admins

Multi-tenancy in the metal-api also has the potential to limit the damage that a provider administrator can cause by mistake. If an administrator has to acquire project permissions on machine-level we can effectively reduce the damage he can make to this single project.

Another example would be the automatic provisioning of a Gitlab CI runner used for integration testing (a use case that we have where we do not require the cloud-api). This can easily be done in automated manner with Ansible and the Metal dynamic inventory + modules. However, with Ansible, mistakes in the automation can be made very quickly and if Ansible would only see machines of a dedicated project, this would also reduce damage it can make.

It is likely that there are more similar use-cases like that to come (maybe even for the storage solution?).

Also the surface for our Gardener components (metal-ccm, gardener-extension-provider-metal, machine-controller-manager) would be reduced to project scopes.

Conclusion

For these reasons the decision we made is very likely to have a negative impact on the adoption-rate of the Metal Stack and we should think about treating machines, networks and ips as first-class citizens as well. This makes us closer to the offer of hyperscalers. As mentioned in the beginning, all the time we tried to keep the possibility open to just offer bare metal machines. Let's continue with decision by adding multi-tenancy to the metal-api.

Required actions

Resource scoping

Just as implemented by the cloud-api, resource scoping needs to be added to almost every endpoint of the metal-api:

  • Machines / Firewalls
    • A user should only be able to view machines / firewalls of the projects he has at least view access to
    • A user should only be able to create and destroy machines / firewalls for projects he has at least editor access to Provider-tenants with at least view access can additionally view machines which have no project assignments Provider-tenants with at least editor access can additionally allocate / reserve machines which have no project assignments
  • Networks
    • A user should only be able to view networks of the projects he has at least view access to
    • A user should only be able to allocate networks of projects he has at least editor access to
    • A user should only be able to free networks assigned to projects he has at least editor access to Provider-tenants with at least view access can additionally view networks which have no project assignments Provider-tenants with at least editor access can additionally edit networks which have no project assignments Provider-tenants with at least admin access can additionally create or remove networks which have no project assignments
  • IPs
    • A user should only be able to view ips of the projects he has at least view access to
    • A user should only be able to allocate ips in networks of projects he has at least editor access to
    • A user should only be able to free ips assigned to projects he has at least editor access to
  • Projects
    • A logged in user is able to create projects when he has the permission to create projects
    • A user should only be able to view projects where he has at least view access to
    • A user should only be able to delete projects where he has admin access to
  • Partitions / Images
    • Only provider-admin users can add, delete, update
    • All logged in users can view
  • IPMI
    • Only provider-tenants can view machine IPMI data
  • Endpoints for internal use
    • Should only be accessible with HMAC auth and the HMAC secrets are only known by components of the Metal Stack (mainly for communication between partition and control plane), never for third-party usage

For all of this we need enhance the database queries with a filter for projects that a user has access to. As we already use a client to the masterdata-api in the metal-api, we can extract project memberships of a logged in user from there.

More permissions

We do not only need kaas-... permissions in the LDAP but also maas-. This way we can differentiate between permissions for the cloud-api and permissions for the metal-api.

Service account tokens / technical users

We need to provide the possibility for users to obtain access tokens to use for technical purposes (CI, third-party tooling like Gardener, ...).

We do not have this functionality yet, but it would also become a necessity for the cloud-api at some point in the future.

Cloud API

  • Project creation and deletion again have to be moved back into the metal-api, this also frees adopters from the need to write an own API in order to manage projects- The cloud-api will (again) only proxy project endpoints through to the metal-api
  • Do not point the secret bindings to a the shared provider secret in a partition. Create an individual provider-secret for the logged in tenant. The Gardener needs to use this tenant-specific provider secret to talk to the metal-api, do not give the Gardener HMAC access anymore.
  • The provider secret partition mapping can be removed from the cloud-api config and from the deployment
diff --git a/dev/development/proposals/MEP5/README/index.html b/dev/development/proposals/MEP5/README/index.html index 32e1020380..a7a6fb6920 100644 --- a/dev/development/proposals/MEP5/README/index.html +++ b/dev/development/proposals/MEP5/README/index.html @@ -1,2 +1,2 @@ -Shared Networks · metal-stack
Edit on GitHub

Shared Networks

Why are shared networks needed

For special purpose machines that serve shared services with performance critical workloads to all machines of a partition (like persistent storage) it would be good to have kind of a "shared network" that is easily accessible. They do not necessarily need another firewall. This would avoid having two firewalls in the datapath between a machine in a private network and the machines of a shared service.

Constraints that need to hold

  • a shared network is usable from all machines that have a firewall in front, that uses it
  • a shared network is only usable within a single partition (currently we are constrained in bandwidth and have no routing of 10.0.0.0/8 addresses btw. partitions and failure domain should be the partition but this constraint might get lifted in the future)
  • networks may be marked as shared after network allocation (but there should be no way back from shared to unshared)
  • neither machines nor firewalls may have multiple private, unshared networks configured
  • machines must have a single primary network configured
    • this might be a shared network
    • OR a plain, unshared private network
  • firewalls may participate in multiple shared networks
  • machines can be allocated with a primary network using auto IP allocation or with noauto and a specific IP

Should shared networks be private

Alternative 1: If we implemented shared networks by extending functions around plain, private networks we would not have to manage another CIDR (mini point) and it would be possible to create a k8s cluster with a private network, mark the network as shared and produce shared services from this k8s cluster.

Alternative 2: If shared networks are implemented as first class networks we could customize the VRF and also accomplish an other goal of our roadmap: being able to create machines directly in an external network.

Together with @majst01 and @Gerrit91 we decided to continue to implement Alternative 1.

Firewalls accessing a shared network

Firewalls that access shared networks need to:

  • hide the private network behind an ip address of the shared network if the shared network was configured with nat=true.
  • import the prefixes of the shared VRF to the private VRF and import the prefixes of the private VRF to the shared VRF so that the communication between the two is working in both directions. As long as no nat=true was set on the shared VRF, the original machine ips are visible in both communication directions.

Setup with shared networks and single consumer

Simple Setup

Setup with single shared network and multiple consumers

Advanced Setup

Getting internet acccess

Machines contained in a shared network can access the internet with different scenarios:

  • if they have an own firewall: this is internet accessibility, as common (check whether all traffic gets routed through it!)
  • if they don't have an own firewall, an external HTTP proxy is needed that has an endpoint exposed as Service Type NodePort
+Shared Networks · metal-stack
Edit on GitHub

Shared Networks

Why are shared networks needed

For special purpose machines that serve shared services with performance critical workloads to all machines of a partition (like persistent storage) it would be good to have kind of a "shared network" that is easily accessible. They do not necessarily need another firewall. This would avoid having two firewalls in the datapath between a machine in a private network and the machines of a shared service.

Constraints that need to hold

  • a shared network is usable from all machines that have a firewall in front, that uses it
  • a shared network is only usable within a single partition (currently we are constrained in bandwidth and have no routing of 10.0.0.0/8 addresses btw. partitions and failure domain should be the partition but this constraint might get lifted in the future)
  • networks may be marked as shared after network allocation (but there should be no way back from shared to unshared)
  • neither machines nor firewalls may have multiple private, unshared networks configured
  • machines must have a single primary network configured
    • this might be a shared network
    • OR a plain, unshared private network
  • firewalls may participate in multiple shared networks
  • machines can be allocated with a primary network using auto IP allocation or with noauto and a specific IP

Should shared networks be private

Alternative 1: If we implemented shared networks by extending functions around plain, private networks we would not have to manage another CIDR (mini point) and it would be possible to create a k8s cluster with a private network, mark the network as shared and produce shared services from this k8s cluster.

Alternative 2: If shared networks are implemented as first class networks we could customize the VRF and also accomplish an other goal of our roadmap: being able to create machines directly in an external network.

Together with @majst01 and @Gerrit91 we decided to continue to implement Alternative 1.

Firewalls accessing a shared network

Firewalls that access shared networks need to:

  • hide the private network behind an ip address of the shared network if the shared network was configured with nat=true.
  • import the prefixes of the shared VRF to the private VRF and import the prefixes of the private VRF to the shared VRF so that the communication between the two is working in both directions. As long as no nat=true was set on the shared VRF, the original machine ips are visible in both communication directions.

Setup with shared networks and single consumer

Simple Setup

Setup with single shared network and multiple consumers

Advanced Setup

Getting internet acccess

Machines contained in a shared network can access the internet with different scenarios:

  • if they have an own firewall: this is internet accessibility, as common (check whether all traffic gets routed through it!)
  • if they don't have an own firewall, an external HTTP proxy is needed that has an endpoint exposed as Service Type NodePort
diff --git a/dev/development/proposals/MEP6/README/index.html b/dev/development/proposals/MEP6/README/index.html index 26e6cd67f7..0af7555406 100644 --- a/dev/development/proposals/MEP6/README/index.html +++ b/dev/development/proposals/MEP6/README/index.html @@ -35,4 +35,4 @@ vrfshared: false nat: true shared: true # it's usable from multiple projects -underlay: false

DMZ firewall

The firewall of the DMZ will intersect its private network for attached machines, the DMZ network and the public internet.

Application Firewall

The firewall of application workloads intersects its private network for attached machines and the DMZ network.

Code Changes / Implications

Decision

We decided to follow the second approach with private DMZ networks.

+underlay: false

DMZ firewall

The firewall of the DMZ will intersect its private network for attached machines, the DMZ network and the public internet.

Application Firewall

The firewall of application workloads intersects its private network for attached machines and the DMZ network.

Code Changes / Implications

Decision

We decided to follow the second approach with private DMZ networks.

diff --git a/dev/development/proposals/MEP8/README/index.html b/dev/development/proposals/MEP8/README/index.html index 2186f27579..a5b285d0f3 100644 --- a/dev/development/proposals/MEP8/README/index.html +++ b/dev/development/proposals/MEP8/README/index.html @@ -364,4 +364,4 @@ - device: "/dev/nvmne0n1" wipeonreinstall: false - device: "/dev/nvmne0n2" - wipeonreinstall: false

Components which requires modifications

+ wipeonreinstall: false

Components which requires modifications

diff --git a/dev/development/proposals/MEP9/README/index.html b/dev/development/proposals/MEP9/README/index.html index 4bbceea29c..d7711f30a0 100644 --- a/dev/development/proposals/MEP9/README/index.html +++ b/dev/development/proposals/MEP9/README/index.html @@ -10,4 +10,4 @@ Status: Boolean field tailscale: Version: Actual version - ...

bmc-reverse-proxy

TODO

References

  1. WireGuard: Next Generation Secure Network Tunnel
  2. How Tailscale works
  3. Tailscale is officially SOC 2 compliant
  4. Why not Wireguard
  5. Wireguard: Known Limitations
  6. Wireguard: Things That Might Be Accomplished
  7. Headscale: Tailscale control protocol v2
+ ...

bmc-reverse-proxy

TODO

References

  1. WireGuard: Next Generation Secure Network Tunnel
  2. How Tailscale works
  3. Tailscale is officially SOC 2 compliant
  4. Why not Wireguard
  5. Wireguard: Known Limitations
  6. Wireguard: Things That Might Be Accomplished
  7. Headscale: Tailscale control protocol v2
diff --git a/dev/development/proposals/index.html b/dev/development/proposals/index.html index 825e666d13..9a6a100d5b 100644 --- a/dev/development/proposals/index.html +++ b/dev/development/proposals/index.html @@ -1,2 +1,2 @@ -Enhancement Proposals · metal-stack
Edit on GitHub

Metal Stack Enhancement Proposals (MEPs)

This section contains proposals which address substantial modifications to metal-stack.

Every proposal has a short name which starts with MEP followed by an incremental, unique number. Proposals should be raised as pull requests in the docs repository and can be discussed in Github issues.

The list of proposal and their current state is listed in the table below.

Possible states are:

  • In Discussion
  • Accepted
  • Declined
  • In Progress
  • Completed
  • Aborted

Once a proposal was accepted, an issue should be raised and the implementation should be done in a separate PR.

NameDescriptionState
MEP-1Distributed Control Plane DeploymentIn Discussion
MEP-2Two Factor AuthenticationAborted
MEP-3Machine Re-Installation to preserve local dataCompleted
MEP-4Multi-tenancy for the metal-apiAccepted
MEP-5Shared NetworksCompleted
MEP-6DMZ NetworksCompleted
MEP-8Configurable FilesystemlayoutCompleted
MEP-9No Open Ports To the Data CenterCompleted
MEP-11Auditing of metal-stack resourcesCompleted
MEP-12Rack SpreadingIn Discussion
+Enhancement Proposals · metal-stack
Edit on GitHub

Metal Stack Enhancement Proposals (MEPs)

This section contains proposals which address substantial modifications to metal-stack.

Every proposal has a short name which starts with MEP followed by an incremental, unique number. Proposals should be raised as pull requests in the docs repository and can be discussed in Github issues.

The list of proposal and their current state is listed in the table below.

Possible states are:

  • In Discussion
  • Accepted
  • Declined
  • In Progress
  • Completed
  • Aborted

Once a proposal was accepted, an issue should be raised and the implementation should be done in a separate PR.

NameDescriptionState
MEP-1Distributed Control Plane DeploymentIn Discussion
MEP-2Two Factor AuthenticationAborted
MEP-3Machine Re-Installation to preserve local dataCompleted
MEP-4Multi-tenancy for the metal-apiAccepted
MEP-5Shared NetworksCompleted
MEP-6DMZ NetworksCompleted
MEP-8Configurable FilesystemlayoutCompleted
MEP-9No Open Ports To the Data CenterCompleted
MEP-11Auditing of metal-stack resourcesCompleted
MEP-12Rack SpreadingIn Discussion
diff --git a/dev/development/roadmap/index.html b/dev/development/roadmap/index.html index 62fa5e330d..490b608310 100644 --- a/dev/development/roadmap/index.html +++ b/dev/development/roadmap/index.html @@ -1,2 +1,2 @@ -Roadmap · metal-stack
Edit on GitHub

Roadmap

A roadmap with short-, mid- and long-term planning will be available soon. For now, there is only a backlog.

Short-term

Available soon.

Mid-term

Available soon.

Long-term

Available soon.

Backlog

The backlog contains ideas of what could become part of the roadmap in the future. The list is ordered alphabetically. Therefore, the order does not express the importance or weight of a backlog item.

We incorporate community feedback into the roadmap. If you think that important points are missing in the backlog, please share your ideas with us. We have a Slack channel. Please check out metal-stack.io for contact information.

Danger

By no means this list is a promise of what is being worked on in the near future. It is just a summary of ideas that was agreed on to be "nice to have". It is up to the investors, maintainers and the community to choose topics from this list and to implement them or to remove them from the list.

  • Add metal-stack to Gardener conformance test grid
  • Autoscaler for metal control plane components
  • CI dashboard and public integration testing
  • Cilium as the default CNI for metal-stack on Gardener K8s clusters
  • Improved release and deploy processes (GitOps, Spinnaker, Flux)
  • Machine internet without firewalls
  • metal-stack dashboard (UI)
  • Offer our metal-stack extensions as enterprise products (accounting, cluster-api, S3) (neither of them will ever be required for running metal-stack, they just add extra value for certain enterprises)
  • Partition managed by Kubernetes (with Kubelets joining the control plane cluster)
  • Public offering / demo playground
  • Resource scoping in the metal-api (MEP-4)
  • Service / API tokens (for scoped technical user access)
+Roadmap · metal-stack
Edit on GitHub

Roadmap

A roadmap with short-, mid- and long-term planning will be available soon. For now, there is only a backlog.

Short-term

Available soon.

Mid-term

Available soon.

Long-term

Available soon.

Backlog

The backlog contains ideas of what could become part of the roadmap in the future. The list is ordered alphabetically. Therefore, the order does not express the importance or weight of a backlog item.

We incorporate community feedback into the roadmap. If you think that important points are missing in the backlog, please share your ideas with us. We have a Slack channel. Please check out metal-stack.io for contact information.

Danger

By no means this list is a promise of what is being worked on in the near future. It is just a summary of ideas that was agreed on to be "nice to have". It is up to the investors, maintainers and the community to choose topics from this list and to implement them or to remove them from the list.

  • Add metal-stack to Gardener conformance test grid
  • Autoscaler for metal control plane components
  • CI dashboard and public integration testing
  • Cilium as the default CNI for metal-stack on Gardener K8s clusters
  • Improved release and deploy processes (GitOps, Spinnaker, Flux)
  • Machine internet without firewalls
  • metal-stack dashboard (UI)
  • Offer our metal-stack extensions as enterprise products (accounting, cluster-api, S3) (neither of them will ever be required for running metal-stack, they just add extra value for certain enterprises)
  • Partition managed by Kubernetes (with Kubelets joining the control plane cluster)
  • Public offering / demo playground
  • Resource scoping in the metal-api (MEP-4)
  • Service / API tokens (for scoped technical user access)
diff --git a/dev/external/csi-driver-lvm/CONTRIBUTING/index.html b/dev/external/csi-driver-lvm/CONTRIBUTING/index.html index d96937fa90..13bb0bdd20 100644 --- a/dev/external/csi-driver-lvm/CONTRIBUTING/index.html +++ b/dev/external/csi-driver-lvm/CONTRIBUTING/index.html @@ -1,2 +1,2 @@ -Contributing · metal-stack
Edit on GitHub
+Contributing · metal-stack
Edit on GitHub
diff --git a/dev/external/csi-driver-lvm/README/index.html b/dev/external/csi-driver-lvm/README/index.html index a75577d797..7d3010a20c 100644 --- a/dev/external/csi-driver-lvm/README/index.html +++ b/dev/external/csi-driver-lvm/README/index.html @@ -13,4 +13,4 @@ kubectl delete -f examples/csi-pvc.yaml

Development

In order to run the integration tests locally, you need to create to loop devices on your host machine. Make sure the loop device mount paths are not used on your system (default path is /dev/loop10{1,2}).

You can create these loop devices like this:

for i in 100 101; do fallocate -l 1G loop${i}.img ; sudo losetup /dev/loop${i} loop${i}.img; done
 sudo losetup -a
 # use this for recreation or cleanup
-# for i in 100 101; do sudo losetup -d /dev/loop${i}; rm -f loop${i}.img; done

You can then run the tests against a kind cluster, running:

make test

To recreate or cleanup the kind cluster:

make test-cleanup

Page Tree

+# for i in 100 101; do sudo losetup -d /dev/loop${i}; rm -f loop${i}.img; done

You can then run the tests against a kind cluster, running:

make test

To recreate or cleanup the kind cluster:

make test-cleanup

Page Tree

diff --git a/dev/external/firewall-controller/CONTRIBUTING/index.html b/dev/external/firewall-controller/CONTRIBUTING/index.html index d61ac6794f..b22599881a 100644 --- a/dev/external/firewall-controller/CONTRIBUTING/index.html +++ b/dev/external/firewall-controller/CONTRIBUTING/index.html @@ -1,2 +1,2 @@ -Contributing · metal-stack
Edit on GitHub
+Contributing · metal-stack
Edit on GitHub
diff --git a/dev/external/firewall-controller/DEVELOP/index.html b/dev/external/firewall-controller/DEVELOP/index.html index 9947515339..0d351f5b8b 100644 --- a/dev/external/firewall-controller/DEVELOP/index.html +++ b/dev/external/firewall-controller/DEVELOP/index.html @@ -18,4 +18,4 @@ # watch results k describe -n firewall firewall cat nftables.v4 -cat hosts +cat hosts diff --git a/dev/external/firewall-controller/README/index.html b/dev/external/firewall-controller/README/index.html index 28c69da805..381bd1cf41 100644 --- a/dev/external/firewall-controller/README/index.html +++ b/dev/external/firewall-controller/README/index.html @@ -122,4 +122,4 @@ droptailer-6d556bd988-4g8gp droptailer 2020-06-17 13:23:30 +0000 UTC {"DPT":"650","DST":"1.2.3.4","ID":"12399","IN":"vrf104009","LEN":"40","MAC":"ca:41:f9:80:fa:89:aa:bb:0e:62:8c:a6:08:00","OUT":"vlan179","PREC":"0x00","PROTO":"TCP","RES":"0x00","SPT":"40194","SRC":"2.3.4.5","SYN":"","TOS":"0x00","TTL":"241","URGP":"0","WINDOW":"1024","timestamp":"2020-06-17 13:23:30 +0000 UTC"} droptailer-6d556bd988-4g8gp droptailer 2020-06-17 13:23:34 +0000 UTC {"DPT":"2362","DST":"1.2.3.4","ID":"44545","IN":"vrf104009","LEN":"40","MAC":"ca:41:f9:80:fa:89:aa:bb:0e:62:8c:a6:08:00","OUT":"","PREC":"0x00","PROTO":"TCP","RES":"0x00","SPT":"40194","SRC":"2.3.4.5","SYN":"","TOS":"0x00","TTL":"242","URGP":"0","WINDOW":"1024","timestamp":"2020-06-17 13:23:34 +0000 UTC"} droptailer-6d556bd988-4g8gp droptailer 2020-06-17 13:23:10 +0000 UTC {"DPT":"63351","DST":"1.2.3.4","ID":"11855","IN":"vrf104009","LEN":"40","MAC":"ca:41:f9:80:fa:89:aa:bb:0e:62:8c:a6:08:00","OUT":"vlan179","PREC":"0x00","PROTO":"TCP","RES":"0x00","SPT":"54589","SRC":"2.3.4.5","SYN":"","TOS":"0x00","TTL":"245","URGP":"0","WINDOW":"1024","timestamp":"2020-06-17 13:23:10 +0000 UTC"} -droptailer-6d556bd988-4g8gp droptailer 2020-06-17 13:23:51 +0000 UTC {"DPT":"8002","DST":"1.2.3.4","ID":"17539","IN":"vrf104009","LEN":"40","MAC":"ca:41:f9:80:fa:89:aa:bb:0e:62:8c:a6:08:00","OUT":"","PREC":"0x00","PROTO":"TCP","RES":"0x00","SPT":"47615","SRC":"2.3.4.5","SYN":"","TOS":"0x08","TTL":"239","URGP":"0","WINDOW":"1024","timestamp":"2020-06-17 13:23:51 +0000 UTC"}

You can forward the droptailer logs to any log aggregation infrastructure you have in place.

Page Tree

+droptailer-6d556bd988-4g8gp droptailer 2020-06-17 13:23:51 +0000 UTC {"DPT":"8002","DST":"1.2.3.4","ID":"17539","IN":"vrf104009","LEN":"40","MAC":"ca:41:f9:80:fa:89:aa:bb:0e:62:8c:a6:08:00","OUT":"","PREC":"0x00","PROTO":"TCP","RES":"0x00","SPT":"47615","SRC":"2.3.4.5","SYN":"","TOS":"0x08","TTL":"239","URGP":"0","WINDOW":"1024","timestamp":"2020-06-17 13:23:51 +0000 UTC"}

You can forward the droptailer logs to any log aggregation infrastructure you have in place.

Page Tree

diff --git a/dev/external/metalctl/CONTRIBUTING/index.html b/dev/external/metalctl/CONTRIBUTING/index.html index 34b7bcdf1b..95db1d2030 100644 --- a/dev/external/metalctl/CONTRIBUTING/index.html +++ b/dev/external/metalctl/CONTRIBUTING/index.html @@ -1,2 +1,2 @@ -Contributing · metal-stack
Edit on GitHub
+Contributing · metal-stack
Edit on GitHub
diff --git a/dev/external/metalctl/README/index.html b/dev/external/metalctl/README/index.html index 5ca45c22d2..475b7fc84d 100644 --- a/dev/external/metalctl/README/index.html +++ b/dev/external/metalctl/README/index.html @@ -35,4 +35,4 @@ issuer_url: https://keycloak.somedomain.io custom_scopes: roles,openid,profile,email client_id: my-client-id - client_secret: my-secret

Available commands

Full documentation is generated out of the cobra command implementation with:

metalctl markdown

generated markdown is here and here

Page Tree

+ client_secret: my-secret

Available commands

Full documentation is generated out of the cobra command implementation with:

metalctl markdown

generated markdown is here and here

Page Tree

diff --git a/dev/external/metalctl/docs/metalctl/index.html b/dev/external/metalctl/docs/metalctl/index.html index ad7ecf183b..cdb709ff6d 100644 --- a/dev/external/metalctl/docs/metalctl/index.html +++ b/dev/external/metalctl/docs/metalctl/index.html @@ -22,4 +22,4 @@ metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" - --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

+ --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

diff --git a/dev/external/metalctl/docs/metalctl_audit/index.html b/dev/external/metalctl/docs/metalctl_audit/index.html index bca338e342..8202b247ae 100644 --- a/dev/external/metalctl/docs/metalctl_audit/index.html +++ b/dev/external/metalctl/docs/metalctl_audit/index.html @@ -21,4 +21,4 @@ metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" - --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

+ --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

diff --git a/dev/external/metalctl/docs/metalctl_audit_describe/index.html b/dev/external/metalctl/docs/metalctl_audit_describe/index.html index 529b42f46c..048b80590c 100644 --- a/dev/external/metalctl/docs/metalctl_audit_describe/index.html +++ b/dev/external/metalctl/docs/metalctl_audit_describe/index.html @@ -21,4 +21,4 @@ metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" - --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

+ --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

diff --git a/dev/external/metalctl/docs/metalctl_audit_list/index.html b/dev/external/metalctl/docs/metalctl_audit_list/index.html index 8fdcdae101..0022fc7209 100644 --- a/dev/external/metalctl/docs/metalctl_audit_list/index.html +++ b/dev/external/metalctl/docs/metalctl_audit_list/index.html @@ -38,4 +38,4 @@ metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" - --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

+ --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

diff --git a/dev/external/metalctl/docs/metalctl_completion/index.html b/dev/external/metalctl/docs/metalctl_completion/index.html index b4f88b90aa..3319aca8de 100644 --- a/dev/external/metalctl/docs/metalctl_completion/index.html +++ b/dev/external/metalctl/docs/metalctl_completion/index.html @@ -21,4 +21,4 @@ metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" - --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

+ --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

diff --git a/dev/external/metalctl/docs/metalctl_completion_bash/index.html b/dev/external/metalctl/docs/metalctl_completion_bash/index.html index 6b3592a5d1..4a8206283c 100644 --- a/dev/external/metalctl/docs/metalctl_completion_bash/index.html +++ b/dev/external/metalctl/docs/metalctl_completion_bash/index.html @@ -22,4 +22,4 @@ metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" - --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

+ --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

diff --git a/dev/external/metalctl/docs/metalctl_completion_fish/index.html b/dev/external/metalctl/docs/metalctl_completion_fish/index.html index 1e5f2bc323..20175da432 100644 --- a/dev/external/metalctl/docs/metalctl_completion_fish/index.html +++ b/dev/external/metalctl/docs/metalctl_completion_fish/index.html @@ -22,4 +22,4 @@ metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" - --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

+ --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

diff --git a/dev/external/metalctl/docs/metalctl_completion_powershell/index.html b/dev/external/metalctl/docs/metalctl_completion_powershell/index.html index 62347f5091..a50fb3dd3b 100644 --- a/dev/external/metalctl/docs/metalctl_completion_powershell/index.html +++ b/dev/external/metalctl/docs/metalctl_completion_powershell/index.html @@ -22,4 +22,4 @@ metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" - --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

+ --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

diff --git a/dev/external/metalctl/docs/metalctl_completion_zsh/index.html b/dev/external/metalctl/docs/metalctl_completion_zsh/index.html index b581cac9df..a30e40a1c8 100644 --- a/dev/external/metalctl/docs/metalctl_completion_zsh/index.html +++ b/dev/external/metalctl/docs/metalctl_completion_zsh/index.html @@ -22,4 +22,4 @@ metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" - --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

+ --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

diff --git a/dev/external/metalctl/docs/metalctl_context/index.html b/dev/external/metalctl/docs/metalctl_context/index.html index 7ac634acd0..6e200786b1 100644 --- a/dev/external/metalctl/docs/metalctl_context/index.html +++ b/dev/external/metalctl/docs/metalctl_context/index.html @@ -37,4 +37,4 @@ metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" - --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

+ --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

diff --git a/dev/external/metalctl/docs/metalctl_context_short/index.html b/dev/external/metalctl/docs/metalctl_context_short/index.html index d20928f5e4..f395e95236 100644 --- a/dev/external/metalctl/docs/metalctl_context_short/index.html +++ b/dev/external/metalctl/docs/metalctl_context_short/index.html @@ -21,4 +21,4 @@ metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" - --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

+ --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

diff --git a/dev/external/metalctl/docs/metalctl_filesystemlayout/index.html b/dev/external/metalctl/docs/metalctl_filesystemlayout/index.html index 3afda874a0..4d80ac9d96 100644 --- a/dev/external/metalctl/docs/metalctl_filesystemlayout/index.html +++ b/dev/external/metalctl/docs/metalctl_filesystemlayout/index.html @@ -21,4 +21,4 @@ metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" - --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

+ --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

diff --git a/dev/external/metalctl/docs/metalctl_filesystemlayout_apply/index.html b/dev/external/metalctl/docs/metalctl_filesystemlayout_apply/index.html index f43730b776..648f1af25f 100644 --- a/dev/external/metalctl/docs/metalctl_filesystemlayout_apply/index.html +++ b/dev/external/metalctl/docs/metalctl_filesystemlayout_apply/index.html @@ -31,4 +31,4 @@ metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" - --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

+ --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

diff --git a/dev/external/metalctl/docs/metalctl_filesystemlayout_create/index.html b/dev/external/metalctl/docs/metalctl_filesystemlayout_create/index.html index a87cf27df6..b8973ecf14 100644 --- a/dev/external/metalctl/docs/metalctl_filesystemlayout_create/index.html +++ b/dev/external/metalctl/docs/metalctl_filesystemlayout_create/index.html @@ -31,4 +31,4 @@ metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" - --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

+ --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

diff --git a/dev/external/metalctl/docs/metalctl_filesystemlayout_delete/index.html b/dev/external/metalctl/docs/metalctl_filesystemlayout_delete/index.html index f470f52e81..2c4f54f61e 100644 --- a/dev/external/metalctl/docs/metalctl_filesystemlayout_delete/index.html +++ b/dev/external/metalctl/docs/metalctl_filesystemlayout_delete/index.html @@ -21,4 +21,4 @@ metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" - --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

+ --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

diff --git a/dev/external/metalctl/docs/metalctl_filesystemlayout_describe/index.html b/dev/external/metalctl/docs/metalctl_filesystemlayout_describe/index.html index 8ad93221fe..ec505b8285 100644 --- a/dev/external/metalctl/docs/metalctl_filesystemlayout_describe/index.html +++ b/dev/external/metalctl/docs/metalctl_filesystemlayout_describe/index.html @@ -21,4 +21,4 @@ metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" - --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

+ --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

diff --git a/dev/external/metalctl/docs/metalctl_filesystemlayout_edit/index.html b/dev/external/metalctl/docs/metalctl_filesystemlayout_edit/index.html index 9c4184330c..3eeff44f46 100644 --- a/dev/external/metalctl/docs/metalctl_filesystemlayout_edit/index.html +++ b/dev/external/metalctl/docs/metalctl_filesystemlayout_edit/index.html @@ -21,4 +21,4 @@ metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" - --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

+ --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

diff --git a/dev/external/metalctl/docs/metalctl_filesystemlayout_list/index.html b/dev/external/metalctl/docs/metalctl_filesystemlayout_list/index.html index d29916a669..da1e2be6d2 100644 --- a/dev/external/metalctl/docs/metalctl_filesystemlayout_list/index.html +++ b/dev/external/metalctl/docs/metalctl_filesystemlayout_list/index.html @@ -22,4 +22,4 @@ metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" - --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

+ --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

diff --git a/dev/external/metalctl/docs/metalctl_filesystemlayout_match/index.html b/dev/external/metalctl/docs/metalctl_filesystemlayout_match/index.html index dfa7c570ff..22d2f07936 100644 --- a/dev/external/metalctl/docs/metalctl_filesystemlayout_match/index.html +++ b/dev/external/metalctl/docs/metalctl_filesystemlayout_match/index.html @@ -23,4 +23,4 @@ metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" - --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

+ --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

diff --git a/dev/external/metalctl/docs/metalctl_filesystemlayout_try/index.html b/dev/external/metalctl/docs/metalctl_filesystemlayout_try/index.html index 857571a77a..9de9900ef3 100644 --- a/dev/external/metalctl/docs/metalctl_filesystemlayout_try/index.html +++ b/dev/external/metalctl/docs/metalctl_filesystemlayout_try/index.html @@ -23,4 +23,4 @@ metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" - --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

+ --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

diff --git a/dev/external/metalctl/docs/metalctl_filesystemlayout_update/index.html b/dev/external/metalctl/docs/metalctl_filesystemlayout_update/index.html index 769adcbb2d..14b58baff3 100644 --- a/dev/external/metalctl/docs/metalctl_filesystemlayout_update/index.html +++ b/dev/external/metalctl/docs/metalctl_filesystemlayout_update/index.html @@ -31,4 +31,4 @@ metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" - --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

+ --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

diff --git a/dev/external/metalctl/docs/metalctl_firewall/index.html b/dev/external/metalctl/docs/metalctl_firewall/index.html index 76d1dfd8e5..ec38101b38 100644 --- a/dev/external/metalctl/docs/metalctl_firewall/index.html +++ b/dev/external/metalctl/docs/metalctl_firewall/index.html @@ -21,4 +21,4 @@ metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" - --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

+ --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

diff --git a/dev/external/metalctl/docs/metalctl_firewall_create/index.html b/dev/external/metalctl/docs/metalctl_firewall_create/index.html index 5d72ac61b7..13c2745167 100644 --- a/dev/external/metalctl/docs/metalctl_firewall_create/index.html +++ b/dev/external/metalctl/docs/metalctl_firewall_create/index.html @@ -55,4 +55,4 @@ metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" - --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

+ --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

diff --git a/dev/external/metalctl/docs/metalctl_firewall_describe/index.html b/dev/external/metalctl/docs/metalctl_firewall_describe/index.html index fa3b99a020..ed77982a9e 100644 --- a/dev/external/metalctl/docs/metalctl_firewall_describe/index.html +++ b/dev/external/metalctl/docs/metalctl_firewall_describe/index.html @@ -21,4 +21,4 @@ metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" - --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

+ --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

diff --git a/dev/external/metalctl/docs/metalctl_firewall_list/index.html b/dev/external/metalctl/docs/metalctl_firewall_list/index.html index f237e04a03..d5c23717e4 100644 --- a/dev/external/metalctl/docs/metalctl_firewall_list/index.html +++ b/dev/external/metalctl/docs/metalctl_firewall_list/index.html @@ -31,4 +31,4 @@ metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" - --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

+ --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

diff --git a/dev/external/metalctl/docs/metalctl_firewall_ssh/index.html b/dev/external/metalctl/docs/metalctl_firewall_ssh/index.html index d769292e8c..40e5e4d1e8 100644 --- a/dev/external/metalctl/docs/metalctl_firewall_ssh/index.html +++ b/dev/external/metalctl/docs/metalctl_firewall_ssh/index.html @@ -22,4 +22,4 @@ metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" - --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

+ --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

diff --git a/dev/external/metalctl/docs/metalctl_firmware/index.html b/dev/external/metalctl/docs/metalctl_firmware/index.html index 81945e1642..164e66c1f0 100644 --- a/dev/external/metalctl/docs/metalctl_firmware/index.html +++ b/dev/external/metalctl/docs/metalctl_firmware/index.html @@ -21,4 +21,4 @@ metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" - --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

+ --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

diff --git a/dev/external/metalctl/docs/metalctl_firmware_delete/index.html b/dev/external/metalctl/docs/metalctl_firmware_delete/index.html index d2ac877519..8a3d7cd90c 100644 --- a/dev/external/metalctl/docs/metalctl_firmware_delete/index.html +++ b/dev/external/metalctl/docs/metalctl_firmware_delete/index.html @@ -25,4 +25,4 @@ metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" - --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

+ --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

diff --git a/dev/external/metalctl/docs/metalctl_firmware_list/index.html b/dev/external/metalctl/docs/metalctl_firmware_list/index.html index 7fa8dc5645..ee6d9b4dba 100644 --- a/dev/external/metalctl/docs/metalctl_firmware_list/index.html +++ b/dev/external/metalctl/docs/metalctl_firmware_list/index.html @@ -25,4 +25,4 @@ metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" - --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

+ --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

diff --git a/dev/external/metalctl/docs/metalctl_firmware_upload/index.html b/dev/external/metalctl/docs/metalctl_firmware_upload/index.html index af6b033460..61a0bfde4f 100644 --- a/dev/external/metalctl/docs/metalctl_firmware_upload/index.html +++ b/dev/external/metalctl/docs/metalctl_firmware_upload/index.html @@ -21,4 +21,4 @@ metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" - --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

+ --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

diff --git a/dev/external/metalctl/docs/metalctl_firmware_upload_bios/index.html b/dev/external/metalctl/docs/metalctl_firmware_upload_bios/index.html index a02acaebe0..d45deed414 100644 --- a/dev/external/metalctl/docs/metalctl_firmware_upload_bios/index.html +++ b/dev/external/metalctl/docs/metalctl_firmware_upload_bios/index.html @@ -24,4 +24,4 @@ metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" - --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

+ --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

diff --git a/dev/external/metalctl/docs/metalctl_firmware_upload_bmc/index.html b/dev/external/metalctl/docs/metalctl_firmware_upload_bmc/index.html index 77c6d55bf1..461acbb6e2 100644 --- a/dev/external/metalctl/docs/metalctl_firmware_upload_bmc/index.html +++ b/dev/external/metalctl/docs/metalctl_firmware_upload_bmc/index.html @@ -24,4 +24,4 @@ metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" - --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

+ --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

diff --git a/dev/external/metalctl/docs/metalctl_health/index.html b/dev/external/metalctl/docs/metalctl_health/index.html index 095bc39c15..8026c0f832 100644 --- a/dev/external/metalctl/docs/metalctl_health/index.html +++ b/dev/external/metalctl/docs/metalctl_health/index.html @@ -21,4 +21,4 @@ metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" - --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

+ --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

diff --git a/dev/external/metalctl/docs/metalctl_image/index.html b/dev/external/metalctl/docs/metalctl_image/index.html index 4044331267..a02a9d7aba 100644 --- a/dev/external/metalctl/docs/metalctl_image/index.html +++ b/dev/external/metalctl/docs/metalctl_image/index.html @@ -21,4 +21,4 @@ metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" - --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

+ --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

diff --git a/dev/external/metalctl/docs/metalctl_image_apply/index.html b/dev/external/metalctl/docs/metalctl_image_apply/index.html index ed585022a2..cb0f7118a6 100644 --- a/dev/external/metalctl/docs/metalctl_image_apply/index.html +++ b/dev/external/metalctl/docs/metalctl_image_apply/index.html @@ -31,4 +31,4 @@ metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" - --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

+ --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

diff --git a/dev/external/metalctl/docs/metalctl_image_create/index.html b/dev/external/metalctl/docs/metalctl_image_create/index.html index 443a3dfe12..ad6cad742f 100644 --- a/dev/external/metalctl/docs/metalctl_image_create/index.html +++ b/dev/external/metalctl/docs/metalctl_image_create/index.html @@ -36,4 +36,4 @@ metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" - --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

+ --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

diff --git a/dev/external/metalctl/docs/metalctl_image_delete/index.html b/dev/external/metalctl/docs/metalctl_image_delete/index.html index a11ec59b9d..a5e63ba138 100644 --- a/dev/external/metalctl/docs/metalctl_image_delete/index.html +++ b/dev/external/metalctl/docs/metalctl_image_delete/index.html @@ -21,4 +21,4 @@ metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" - --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

+ --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

diff --git a/dev/external/metalctl/docs/metalctl_image_describe/index.html b/dev/external/metalctl/docs/metalctl_image_describe/index.html index 4b781a2954..953e75701d 100644 --- a/dev/external/metalctl/docs/metalctl_image_describe/index.html +++ b/dev/external/metalctl/docs/metalctl_image_describe/index.html @@ -21,4 +21,4 @@ metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" - --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

+ --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

diff --git a/dev/external/metalctl/docs/metalctl_image_edit/index.html b/dev/external/metalctl/docs/metalctl_image_edit/index.html index 8fe59e8ac3..3155cd0c04 100644 --- a/dev/external/metalctl/docs/metalctl_image_edit/index.html +++ b/dev/external/metalctl/docs/metalctl_image_edit/index.html @@ -21,4 +21,4 @@ metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" - --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

+ --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

diff --git a/dev/external/metalctl/docs/metalctl_image_list/index.html b/dev/external/metalctl/docs/metalctl_image_list/index.html index 58006b5704..aa68e70aeb 100644 --- a/dev/external/metalctl/docs/metalctl_image_list/index.html +++ b/dev/external/metalctl/docs/metalctl_image_list/index.html @@ -29,4 +29,4 @@ metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" - --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

+ --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

diff --git a/dev/external/metalctl/docs/metalctl_image_update/index.html b/dev/external/metalctl/docs/metalctl_image_update/index.html index d1de513098..a2c2aa1015 100644 --- a/dev/external/metalctl/docs/metalctl_image_update/index.html +++ b/dev/external/metalctl/docs/metalctl_image_update/index.html @@ -31,4 +31,4 @@ metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" - --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

+ --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

diff --git a/dev/external/metalctl/docs/metalctl_login/index.html b/dev/external/metalctl/docs/metalctl_login/index.html index 247b3aeba7..890239706a 100644 --- a/dev/external/metalctl/docs/metalctl_login/index.html +++ b/dev/external/metalctl/docs/metalctl_login/index.html @@ -22,4 +22,4 @@ metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" - --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

+ --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

diff --git a/dev/external/metalctl/docs/metalctl_logout/index.html b/dev/external/metalctl/docs/metalctl_logout/index.html index 82be13beb3..ec7b67194b 100644 --- a/dev/external/metalctl/docs/metalctl_logout/index.html +++ b/dev/external/metalctl/docs/metalctl_logout/index.html @@ -21,4 +21,4 @@ metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" - --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

+ --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

diff --git a/dev/external/metalctl/docs/metalctl_machine/index.html b/dev/external/metalctl/docs/metalctl_machine/index.html index 063aa5f637..f7b5c44bfc 100644 --- a/dev/external/metalctl/docs/metalctl_machine/index.html +++ b/dev/external/metalctl/docs/metalctl_machine/index.html @@ -21,4 +21,4 @@ metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" - --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

+ --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

diff --git a/dev/external/metalctl/docs/metalctl_machine_apply/index.html b/dev/external/metalctl/docs/metalctl_machine_apply/index.html index 1b7ca6c0f1..daf253d937 100644 --- a/dev/external/metalctl/docs/metalctl_machine_apply/index.html +++ b/dev/external/metalctl/docs/metalctl_machine_apply/index.html @@ -31,4 +31,4 @@ metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" - --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

+ --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

diff --git a/dev/external/metalctl/docs/metalctl_machine_console/index.html b/dev/external/metalctl/docs/metalctl_machine_console/index.html index 363a58c8c1..756340c772 100644 --- a/dev/external/metalctl/docs/metalctl_machine_console/index.html +++ b/dev/external/metalctl/docs/metalctl_machine_console/index.html @@ -25,4 +25,4 @@ metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" - --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

+ --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

diff --git a/dev/external/metalctl/docs/metalctl_machine_consolepassword/index.html b/dev/external/metalctl/docs/metalctl_machine_consolepassword/index.html index b2f89b1d94..750b5f0c45 100644 --- a/dev/external/metalctl/docs/metalctl_machine_consolepassword/index.html +++ b/dev/external/metalctl/docs/metalctl_machine_consolepassword/index.html @@ -22,4 +22,4 @@ metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" - --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

+ --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

diff --git a/dev/external/metalctl/docs/metalctl_machine_create/index.html b/dev/external/metalctl/docs/metalctl_machine_create/index.html index beeb255b3d..d12b00e9c5 100644 --- a/dev/external/metalctl/docs/metalctl_machine_create/index.html +++ b/dev/external/metalctl/docs/metalctl_machine_create/index.html @@ -88,4 +88,4 @@ metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" - --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

+ --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

diff --git a/dev/external/metalctl/docs/metalctl_machine_delete/index.html b/dev/external/metalctl/docs/metalctl_machine_delete/index.html index 5010d0493b..76a4cea11f 100644 --- a/dev/external/metalctl/docs/metalctl_machine_delete/index.html +++ b/dev/external/metalctl/docs/metalctl_machine_delete/index.html @@ -22,4 +22,4 @@ metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" - --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

+ --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

diff --git a/dev/external/metalctl/docs/metalctl_machine_describe/index.html b/dev/external/metalctl/docs/metalctl_machine_describe/index.html index 6bc6510d2c..6821b221f1 100644 --- a/dev/external/metalctl/docs/metalctl_machine_describe/index.html +++ b/dev/external/metalctl/docs/metalctl_machine_describe/index.html @@ -21,4 +21,4 @@ metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" - --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

+ --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

diff --git a/dev/external/metalctl/docs/metalctl_machine_edit/index.html b/dev/external/metalctl/docs/metalctl_machine_edit/index.html index 93c9c9dd8d..cf5c39c999 100644 --- a/dev/external/metalctl/docs/metalctl_machine_edit/index.html +++ b/dev/external/metalctl/docs/metalctl_machine_edit/index.html @@ -21,4 +21,4 @@ metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" - --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

+ --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

diff --git a/dev/external/metalctl/docs/metalctl_machine_identify/index.html b/dev/external/metalctl/docs/metalctl_machine_identify/index.html index 58d77bf357..4ac34cd6e7 100644 --- a/dev/external/metalctl/docs/metalctl_machine_identify/index.html +++ b/dev/external/metalctl/docs/metalctl_machine_identify/index.html @@ -21,4 +21,4 @@ metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" - --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

+ --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

diff --git a/dev/external/metalctl/docs/metalctl_machine_identify_off/index.html b/dev/external/metalctl/docs/metalctl_machine_identify_off/index.html index 05e76cd10f..01213a05e9 100644 --- a/dev/external/metalctl/docs/metalctl_machine_identify_off/index.html +++ b/dev/external/metalctl/docs/metalctl_machine_identify_off/index.html @@ -22,4 +22,4 @@ metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" - --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

+ --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

diff --git a/dev/external/metalctl/docs/metalctl_machine_identify_on/index.html b/dev/external/metalctl/docs/metalctl_machine_identify_on/index.html index 52406d1f81..2a9953e3e1 100644 --- a/dev/external/metalctl/docs/metalctl_machine_identify_on/index.html +++ b/dev/external/metalctl/docs/metalctl_machine_identify_on/index.html @@ -22,4 +22,4 @@ metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" - --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

+ --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

diff --git a/dev/external/metalctl/docs/metalctl_machine_ipmi/index.html b/dev/external/metalctl/docs/metalctl_machine_ipmi/index.html index 7457ad239e..31d964fe3c 100644 --- a/dev/external/metalctl/docs/metalctl_machine_ipmi/index.html +++ b/dev/external/metalctl/docs/metalctl_machine_ipmi/index.html @@ -33,4 +33,4 @@ metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" - --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

+ --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

diff --git a/dev/external/metalctl/docs/metalctl_machine_ipmi_events/index.html b/dev/external/metalctl/docs/metalctl_machine_ipmi_events/index.html index ebc28677a5..8cc78816c2 100644 --- a/dev/external/metalctl/docs/metalctl_machine_ipmi_events/index.html +++ b/dev/external/metalctl/docs/metalctl_machine_ipmi_events/index.html @@ -24,4 +24,4 @@ metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" - --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

+ --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

diff --git a/dev/external/metalctl/docs/metalctl_machine_issues/index.html b/dev/external/metalctl/docs/metalctl_machine_issues/index.html index a025d6b024..077534623e 100644 --- a/dev/external/metalctl/docs/metalctl_machine_issues/index.html +++ b/dev/external/metalctl/docs/metalctl_machine_issues/index.html @@ -36,4 +36,4 @@ metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" - --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

+ --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

diff --git a/dev/external/metalctl/docs/metalctl_machine_list/index.html b/dev/external/metalctl/docs/metalctl_machine_list/index.html index fab32d563d..cfa1f17444 100644 --- a/dev/external/metalctl/docs/metalctl_machine_list/index.html +++ b/dev/external/metalctl/docs/metalctl_machine_list/index.html @@ -34,4 +34,4 @@ metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" - --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

+ --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

diff --git a/dev/external/metalctl/docs/metalctl_machine_lock/index.html b/dev/external/metalctl/docs/metalctl_machine_lock/index.html index 79de71d6a9..ef330186a7 100644 --- a/dev/external/metalctl/docs/metalctl_machine_lock/index.html +++ b/dev/external/metalctl/docs/metalctl_machine_lock/index.html @@ -23,4 +23,4 @@ metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" - --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

+ --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

diff --git a/dev/external/metalctl/docs/metalctl_machine_logs/index.html b/dev/external/metalctl/docs/metalctl_machine_logs/index.html index 4377e6f05c..5a1ad2196d 100644 --- a/dev/external/metalctl/docs/metalctl_machine_logs/index.html +++ b/dev/external/metalctl/docs/metalctl_machine_logs/index.html @@ -22,4 +22,4 @@ metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" - --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

+ --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

diff --git a/dev/external/metalctl/docs/metalctl_machine_power/index.html b/dev/external/metalctl/docs/metalctl_machine_power/index.html index 3c1153eded..f226edece0 100644 --- a/dev/external/metalctl/docs/metalctl_machine_power/index.html +++ b/dev/external/metalctl/docs/metalctl_machine_power/index.html @@ -21,4 +21,4 @@ metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" - --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

+ --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

diff --git a/dev/external/metalctl/docs/metalctl_machine_power_bios/index.html b/dev/external/metalctl/docs/metalctl_machine_power_bios/index.html index d55d65f91e..e37e7d2e77 100644 --- a/dev/external/metalctl/docs/metalctl_machine_power_bios/index.html +++ b/dev/external/metalctl/docs/metalctl_machine_power_bios/index.html @@ -21,4 +21,4 @@ metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" - --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

+ --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

diff --git a/dev/external/metalctl/docs/metalctl_machine_power_cycle/index.html b/dev/external/metalctl/docs/metalctl_machine_power_cycle/index.html index 0b41bc688e..670b86d7bd 100644 --- a/dev/external/metalctl/docs/metalctl_machine_power_cycle/index.html +++ b/dev/external/metalctl/docs/metalctl_machine_power_cycle/index.html @@ -21,4 +21,4 @@ metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" - --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

+ --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

diff --git a/dev/external/metalctl/docs/metalctl_machine_power_disk/index.html b/dev/external/metalctl/docs/metalctl_machine_power_disk/index.html index 07d119e7c1..df8a82fd01 100644 --- a/dev/external/metalctl/docs/metalctl_machine_power_disk/index.html +++ b/dev/external/metalctl/docs/metalctl_machine_power_disk/index.html @@ -21,4 +21,4 @@ metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" - --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

+ --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

diff --git a/dev/external/metalctl/docs/metalctl_machine_power_off/index.html b/dev/external/metalctl/docs/metalctl_machine_power_off/index.html index c143a92ab8..3e9a20ca26 100644 --- a/dev/external/metalctl/docs/metalctl_machine_power_off/index.html +++ b/dev/external/metalctl/docs/metalctl_machine_power_off/index.html @@ -21,4 +21,4 @@ metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" - --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

+ --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

diff --git a/dev/external/metalctl/docs/metalctl_machine_power_on/index.html b/dev/external/metalctl/docs/metalctl_machine_power_on/index.html index 817fb7ba4b..65feb0f70b 100644 --- a/dev/external/metalctl/docs/metalctl_machine_power_on/index.html +++ b/dev/external/metalctl/docs/metalctl_machine_power_on/index.html @@ -21,4 +21,4 @@ metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" - --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

+ --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

diff --git a/dev/external/metalctl/docs/metalctl_machine_power_pxe/index.html b/dev/external/metalctl/docs/metalctl_machine_power_pxe/index.html index 0d70721ed2..b12248c085 100644 --- a/dev/external/metalctl/docs/metalctl_machine_power_pxe/index.html +++ b/dev/external/metalctl/docs/metalctl_machine_power_pxe/index.html @@ -21,4 +21,4 @@ metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" - --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

+ --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

diff --git a/dev/external/metalctl/docs/metalctl_machine_power_reset/index.html b/dev/external/metalctl/docs/metalctl_machine_power_reset/index.html index d3e9f4b360..2868994b70 100644 --- a/dev/external/metalctl/docs/metalctl_machine_power_reset/index.html +++ b/dev/external/metalctl/docs/metalctl_machine_power_reset/index.html @@ -21,4 +21,4 @@ metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" - --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

+ --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

diff --git a/dev/external/metalctl/docs/metalctl_machine_reinstall/index.html b/dev/external/metalctl/docs/metalctl_machine_reinstall/index.html index 74b6a9800c..286a5b8870 100644 --- a/dev/external/metalctl/docs/metalctl_machine_reinstall/index.html +++ b/dev/external/metalctl/docs/metalctl_machine_reinstall/index.html @@ -23,4 +23,4 @@ metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" - --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

+ --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

diff --git a/dev/external/metalctl/docs/metalctl_machine_reserve/index.html b/dev/external/metalctl/docs/metalctl_machine_reserve/index.html index 7e26d70d5f..21f789f1ce 100644 --- a/dev/external/metalctl/docs/metalctl_machine_reserve/index.html +++ b/dev/external/metalctl/docs/metalctl_machine_reserve/index.html @@ -23,4 +23,4 @@ metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" - --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

+ --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

diff --git a/dev/external/metalctl/docs/metalctl_machine_update-firmware/index.html b/dev/external/metalctl/docs/metalctl_machine_update-firmware/index.html index 2d2454015b..e93adacc7f 100644 --- a/dev/external/metalctl/docs/metalctl_machine_update-firmware/index.html +++ b/dev/external/metalctl/docs/metalctl_machine_update-firmware/index.html @@ -21,4 +21,4 @@ metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" - --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

+ --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

diff --git a/dev/external/metalctl/docs/metalctl_machine_update-firmware_bios/index.html b/dev/external/metalctl/docs/metalctl_machine_update-firmware_bios/index.html index 84a78dd8be..7cde2e7db7 100644 --- a/dev/external/metalctl/docs/metalctl_machine_update-firmware_bios/index.html +++ b/dev/external/metalctl/docs/metalctl_machine_update-firmware_bios/index.html @@ -23,4 +23,4 @@ metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" - --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

+ --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

diff --git a/dev/external/metalctl/docs/metalctl_machine_update-firmware_bmc/index.html b/dev/external/metalctl/docs/metalctl_machine_update-firmware_bmc/index.html index 89f5462108..6367d37aa5 100644 --- a/dev/external/metalctl/docs/metalctl_machine_update-firmware_bmc/index.html +++ b/dev/external/metalctl/docs/metalctl_machine_update-firmware_bmc/index.html @@ -23,4 +23,4 @@ metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" - --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

+ --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

diff --git a/dev/external/metalctl/docs/metalctl_machine_update/index.html b/dev/external/metalctl/docs/metalctl_machine_update/index.html index 7d4b65992f..beb3268823 100644 --- a/dev/external/metalctl/docs/metalctl_machine_update/index.html +++ b/dev/external/metalctl/docs/metalctl_machine_update/index.html @@ -34,4 +34,4 @@ metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" - --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

+ --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

diff --git a/dev/external/metalctl/docs/metalctl_markdown/index.html b/dev/external/metalctl/docs/metalctl_markdown/index.html index 00967982b4..1d0be40ecc 100644 --- a/dev/external/metalctl/docs/metalctl_markdown/index.html +++ b/dev/external/metalctl/docs/metalctl_markdown/index.html @@ -21,4 +21,4 @@ metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" - --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

+ --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

diff --git a/dev/external/metalctl/docs/metalctl_network/index.html b/dev/external/metalctl/docs/metalctl_network/index.html index cdbdba9388..e81dc189f5 100644 --- a/dev/external/metalctl/docs/metalctl_network/index.html +++ b/dev/external/metalctl/docs/metalctl_network/index.html @@ -21,4 +21,4 @@ metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" - --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

+ --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

diff --git a/dev/external/metalctl/docs/metalctl_network_allocate/index.html b/dev/external/metalctl/docs/metalctl_network_allocate/index.html index 1cdbf7fece..97c229aa38 100644 --- a/dev/external/metalctl/docs/metalctl_network_allocate/index.html +++ b/dev/external/metalctl/docs/metalctl_network_allocate/index.html @@ -28,4 +28,4 @@ metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" - --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

+ --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

diff --git a/dev/external/metalctl/docs/metalctl_network_apply/index.html b/dev/external/metalctl/docs/metalctl_network_apply/index.html index 970aad01e9..62c7a88028 100644 --- a/dev/external/metalctl/docs/metalctl_network_apply/index.html +++ b/dev/external/metalctl/docs/metalctl_network_apply/index.html @@ -31,4 +31,4 @@ metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" - --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

+ --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

diff --git a/dev/external/metalctl/docs/metalctl_network_create/index.html b/dev/external/metalctl/docs/metalctl_network_create/index.html index b1de9312f6..fb5e66e14f 100644 --- a/dev/external/metalctl/docs/metalctl_network_create/index.html +++ b/dev/external/metalctl/docs/metalctl_network_create/index.html @@ -44,4 +44,4 @@ metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" - --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

+ --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

diff --git a/dev/external/metalctl/docs/metalctl_network_delete/index.html b/dev/external/metalctl/docs/metalctl_network_delete/index.html index 779b50cc58..67c66bc63f 100644 --- a/dev/external/metalctl/docs/metalctl_network_delete/index.html +++ b/dev/external/metalctl/docs/metalctl_network_delete/index.html @@ -21,4 +21,4 @@ metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" - --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

+ --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

diff --git a/dev/external/metalctl/docs/metalctl_network_describe/index.html b/dev/external/metalctl/docs/metalctl_network_describe/index.html index 5aa2810d90..9a440b3eeb 100644 --- a/dev/external/metalctl/docs/metalctl_network_describe/index.html +++ b/dev/external/metalctl/docs/metalctl_network_describe/index.html @@ -21,4 +21,4 @@ metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" - --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

+ --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

diff --git a/dev/external/metalctl/docs/metalctl_network_edit/index.html b/dev/external/metalctl/docs/metalctl_network_edit/index.html index 21c486ad59..75817a78c1 100644 --- a/dev/external/metalctl/docs/metalctl_network_edit/index.html +++ b/dev/external/metalctl/docs/metalctl_network_edit/index.html @@ -21,4 +21,4 @@ metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" - --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

+ --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

diff --git a/dev/external/metalctl/docs/metalctl_network_free/index.html b/dev/external/metalctl/docs/metalctl_network_free/index.html index 7c6ca765bd..cc99895baa 100644 --- a/dev/external/metalctl/docs/metalctl_network_free/index.html +++ b/dev/external/metalctl/docs/metalctl_network_free/index.html @@ -21,4 +21,4 @@ metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" - --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

+ --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

diff --git a/dev/external/metalctl/docs/metalctl_network_ip/index.html b/dev/external/metalctl/docs/metalctl_network_ip/index.html index 1742b7f495..b57a558251 100644 --- a/dev/external/metalctl/docs/metalctl_network_ip/index.html +++ b/dev/external/metalctl/docs/metalctl_network_ip/index.html @@ -21,4 +21,4 @@ metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" - --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

+ --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

diff --git a/dev/external/metalctl/docs/metalctl_network_ip_apply/index.html b/dev/external/metalctl/docs/metalctl_network_ip_apply/index.html index 057edfa02b..d9d20e0a73 100644 --- a/dev/external/metalctl/docs/metalctl_network_ip_apply/index.html +++ b/dev/external/metalctl/docs/metalctl_network_ip_apply/index.html @@ -31,4 +31,4 @@ metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" - --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

+ --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

diff --git a/dev/external/metalctl/docs/metalctl_network_ip_create/index.html b/dev/external/metalctl/docs/metalctl_network_ip_create/index.html index 60d0e4f90c..42e4ecf0db 100644 --- a/dev/external/metalctl/docs/metalctl_network_ip_create/index.html +++ b/dev/external/metalctl/docs/metalctl_network_ip_create/index.html @@ -38,4 +38,4 @@ metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" - --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

+ --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

diff --git a/dev/external/metalctl/docs/metalctl_network_ip_delete/index.html b/dev/external/metalctl/docs/metalctl_network_ip_delete/index.html index 6e579a0fae..9a1912cb33 100644 --- a/dev/external/metalctl/docs/metalctl_network_ip_delete/index.html +++ b/dev/external/metalctl/docs/metalctl_network_ip_delete/index.html @@ -21,4 +21,4 @@ metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" - --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

+ --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

diff --git a/dev/external/metalctl/docs/metalctl_network_ip_describe/index.html b/dev/external/metalctl/docs/metalctl_network_ip_describe/index.html index dccc3bfac6..23a68288af 100644 --- a/dev/external/metalctl/docs/metalctl_network_ip_describe/index.html +++ b/dev/external/metalctl/docs/metalctl_network_ip_describe/index.html @@ -21,4 +21,4 @@ metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" - --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

+ --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

diff --git a/dev/external/metalctl/docs/metalctl_network_ip_edit/index.html b/dev/external/metalctl/docs/metalctl_network_ip_edit/index.html index 97cc3b294d..1406324a9c 100644 --- a/dev/external/metalctl/docs/metalctl_network_ip_edit/index.html +++ b/dev/external/metalctl/docs/metalctl_network_ip_edit/index.html @@ -21,4 +21,4 @@ metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" - --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

+ --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

diff --git a/dev/external/metalctl/docs/metalctl_network_ip_issues/index.html b/dev/external/metalctl/docs/metalctl_network_ip_issues/index.html index efae24f8d3..0d06e7360d 100644 --- a/dev/external/metalctl/docs/metalctl_network_ip_issues/index.html +++ b/dev/external/metalctl/docs/metalctl_network_ip_issues/index.html @@ -21,4 +21,4 @@ metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" - --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

+ --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

diff --git a/dev/external/metalctl/docs/metalctl_network_ip_list/index.html b/dev/external/metalctl/docs/metalctl_network_ip_list/index.html index 4719748717..d250b2a610 100644 --- a/dev/external/metalctl/docs/metalctl_network_ip_list/index.html +++ b/dev/external/metalctl/docs/metalctl_network_ip_list/index.html @@ -30,4 +30,4 @@ metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" - --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

+ --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

diff --git a/dev/external/metalctl/docs/metalctl_network_ip_update/index.html b/dev/external/metalctl/docs/metalctl_network_ip_update/index.html index 0904953995..f856a0c20c 100644 --- a/dev/external/metalctl/docs/metalctl_network_ip_update/index.html +++ b/dev/external/metalctl/docs/metalctl_network_ip_update/index.html @@ -31,4 +31,4 @@ metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" - --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

+ --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

diff --git a/dev/external/metalctl/docs/metalctl_network_list/index.html b/dev/external/metalctl/docs/metalctl_network_list/index.html index 96035866a3..05fd1ae12d 100644 --- a/dev/external/metalctl/docs/metalctl_network_list/index.html +++ b/dev/external/metalctl/docs/metalctl_network_list/index.html @@ -33,4 +33,4 @@ metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" - --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

+ --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

diff --git a/dev/external/metalctl/docs/metalctl_network_update/index.html b/dev/external/metalctl/docs/metalctl_network_update/index.html index d7b27c0c40..d7ad06e637 100644 --- a/dev/external/metalctl/docs/metalctl_network_update/index.html +++ b/dev/external/metalctl/docs/metalctl_network_update/index.html @@ -39,4 +39,4 @@ metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" - --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

+ --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

diff --git a/dev/external/metalctl/docs/metalctl_partition/index.html b/dev/external/metalctl/docs/metalctl_partition/index.html index b71e5c1df9..fcbd911699 100644 --- a/dev/external/metalctl/docs/metalctl_partition/index.html +++ b/dev/external/metalctl/docs/metalctl_partition/index.html @@ -21,4 +21,4 @@ metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" - --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

+ --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

diff --git a/dev/external/metalctl/docs/metalctl_partition_apply/index.html b/dev/external/metalctl/docs/metalctl_partition_apply/index.html index b8281c56a6..04d07ea225 100644 --- a/dev/external/metalctl/docs/metalctl_partition_apply/index.html +++ b/dev/external/metalctl/docs/metalctl_partition_apply/index.html @@ -31,4 +31,4 @@ metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" - --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

+ --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

diff --git a/dev/external/metalctl/docs/metalctl_partition_capacity/index.html b/dev/external/metalctl/docs/metalctl_partition_capacity/index.html index aaab6a9686..c2f61b4edb 100644 --- a/dev/external/metalctl/docs/metalctl_partition_capacity/index.html +++ b/dev/external/metalctl/docs/metalctl_partition_capacity/index.html @@ -24,4 +24,4 @@ metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" - --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

+ --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

diff --git a/dev/external/metalctl/docs/metalctl_partition_create/index.html b/dev/external/metalctl/docs/metalctl_partition_create/index.html index 6bd413422e..8c95241c86 100644 --- a/dev/external/metalctl/docs/metalctl_partition_create/index.html +++ b/dev/external/metalctl/docs/metalctl_partition_create/index.html @@ -38,4 +38,4 @@ metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" - --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

+ --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

diff --git a/dev/external/metalctl/docs/metalctl_partition_delete/index.html b/dev/external/metalctl/docs/metalctl_partition_delete/index.html index 3db8aa5e7e..233f0b5275 100644 --- a/dev/external/metalctl/docs/metalctl_partition_delete/index.html +++ b/dev/external/metalctl/docs/metalctl_partition_delete/index.html @@ -21,4 +21,4 @@ metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" - --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

+ --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

diff --git a/dev/external/metalctl/docs/metalctl_partition_describe/index.html b/dev/external/metalctl/docs/metalctl_partition_describe/index.html index a9ed718f8f..11dec3609e 100644 --- a/dev/external/metalctl/docs/metalctl_partition_describe/index.html +++ b/dev/external/metalctl/docs/metalctl_partition_describe/index.html @@ -21,4 +21,4 @@ metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" - --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

+ --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

diff --git a/dev/external/metalctl/docs/metalctl_partition_edit/index.html b/dev/external/metalctl/docs/metalctl_partition_edit/index.html index 1028c0c556..f64ccd1f66 100644 --- a/dev/external/metalctl/docs/metalctl_partition_edit/index.html +++ b/dev/external/metalctl/docs/metalctl_partition_edit/index.html @@ -21,4 +21,4 @@ metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" - --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

+ --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

diff --git a/dev/external/metalctl/docs/metalctl_partition_list/index.html b/dev/external/metalctl/docs/metalctl_partition_list/index.html index 7a944ddf4a..f49da30273 100644 --- a/dev/external/metalctl/docs/metalctl_partition_list/index.html +++ b/dev/external/metalctl/docs/metalctl_partition_list/index.html @@ -22,4 +22,4 @@ metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" - --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

+ --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

diff --git a/dev/external/metalctl/docs/metalctl_partition_update/index.html b/dev/external/metalctl/docs/metalctl_partition_update/index.html index c50da7b39f..ae4fb2ee75 100644 --- a/dev/external/metalctl/docs/metalctl_partition_update/index.html +++ b/dev/external/metalctl/docs/metalctl_partition_update/index.html @@ -31,4 +31,4 @@ metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" - --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

+ --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

diff --git a/dev/external/metalctl/docs/metalctl_project/index.html b/dev/external/metalctl/docs/metalctl_project/index.html index d43d731818..de3e844b9d 100644 --- a/dev/external/metalctl/docs/metalctl_project/index.html +++ b/dev/external/metalctl/docs/metalctl_project/index.html @@ -21,4 +21,4 @@ metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" - --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

+ --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

diff --git a/dev/external/metalctl/docs/metalctl_project_apply/index.html b/dev/external/metalctl/docs/metalctl_project_apply/index.html index 1db845421c..afc7998b9d 100644 --- a/dev/external/metalctl/docs/metalctl_project_apply/index.html +++ b/dev/external/metalctl/docs/metalctl_project_apply/index.html @@ -31,4 +31,4 @@ metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" - --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

+ --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

diff --git a/dev/external/metalctl/docs/metalctl_project_create/index.html b/dev/external/metalctl/docs/metalctl_project_create/index.html index b79fdc4745..51617c3393 100644 --- a/dev/external/metalctl/docs/metalctl_project_create/index.html +++ b/dev/external/metalctl/docs/metalctl_project_create/index.html @@ -39,4 +39,4 @@ metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" - --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

+ --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

diff --git a/dev/external/metalctl/docs/metalctl_project_delete/index.html b/dev/external/metalctl/docs/metalctl_project_delete/index.html index 8c1dce0da5..b0e0427f46 100644 --- a/dev/external/metalctl/docs/metalctl_project_delete/index.html +++ b/dev/external/metalctl/docs/metalctl_project_delete/index.html @@ -21,4 +21,4 @@ metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" - --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

+ --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

diff --git a/dev/external/metalctl/docs/metalctl_project_describe/index.html b/dev/external/metalctl/docs/metalctl_project_describe/index.html index 35faf6bbb0..4c0f61acd6 100644 --- a/dev/external/metalctl/docs/metalctl_project_describe/index.html +++ b/dev/external/metalctl/docs/metalctl_project_describe/index.html @@ -21,4 +21,4 @@ metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" - --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

+ --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

diff --git a/dev/external/metalctl/docs/metalctl_project_edit/index.html b/dev/external/metalctl/docs/metalctl_project_edit/index.html index 7621101d56..54fa8248c8 100644 --- a/dev/external/metalctl/docs/metalctl_project_edit/index.html +++ b/dev/external/metalctl/docs/metalctl_project_edit/index.html @@ -21,4 +21,4 @@ metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" - --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

+ --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

diff --git a/dev/external/metalctl/docs/metalctl_project_list/index.html b/dev/external/metalctl/docs/metalctl_project_list/index.html index c258d5aa27..a00c114713 100644 --- a/dev/external/metalctl/docs/metalctl_project_list/index.html +++ b/dev/external/metalctl/docs/metalctl_project_list/index.html @@ -25,4 +25,4 @@ metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" - --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

+ --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

diff --git a/dev/external/metalctl/docs/metalctl_project_update/index.html b/dev/external/metalctl/docs/metalctl_project_update/index.html index f05dc9ca99..c5c5e24acd 100644 --- a/dev/external/metalctl/docs/metalctl_project_update/index.html +++ b/dev/external/metalctl/docs/metalctl_project_update/index.html @@ -31,4 +31,4 @@ metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" - --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

+ --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

diff --git a/dev/external/metalctl/docs/metalctl_size/index.html b/dev/external/metalctl/docs/metalctl_size/index.html index 22f846bde7..bed62efc2c 100644 --- a/dev/external/metalctl/docs/metalctl_size/index.html +++ b/dev/external/metalctl/docs/metalctl_size/index.html @@ -21,4 +21,4 @@ metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" - --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

+ --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

diff --git a/dev/external/metalctl/docs/metalctl_size_apply/index.html b/dev/external/metalctl/docs/metalctl_size_apply/index.html index 130acb3829..0ad9464212 100644 --- a/dev/external/metalctl/docs/metalctl_size_apply/index.html +++ b/dev/external/metalctl/docs/metalctl_size_apply/index.html @@ -31,4 +31,4 @@ metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" - --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

+ --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

diff --git a/dev/external/metalctl/docs/metalctl_size_create/index.html b/dev/external/metalctl/docs/metalctl_size_create/index.html index 1da347e6ff..8d0e6b8655 100644 --- a/dev/external/metalctl/docs/metalctl_size_create/index.html +++ b/dev/external/metalctl/docs/metalctl_size_create/index.html @@ -37,4 +37,4 @@ metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" - --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

+ --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

diff --git a/dev/external/metalctl/docs/metalctl_size_delete/index.html b/dev/external/metalctl/docs/metalctl_size_delete/index.html index 011c5bc5e4..d0c3a7d8a5 100644 --- a/dev/external/metalctl/docs/metalctl_size_delete/index.html +++ b/dev/external/metalctl/docs/metalctl_size_delete/index.html @@ -21,4 +21,4 @@ metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" - --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

+ --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

diff --git a/dev/external/metalctl/docs/metalctl_size_describe/index.html b/dev/external/metalctl/docs/metalctl_size_describe/index.html index 78d510b44e..b10894be2b 100644 --- a/dev/external/metalctl/docs/metalctl_size_describe/index.html +++ b/dev/external/metalctl/docs/metalctl_size_describe/index.html @@ -21,4 +21,4 @@ metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" - --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

+ --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

diff --git a/dev/external/metalctl/docs/metalctl_size_edit/index.html b/dev/external/metalctl/docs/metalctl_size_edit/index.html index 6838e525c3..dde3a0e021 100644 --- a/dev/external/metalctl/docs/metalctl_size_edit/index.html +++ b/dev/external/metalctl/docs/metalctl_size_edit/index.html @@ -21,4 +21,4 @@ metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" - --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

+ --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

diff --git a/dev/external/metalctl/docs/metalctl_size_imageconstraint/index.html b/dev/external/metalctl/docs/metalctl_size_imageconstraint/index.html index a738c27633..987b5f7309 100644 --- a/dev/external/metalctl/docs/metalctl_size_imageconstraint/index.html +++ b/dev/external/metalctl/docs/metalctl_size_imageconstraint/index.html @@ -21,4 +21,4 @@ metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" - --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

+ --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

diff --git a/dev/external/metalctl/docs/metalctl_size_imageconstraint_apply/index.html b/dev/external/metalctl/docs/metalctl_size_imageconstraint_apply/index.html index 35590bef3b..5518cb873d 100644 --- a/dev/external/metalctl/docs/metalctl_size_imageconstraint_apply/index.html +++ b/dev/external/metalctl/docs/metalctl_size_imageconstraint_apply/index.html @@ -31,4 +31,4 @@ metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" - --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

+ --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

diff --git a/dev/external/metalctl/docs/metalctl_size_imageconstraint_create/index.html b/dev/external/metalctl/docs/metalctl_size_imageconstraint_create/index.html index 48d5691afc..37d4e600f3 100644 --- a/dev/external/metalctl/docs/metalctl_size_imageconstraint_create/index.html +++ b/dev/external/metalctl/docs/metalctl_size_imageconstraint_create/index.html @@ -31,4 +31,4 @@ metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" - --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

+ --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

diff --git a/dev/external/metalctl/docs/metalctl_size_imageconstraint_delete/index.html b/dev/external/metalctl/docs/metalctl_size_imageconstraint_delete/index.html index 09e16711ae..ff28d08d23 100644 --- a/dev/external/metalctl/docs/metalctl_size_imageconstraint_delete/index.html +++ b/dev/external/metalctl/docs/metalctl_size_imageconstraint_delete/index.html @@ -21,4 +21,4 @@ metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" - --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

+ --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

diff --git a/dev/external/metalctl/docs/metalctl_size_imageconstraint_describe/index.html b/dev/external/metalctl/docs/metalctl_size_imageconstraint_describe/index.html index 2921acefad..5ab7623f17 100644 --- a/dev/external/metalctl/docs/metalctl_size_imageconstraint_describe/index.html +++ b/dev/external/metalctl/docs/metalctl_size_imageconstraint_describe/index.html @@ -21,4 +21,4 @@ metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" - --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

+ --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

diff --git a/dev/external/metalctl/docs/metalctl_size_imageconstraint_edit/index.html b/dev/external/metalctl/docs/metalctl_size_imageconstraint_edit/index.html index c6c8d53f2b..0302218261 100644 --- a/dev/external/metalctl/docs/metalctl_size_imageconstraint_edit/index.html +++ b/dev/external/metalctl/docs/metalctl_size_imageconstraint_edit/index.html @@ -21,4 +21,4 @@ metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" - --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

+ --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

diff --git a/dev/external/metalctl/docs/metalctl_size_imageconstraint_list/index.html b/dev/external/metalctl/docs/metalctl_size_imageconstraint_list/index.html index 70eaf3dd38..587dbd4a41 100644 --- a/dev/external/metalctl/docs/metalctl_size_imageconstraint_list/index.html +++ b/dev/external/metalctl/docs/metalctl_size_imageconstraint_list/index.html @@ -22,4 +22,4 @@ metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" - --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

+ --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

diff --git a/dev/external/metalctl/docs/metalctl_size_imageconstraint_try/index.html b/dev/external/metalctl/docs/metalctl_size_imageconstraint_try/index.html index 9d829c8502..50a2f48a64 100644 --- a/dev/external/metalctl/docs/metalctl_size_imageconstraint_try/index.html +++ b/dev/external/metalctl/docs/metalctl_size_imageconstraint_try/index.html @@ -23,4 +23,4 @@ metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" - --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

+ --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

diff --git a/dev/external/metalctl/docs/metalctl_size_imageconstraint_update/index.html b/dev/external/metalctl/docs/metalctl_size_imageconstraint_update/index.html index 85f8d48fd9..d2f62c6392 100644 --- a/dev/external/metalctl/docs/metalctl_size_imageconstraint_update/index.html +++ b/dev/external/metalctl/docs/metalctl_size_imageconstraint_update/index.html @@ -31,4 +31,4 @@ metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" - --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

+ --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

diff --git a/dev/external/metalctl/docs/metalctl_size_list/index.html b/dev/external/metalctl/docs/metalctl_size_list/index.html index 21c140e864..869a80356b 100644 --- a/dev/external/metalctl/docs/metalctl_size_list/index.html +++ b/dev/external/metalctl/docs/metalctl_size_list/index.html @@ -22,4 +22,4 @@ metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" - --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

+ --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

diff --git a/dev/external/metalctl/docs/metalctl_size_try/index.html b/dev/external/metalctl/docs/metalctl_size_try/index.html index 38d112d43d..96f8b344b9 100644 --- a/dev/external/metalctl/docs/metalctl_size_try/index.html +++ b/dev/external/metalctl/docs/metalctl_size_try/index.html @@ -24,4 +24,4 @@ metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" - --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

+ --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

diff --git a/dev/external/metalctl/docs/metalctl_size_update/index.html b/dev/external/metalctl/docs/metalctl_size_update/index.html index 6eba3b0a99..04aa9f7a22 100644 --- a/dev/external/metalctl/docs/metalctl_size_update/index.html +++ b/dev/external/metalctl/docs/metalctl_size_update/index.html @@ -31,4 +31,4 @@ metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" - --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

+ --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

diff --git a/dev/external/metalctl/docs/metalctl_switch/index.html b/dev/external/metalctl/docs/metalctl_switch/index.html index c0b0d2dd73..235d77b514 100644 --- a/dev/external/metalctl/docs/metalctl_switch/index.html +++ b/dev/external/metalctl/docs/metalctl_switch/index.html @@ -21,4 +21,4 @@ metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" - --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

+ --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

diff --git a/dev/external/metalctl/docs/metalctl_switch_connected-machines/index.html b/dev/external/metalctl/docs/metalctl_switch_connected-machines/index.html index c93a856eef..e1d9ac6898 100644 --- a/dev/external/metalctl/docs/metalctl_switch_connected-machines/index.html +++ b/dev/external/metalctl/docs/metalctl_switch_connected-machines/index.html @@ -36,4 +36,4 @@ metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" - --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

+ --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

diff --git a/dev/external/metalctl/docs/metalctl_switch_console/index.html b/dev/external/metalctl/docs/metalctl_switch_console/index.html index abb0f0430a..068a07b057 100644 --- a/dev/external/metalctl/docs/metalctl_switch_console/index.html +++ b/dev/external/metalctl/docs/metalctl_switch_console/index.html @@ -21,4 +21,4 @@ metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" - --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

+ --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

diff --git a/dev/external/metalctl/docs/metalctl_switch_delete/index.html b/dev/external/metalctl/docs/metalctl_switch_delete/index.html index e8a8e004d0..c39c6de9a8 100644 --- a/dev/external/metalctl/docs/metalctl_switch_delete/index.html +++ b/dev/external/metalctl/docs/metalctl_switch_delete/index.html @@ -21,4 +21,4 @@ metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" - --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

+ --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

diff --git a/dev/external/metalctl/docs/metalctl_switch_describe/index.html b/dev/external/metalctl/docs/metalctl_switch_describe/index.html index 713d5f75a7..1ce71b1eee 100644 --- a/dev/external/metalctl/docs/metalctl_switch_describe/index.html +++ b/dev/external/metalctl/docs/metalctl_switch_describe/index.html @@ -21,4 +21,4 @@ metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" - --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

+ --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

diff --git a/dev/external/metalctl/docs/metalctl_switch_detail/index.html b/dev/external/metalctl/docs/metalctl_switch_detail/index.html index 317787285d..1ea85eba52 100644 --- a/dev/external/metalctl/docs/metalctl_switch_detail/index.html +++ b/dev/external/metalctl/docs/metalctl_switch_detail/index.html @@ -27,4 +27,4 @@ metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" - --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

+ --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

diff --git a/dev/external/metalctl/docs/metalctl_switch_edit/index.html b/dev/external/metalctl/docs/metalctl_switch_edit/index.html index 4bf2dccb62..422c71e58e 100644 --- a/dev/external/metalctl/docs/metalctl_switch_edit/index.html +++ b/dev/external/metalctl/docs/metalctl_switch_edit/index.html @@ -21,4 +21,4 @@ metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" - --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

+ --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

diff --git a/dev/external/metalctl/docs/metalctl_switch_list/index.html b/dev/external/metalctl/docs/metalctl_switch_list/index.html index 1f88983b3f..6b3390f19f 100644 --- a/dev/external/metalctl/docs/metalctl_switch_list/index.html +++ b/dev/external/metalctl/docs/metalctl_switch_list/index.html @@ -28,4 +28,4 @@ metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" - --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

+ --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

diff --git a/dev/external/metalctl/docs/metalctl_switch_replace/index.html b/dev/external/metalctl/docs/metalctl_switch_replace/index.html index 9d56611c1e..17bd0bc6a3 100644 --- a/dev/external/metalctl/docs/metalctl_switch_replace/index.html +++ b/dev/external/metalctl/docs/metalctl_switch_replace/index.html @@ -21,4 +21,4 @@ metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" - --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

+ --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

diff --git a/dev/external/metalctl/docs/metalctl_switch_ssh/index.html b/dev/external/metalctl/docs/metalctl_switch_ssh/index.html index b193f1f376..22de72f2d0 100644 --- a/dev/external/metalctl/docs/metalctl_switch_ssh/index.html +++ b/dev/external/metalctl/docs/metalctl_switch_ssh/index.html @@ -21,4 +21,4 @@ metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" - --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

+ --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

diff --git a/dev/external/metalctl/docs/metalctl_switch_update/index.html b/dev/external/metalctl/docs/metalctl_switch_update/index.html index 54fcf33b6e..7c01edd328 100644 --- a/dev/external/metalctl/docs/metalctl_switch_update/index.html +++ b/dev/external/metalctl/docs/metalctl_switch_update/index.html @@ -31,4 +31,4 @@ metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" - --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

+ --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

diff --git a/dev/external/metalctl/docs/metalctl_update/index.html b/dev/external/metalctl/docs/metalctl_update/index.html index 8b8aa3983c..d2e2db50ba 100644 --- a/dev/external/metalctl/docs/metalctl_update/index.html +++ b/dev/external/metalctl/docs/metalctl_update/index.html @@ -21,4 +21,4 @@ metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" - --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

+ --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

diff --git a/dev/external/metalctl/docs/metalctl_update_check/index.html b/dev/external/metalctl/docs/metalctl_update_check/index.html index f664df2a8c..11f2b97a2c 100644 --- a/dev/external/metalctl/docs/metalctl_update_check/index.html +++ b/dev/external/metalctl/docs/metalctl_update_check/index.html @@ -21,4 +21,4 @@ metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" - --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

+ --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

diff --git a/dev/external/metalctl/docs/metalctl_update_do/index.html b/dev/external/metalctl/docs/metalctl_update_do/index.html index 587c6eae7e..8ebc350275 100644 --- a/dev/external/metalctl/docs/metalctl_update_do/index.html +++ b/dev/external/metalctl/docs/metalctl_update_do/index.html @@ -21,4 +21,4 @@ metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" - --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

+ --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

diff --git a/dev/external/metalctl/docs/metalctl_version/index.html b/dev/external/metalctl/docs/metalctl_version/index.html index b6f51e4738..d3cb1706cc 100644 --- a/dev/external/metalctl/docs/metalctl_version/index.html +++ b/dev/external/metalctl/docs/metalctl_version/index.html @@ -21,4 +21,4 @@ metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" - --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

+ --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

diff --git a/dev/external/metalctl/docs/metalctl_vpn/index.html b/dev/external/metalctl/docs/metalctl_vpn/index.html index 7754092ede..7b80494dbf 100644 --- a/dev/external/metalctl/docs/metalctl_vpn/index.html +++ b/dev/external/metalctl/docs/metalctl_vpn/index.html @@ -21,4 +21,4 @@ metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" - --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

+ --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

diff --git a/dev/external/metalctl/docs/metalctl_vpn_key/index.html b/dev/external/metalctl/docs/metalctl_vpn_key/index.html index 9d87ca228f..fcd6781705 100644 --- a/dev/external/metalctl/docs/metalctl_vpn_key/index.html +++ b/dev/external/metalctl/docs/metalctl_vpn_key/index.html @@ -26,4 +26,4 @@ metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" - --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

+ --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

diff --git a/dev/external/metalctl/docs/metalctl_whoami/index.html b/dev/external/metalctl/docs/metalctl_whoami/index.html index 9d0b1845d2..eb17322808 100644 --- a/dev/external/metalctl/docs/metalctl_whoami/index.html +++ b/dev/external/metalctl/docs/metalctl_whoami/index.html @@ -21,4 +21,4 @@ metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" - --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

+ --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

diff --git a/dev/external/mini-lab/CONTRIBUTING/index.html b/dev/external/mini-lab/CONTRIBUTING/index.html index 217612418d..9f01a7689d 100644 --- a/dev/external/mini-lab/CONTRIBUTING/index.html +++ b/dev/external/mini-lab/CONTRIBUTING/index.html @@ -1,2 +1,2 @@ -Contributing · metal-stack
Edit on GitHub
+Contributing · metal-stack
Edit on GitHub
diff --git a/dev/external/mini-lab/README/index.html b/dev/external/mini-lab/README/index.html index eb74a4bd3f..fef9b48e69 100644 --- a/dev/external/mini-lab/README/index.html +++ b/dev/external/mini-lab/README/index.html @@ -75,4 +75,4 @@ ssh metal@100.255.254.1

To remove the kind cluster, the switches and machines, run:

make cleanup

Reinstall machine

Reinstall a machine with

docker-compose run --rm metalctl machine reinstall \
         --image ubuntu-20.04 \
         e0ab02d2-27cd-5a5e-8efc-080ba80cf258

Free machine

Free a machine with make free-machine01 or

docker-compose run --rm metalctl machine rm e0ab02d2-27cd-5a5e-8efc-080ba80cf258

Flavors

There's few versions of mini-lab environment that you can run. We call them flavors. There's 2 flavors at the moment:

In order to start specific flavor, you can define the flavor as follows:

export MINI_LAB_FLAVOR=cluster-api
-make

Page Tree

+make

Page Tree

diff --git a/dev/index.html b/dev/index.html index deb41a9f53..b2ac8fc923 100644 --- a/dev/index.html +++ b/dev/index.html @@ -1,2 +1,2 @@ -Introduction · metal-stack
Edit on GitHub

Welcome to the metal-stack docs!

metal-stack is an open source software that provides an API for provisioning and managing physical servers in the data center. To categorize this product, we use the terms Metal-as-a-Service (MaaS) or bare metal cloud.

From the perspective of a user, the metal-stack does not feel any different from working with a conventional cloud provider. Users manage their resources (machines, networks and ip addresses, etc.) by themselves, which effectively turns your data center into an elastic cloud infrastructure.

The major difference to other cloud providers is that compute power and data reside in your own data center.

Releases

Your are currently reading the documentation for the metal-stack master release.

Releases and integration tests are published through our release repository. You can also find the release notes for this metal-stack version in there. The release notes contain information about new features, upgrade paths and bug fixes.

If you want, you can sign up at our Slack channel where we are announcing every new release. Often, we provide additional information for metal-stack administrators and adopters at this place, too.

Why metal-stack?

Before we started with our mission to implement the metal-stack, we decided on a couple of key characteristics and constraints that we think are unique in the domain (otherwise we would definitely have chosen an existing solution).

We hope that the following properties appeal to you as well.

On-Premise

Running on-premise gives you data sovereignty and usually a better price / performance ratio than with hyperscalers — especially the larger you grow your environment. Another benefit of running on-premise is an easier connectivity to existing company networks.

Fast Provisioning

Provisioning bare metal machines should not feel much different from virtual machines. metal-stack is capable of provisioning servers in less than a minute. The underlying network topology is based on BGP and allows announcing new routes to your host machines in a matter of seconds.

No-Ops

Part of the metal-stack runs on dedicated switches in your data center. This way, it is possible to automate server inventorization, permanently reconcile network configuration and automatically manage machine lifecycles. Manual configuration is neither required nor wanted.

Security

Our networking approach was designed for highest standards on security. Also, we enforce firewalling on dedicated tenant firewalls before users can establish connections to other networks than their private tenant network. API authentication and authorization is done with the help of OIDC.

API driven

The development of metal-stack is strictly API driven and offers self-service to end-users. This approach delivers the highest possible degree of automation, maintainability and performance.

Ready for Kubernetes

Not only does the metal-stack run smoothly on Kubernetes (K8s). The major intent of metal-stack has always been to build a scalable machine infrastructure for Kubernetes as a Service (KaaS). In partnership with the open-source project Gardener, we can provision Kubernetes clusters on metal-stack at scale.

From the perspective of the Gardener, the metal-stack is just another cloud provider. The time savings compared to providing machines and Kubernetes by hand are significant. We actually want to be able to compete with offers of public cloud providers, especially regarding speed and usability.

Of course, you can use metal-stack only for machine provisioning as well and just put something else on top of your metal infrastructure.

Open Source

The metal-stack is open source and free of constraints regarding vendors and third-party products. The stack is completely built on open source products. We have a community actively working on the metal-stack, which can assist you delivering all reasonable features you are gonna need.

Why Bare Metal?

Bare metal has several advantages over virtual environments and overcomes several drawbacks of virtual machines. We also listed drawbacks of the bare metal approach. Bare in mind though that it is still possible to virtualize on bare metal environments when you have your stack up and running.

Virtual Environment Drawbacks

  • Spectre and Meltdown can only be mitigated with a "cluster per tenant" approach
  • Missing isolation of multi-tenant change impacts
  • Licensing restrictions
  • Noisy-neighbors

Bare Metal Advantages

  • Guaranteed and fastest possible performance (especially disk i/o)
  • Reduced stack depth (Host / VM / Application vs. Host / Container)
    • Reduced attack surface
    • Lower costs, higher performance
    • No VM live-migrations
  • Bigger hardware configurations possible (hypervisors have restrictions, e.g. it is not possible to assign all CPUs to a single VM)

Bare Metal Drawbacks

  • Hardware defects have direct impact (should be considered by design) and can not be mitigated by live-migration as in virtual environments
  • Capacity planning is more difficult (no resource overbooking possible)
+Introduction · metal-stack
Edit on GitHub

Welcome to the metal-stack docs!

metal-stack is an open source software that provides an API for provisioning and managing physical servers in the data center. To categorize this product, we use the terms Metal-as-a-Service (MaaS) or bare metal cloud.

From the perspective of a user, the metal-stack does not feel any different from working with a conventional cloud provider. Users manage their resources (machines, networks and ip addresses, etc.) by themselves, which effectively turns your data center into an elastic cloud infrastructure.

The major difference to other cloud providers is that compute power and data reside in your own data center.

Releases

Your are currently reading the documentation for the metal-stack master release.

Releases and integration tests are published through our release repository. You can also find the release notes for this metal-stack version in there. The release notes contain information about new features, upgrade paths and bug fixes.

If you want, you can sign up at our Slack channel where we are announcing every new release. Often, we provide additional information for metal-stack administrators and adopters at this place, too.

Why metal-stack?

Before we started with our mission to implement the metal-stack, we decided on a couple of key characteristics and constraints that we think are unique in the domain (otherwise we would definitely have chosen an existing solution).

We hope that the following properties appeal to you as well.

On-Premise

Running on-premise gives you data sovereignty and usually a better price / performance ratio than with hyperscalers — especially the larger you grow your environment. Another benefit of running on-premise is an easier connectivity to existing company networks.

Fast Provisioning

Provisioning bare metal machines should not feel much different from virtual machines. metal-stack is capable of provisioning servers in less than a minute. The underlying network topology is based on BGP and allows announcing new routes to your host machines in a matter of seconds.

No-Ops

Part of the metal-stack runs on dedicated switches in your data center. This way, it is possible to automate server inventorization, permanently reconcile network configuration and automatically manage machine lifecycles. Manual configuration is neither required nor wanted.

Security

Our networking approach was designed for highest standards on security. Also, we enforce firewalling on dedicated tenant firewalls before users can establish connections to other networks than their private tenant network. API authentication and authorization is done with the help of OIDC.

API driven

The development of metal-stack is strictly API driven and offers self-service to end-users. This approach delivers the highest possible degree of automation, maintainability and performance.

Ready for Kubernetes

Not only does the metal-stack run smoothly on Kubernetes (K8s). The major intent of metal-stack has always been to build a scalable machine infrastructure for Kubernetes as a Service (KaaS). In partnership with the open-source project Gardener, we can provision Kubernetes clusters on metal-stack at scale.

From the perspective of the Gardener, the metal-stack is just another cloud provider. The time savings compared to providing machines and Kubernetes by hand are significant. We actually want to be able to compete with offers of public cloud providers, especially regarding speed and usability.

Of course, you can use metal-stack only for machine provisioning as well and just put something else on top of your metal infrastructure.

Open Source

The metal-stack is open source and free of constraints regarding vendors and third-party products. The stack is completely built on open source products. We have a community actively working on the metal-stack, which can assist you delivering all reasonable features you are gonna need.

Why Bare Metal?

Bare metal has several advantages over virtual environments and overcomes several drawbacks of virtual machines. We also listed drawbacks of the bare metal approach. Bare in mind though that it is still possible to virtualize on bare metal environments when you have your stack up and running.

Virtual Environment Drawbacks

  • Spectre and Meltdown can only be mitigated with a "cluster per tenant" approach
  • Missing isolation of multi-tenant change impacts
  • Licensing restrictions
  • Noisy-neighbors

Bare Metal Advantages

  • Guaranteed and fastest possible performance (especially disk i/o)
  • Reduced stack depth (Host / VM / Application vs. Host / Container)
    • Reduced attack surface
    • Lower costs, higher performance
    • No VM live-migrations
  • Bigger hardware configurations possible (hypervisors have restrictions, e.g. it is not possible to assign all CPUs to a single VM)

Bare Metal Drawbacks

  • Hardware defects have direct impact (should be considered by design) and can not be mitigated by live-migration as in virtual environments
  • Capacity planning is more difficult (no resource overbooking possible)
diff --git a/dev/installation/deployment/index.html b/dev/installation/deployment/index.html index 812a669378..877d73813f 100644 --- a/dev/installation/deployment/index.html +++ b/dev/installation/deployment/index.html @@ -324,4 +324,4 @@ mode: Always autoDetectionMethod: interface=lo typha: - enabled: false
  • For your seed cluster you will need to provide the provider secret for metal-stack containing the key metalAPIHMac, which is the API HMAC to grant editor access to the metal-api
  • Checkout our current provider configuration for infrastructure and control-plane before deploying your shoot
    • Tip

    We are officially supported by Gardener dashboard. The dashboard can also help you setting up some of the resources mentioned above.

    + enabled: false
  • For your seed cluster you will need to provide the provider secret for metal-stack containing the key metalAPIHMac, which is the API HMAC to grant editor access to the metal-api
  • Checkout our current provider configuration for infrastructure and control-plane before deploying your shoot
    • Tip

    We are officially supported by Gardener dashboard. The dashboard can also help you setting up some of the resources mentioned above.

    diff --git a/dev/installation/monitoring/index.html b/dev/installation/monitoring/index.html index f6e7e38b64..699063cc97 100644 --- a/dev/installation/monitoring/index.html +++ b/dev/installation/monitoring/index.html @@ -1,2 +1,2 @@ -Monitoring · metal-stack
    Edit on GitHub

    Monitoring the metal-stack

    We are currently working on providing the sources of our monitoring deployment for public usage. Please come back later.

    +Monitoring · metal-stack
    Edit on GitHub

    Monitoring the metal-stack

    We are currently working on providing the sources of our monitoring deployment for public usage. Please come back later.

    diff --git a/dev/installation/troubleshoot/index.html b/dev/installation/troubleshoot/index.html index 1888078cbe..62ac01662e 100644 --- a/dev/installation/troubleshoot/index.html +++ b/dev/installation/troubleshoot/index.html @@ -32,4 +32,4 @@ deploy-control-plane | PLAY RECAP ********************************************************************* deploy-control-plane | localhost : ok=29 changed=4 unreachable=0 failed=1 skipped=7 rescued=0 ignored=0 deploy-control-plane | -deploy-control-plane exited with code 2

    Some home routers have a security feature that prevents DNS Servers to resolve anything in the router's local IP range (DNS-Rebind-Protection).

    You need to add an exception for nip.io in your router configuration or add 127.0.0.1 api.172.17.0.1.nip.io to your /etc/hosts.

    FritzBox

    Home Network -> Network -> Network Settings -> Additional Settings -> DNS Rebind Protection -> Host name exceptions -> nip.io

    Operations

    Fixing Machine Issues

    The metalctl machine issues command gives you an overview over machines in your metal-stack environment that are in an unusual state.

    Tip

    Machines that are known not to function properly, should be locked through metalctl machine lock and annotated with a description of the problem. This way, you can mark machine for replacement without being in danger of having a user allocating the faulty machine.

    In the following sections, you can look up the machine issues that are returned by metalctl and find out how to deal with them properly.

    no-partition

    When a machine has no partition, the metal-hammer has not yet registered the machine at the metal-api. Instead, the machine was created through metal-stack's event machinery, which does not have a lot of information about a machine (e.g. a PXE boot event was reported from the pixiecore).

    This can usually happen on the very first boot of a machine and the machine's hardware is not supported by metal-stack, leading to the metal-bmc being unable to report BMC details to the metal-api (a metal-bmc report sets the partition id of a machine) and the metal-hammer not finishing the machine registration phase.

    To resolve this issue, you need to identify the machine in your metal-stack partition that emits PXE boot events and find the reason why it is not properly booting into the metal-hammer. The console logs of this machine should enable you to find out the root cause.

    liveliness-dead

    For machines without an allocation, the metal-hammer consistently reports whether a machine is still being responsive or not. When the liveliness is Dead, there were no events received from this machine for longer than ~5 minutes.

    Reasons for this can be:

    Info

    In order to minimize maintenance overhead, a machine which is dead for longer than an hour will be rebooted through the metal-api.

    In case you want to prevent this action from happening for a machine, you can lock the machine through metalctl machine lock.

    If the machine is dead for a long time and you are sure that it will never come back, you can clean up the machine through metalctl machine rm --remove-from-database.

    liveliness-unknown

    For machines that are allocated by a user, the ownership has gone over to this user and as an operator you cannot access the machine anymore. This makes it harder to detect whether a machine is in a healthy state or not. Typically, all official metal-stack OS images deploy an LLDP daemon, that consistently emits alive messages. These messages are caught by the metal-core and turned into a Phoned Home event. Internally, the metal-api uses these events as an indicator to decide whether the machine is still responsive or not.

    When the LLDP daemon stopped sending packages, the reasons are identical to those of dead machines. However, it's not possible anymore to decide whether the user is responsible for reaching this state or not.

    In most of the cases, there is not much that can be done from the operator's perspective. You will need to wait for the user to report an issue with the machine. When you do support, you can use this issue type to quickly identify this machine.

    failed-machine-reclaim

    If a machine remains in the Phoned Home state without having an allocation, this indicates that the metal-bmc was not able to put the machine back into PXE boot mode after metalctl machine rm. The machine is still running the operating system and it does not return back into the allocatable machine pool. Effectively, you lost a machine in your environment and no-one pays for it. Therefore, you should resolve this issue as soon as possible.

    In bad scenarios, when the machine was a firewall, the machine can still reach the internet through the PXE boot network and also attract traffic, which it cannot route anymore inside the tenant VRF. This can cause traffic loss inside a tenant network.

    In most of the cases, it should be sufficient to run another metalctl machine rm on this machine in order to retry booting into PXE mode. If this still does not succeed, you can boot the machine into the BIOS and manually and change the boot order to PXE boot. This should force booting the metal-hammer again and add the machine back into your pool of allocatable machines.

    For further reference, see metal-api#145.

    crashloop

    Under bad circumstances, a machine diverges from its typical machine lifecycle. When this happens, the internal state-machine of the metal-api detects that the machine reboots unexpectedly during the provisioning phase. It is likely that the machine has entered a crash loop where it PXE boots again and again without the machine ever becoming usable.

    Reasons for this can be:

    Please also consider console logs of the machine for investigating the issue.

    The incomplete cycle count is reset as soon as the machine reaches Phoned Home state or there is a Planned Reboot of the machine (planned reboot is also done by the metal-hammer once a day in order to reboot with the latest version).

    last-event-error

    The machine had an error during the provisioning lifecycle recently or events are arriving out of order at the metal-api. This can be an interesting hint for the operator that something during machine provisioning went wrong. You can look at the error through metalctl machine describe or metalctl machine logs.

    This error will disappear after a certain time period from machine issues. You can still look up the error as described above.

    asn-not-unique

    This issue was introduced by a bug in earlier versions of metal-stack and was fixed in https://github.com/metal-stack/metal-api/pull/105.

    To resolve the issue, you need to recreate the firewalls that use the same ASN.

    bmc-without-mac

    The metal-bmc is responsible to report connection data for the machine's BMC.

    If it's uncapable of discovering this information, your hardware might not be supported. Please investigate the logs of the metal-bmc to find out what's going wrong with this machine.

    bmc-without-ip

    The metal-bmc is responsible to report connection data for the machine's BMC.

    If it's uncapable of discovering this information, your hardware might not be supported. Please investigate the logs of the metal-bmc to find out what's going wrong with this machine.

    bmc-no-distinct-ip

    The metal-bmc is responsible to report connection data for the machine's BMC.

    When there is no distinct IP address for the BMC, it can be that an orphaned machine used this IP in the past. In this case, you need to clean up the orphaned machine through metalctl machine rm --remove-from-database.

    A machine has registered with a different UUID after reboot

    metal-stack heavily relies on steady machine UUIDs as the UUID is the primary key of the machine entity in the metal-api.

    For further reference also see metal-stack/metal-hammer#52.

    Reasons

    There are some scenarios (can be vendor-specific), which can cause a machine UUID to change over time, e.g.:

    Solution

    1. After five minutes, the orphaned machine UUID will be marked dead (💀) because machine events will be sent only to the most recent UUID
    2. Identify the dead machine through metalctl machine ls
    3. Remove the dead machine forcefully with metalctl machine rm --remove-from-database --yes-i-really-mean-it <uuid>
    +deploy-control-plane exited with code 2

    Some home routers have a security feature that prevents DNS Servers to resolve anything in the router's local IP range (DNS-Rebind-Protection).

    You need to add an exception for nip.io in your router configuration or add 127.0.0.1 api.172.17.0.1.nip.io to your /etc/hosts.

    FritzBox

    Home Network -> Network -> Network Settings -> Additional Settings -> DNS Rebind Protection -> Host name exceptions -> nip.io

    Operations

    Fixing Machine Issues

    The metalctl machine issues command gives you an overview over machines in your metal-stack environment that are in an unusual state.

    Tip

    Machines that are known not to function properly, should be locked through metalctl machine lock and annotated with a description of the problem. This way, you can mark machine for replacement without being in danger of having a user allocating the faulty machine.

    In the following sections, you can look up the machine issues that are returned by metalctl and find out how to deal with them properly.

    no-partition

    When a machine has no partition, the metal-hammer has not yet registered the machine at the metal-api. Instead, the machine was created through metal-stack's event machinery, which does not have a lot of information about a machine (e.g. a PXE boot event was reported from the pixiecore).

    This can usually happen on the very first boot of a machine and the machine's hardware is not supported by metal-stack, leading to the metal-bmc being unable to report BMC details to the metal-api (a metal-bmc report sets the partition id of a machine) and the metal-hammer not finishing the machine registration phase.

    To resolve this issue, you need to identify the machine in your metal-stack partition that emits PXE boot events and find the reason why it is not properly booting into the metal-hammer. The console logs of this machine should enable you to find out the root cause.

    liveliness-dead

    For machines without an allocation, the metal-hammer consistently reports whether a machine is still being responsive or not. When the liveliness is Dead, there were no events received from this machine for longer than ~5 minutes.

    Reasons for this can be:

    Info

    In order to minimize maintenance overhead, a machine which is dead for longer than an hour will be rebooted through the metal-api.

    In case you want to prevent this action from happening for a machine, you can lock the machine through metalctl machine lock.

    If the machine is dead for a long time and you are sure that it will never come back, you can clean up the machine through metalctl machine rm --remove-from-database.

    liveliness-unknown

    For machines that are allocated by a user, the ownership has gone over to this user and as an operator you cannot access the machine anymore. This makes it harder to detect whether a machine is in a healthy state or not. Typically, all official metal-stack OS images deploy an LLDP daemon, that consistently emits alive messages. These messages are caught by the metal-core and turned into a Phoned Home event. Internally, the metal-api uses these events as an indicator to decide whether the machine is still responsive or not.

    When the LLDP daemon stopped sending packages, the reasons are identical to those of dead machines. However, it's not possible anymore to decide whether the user is responsible for reaching this state or not.

    In most of the cases, there is not much that can be done from the operator's perspective. You will need to wait for the user to report an issue with the machine. When you do support, you can use this issue type to quickly identify this machine.

    failed-machine-reclaim

    If a machine remains in the Phoned Home state without having an allocation, this indicates that the metal-bmc was not able to put the machine back into PXE boot mode after metalctl machine rm. The machine is still running the operating system and it does not return back into the allocatable machine pool. Effectively, you lost a machine in your environment and no-one pays for it. Therefore, you should resolve this issue as soon as possible.

    In bad scenarios, when the machine was a firewall, the machine can still reach the internet through the PXE boot network and also attract traffic, which it cannot route anymore inside the tenant VRF. This can cause traffic loss inside a tenant network.

    In most of the cases, it should be sufficient to run another metalctl machine rm on this machine in order to retry booting into PXE mode. If this still does not succeed, you can boot the machine into the BIOS and manually and change the boot order to PXE boot. This should force booting the metal-hammer again and add the machine back into your pool of allocatable machines.

    For further reference, see metal-api#145.

    crashloop

    Under bad circumstances, a machine diverges from its typical machine lifecycle. When this happens, the internal state-machine of the metal-api detects that the machine reboots unexpectedly during the provisioning phase. It is likely that the machine has entered a crash loop where it PXE boots again and again without the machine ever becoming usable.

    Reasons for this can be:

    Please also consider console logs of the machine for investigating the issue.

    The incomplete cycle count is reset as soon as the machine reaches Phoned Home state or there is a Planned Reboot of the machine (planned reboot is also done by the metal-hammer once a day in order to reboot with the latest version).

    last-event-error

    The machine had an error during the provisioning lifecycle recently or events are arriving out of order at the metal-api. This can be an interesting hint for the operator that something during machine provisioning went wrong. You can look at the error through metalctl machine describe or metalctl machine logs.

    This error will disappear after a certain time period from machine issues. You can still look up the error as described above.

    asn-not-unique

    This issue was introduced by a bug in earlier versions of metal-stack and was fixed in https://github.com/metal-stack/metal-api/pull/105.

    To resolve the issue, you need to recreate the firewalls that use the same ASN.

    bmc-without-mac

    The metal-bmc is responsible to report connection data for the machine's BMC.

    If it's uncapable of discovering this information, your hardware might not be supported. Please investigate the logs of the metal-bmc to find out what's going wrong with this machine.

    bmc-without-ip

    The metal-bmc is responsible to report connection data for the machine's BMC.

    If it's uncapable of discovering this information, your hardware might not be supported. Please investigate the logs of the metal-bmc to find out what's going wrong with this machine.

    bmc-no-distinct-ip

    The metal-bmc is responsible to report connection data for the machine's BMC.

    When there is no distinct IP address for the BMC, it can be that an orphaned machine used this IP in the past. In this case, you need to clean up the orphaned machine through metalctl machine rm --remove-from-database.

    A machine has registered with a different UUID after reboot

    metal-stack heavily relies on steady machine UUIDs as the UUID is the primary key of the machine entity in the metal-api.

    For further reference also see metal-stack/metal-hammer#52.

    Reasons

    There are some scenarios (can be vendor-specific), which can cause a machine UUID to change over time, e.g.:

    Solution

    1. After five minutes, the orphaned machine UUID will be marked dead (💀) because machine events will be sent only to the most recent UUID
    2. Identify the dead machine through metalctl machine ls
    3. Remove the dead machine forcefully with metalctl machine rm --remove-from-database --yes-i-really-mean-it <uuid>
    diff --git a/dev/overview/architecture/index.html b/dev/overview/architecture/index.html index 1c462c946b..38599c9141 100644 --- a/dev/overview/architecture/index.html +++ b/dev/overview/architecture/index.html @@ -1,4 +1,4 @@ Architecture · metal-stack
    Edit on GitHub

    Architecture

    The metal-stack is a compound of microservices predominantly written in Golang.

    This page gives you an overview over which microservices exist, how they communicate with each other and where they are deployed.

    Target Deployment Platforms

    For our environments, we chose to deploy the metal-stack into a Kubernetes cluster. This means that also our entire installation was developed for metal-stack being run on Kubernetes. Running applications on Kubernetes gives you a lot of benefits regarding ease-of-deployment, scalability, reliability and so on.

    However, very early we decided that we do not want to depend on technical Kubernetes functionality with our software (i.e. we did not implement the stack "kube-native" by using controllers and Kubernetes CRDs and things like that). With the following paragraph we want to point out the reasoning behind this "philosophical" decision that may sound conservative at first glance. But not relying on Kubernetes technology:

    • Makes deployments of the stack without Kubernetes theoretically possible.
      • We believe that cloud providers should be able to act beneath Kubernetes
      • This way it is possible to use metal-stack for providing your own Kubernetes offering without relying on Kubernetes yourself (breaks the chicken-egg problem)
    • Follows an important claim in microservice development: "Be agnostic to your choice of technology"
      • For applications that are purely made for being run on Kubernetes, it does not matter to rely on this technology (we even do the same a lot with our applications that integrate the metal-stack with Gardener) but as soon as you start using things like the underlying reconciliation abilities (which admittedly are fanstatic) you are locking your code into a certain technology
      • We don't know what comes after Kubernetes but we believe that a cloud offering should have the potential to survive a choice of technology
      • By this decision we ensured that we can migrate the stack to another future technology and survive the change

    One more word towards determining the location for your metal control plane: It is not strictly required to run the control plane inside the same data center as your servers. It even makes sense not to do so because this way you can place your control plane and your servers into a different failure domains, which makes your installation more robust to data center meltdown. Externally hosting the control plane brings you up and running quickly plus having the advantage of higher security through geo-distribution.

    Metal Control Plane

    The foundation of the metal-stack is what we call the metal control plane.

    The control plane contains of a couple of essential microservices for the metal-stack including:

    • metal-api The API to manage and control plane resources like machines, switches, operating system images, machine sizes, networks, IP addresses and more. The exposed API is an old-fashioned REST API with different authentication methods. The metal-api stores the state of these entities in a RethinkDB database. The metal-api also has its own IP address management (go-ipam), which writes IP address and network allocations into a PostgreSQL backend.
    • masterdata-api Manages tenant and project entities, which can be described as entities used for company-specific resource separation and grouping. Having these "higher level entities" managed by a separate microservice was a design choice that allows to re-use the information by other microservices without having them to know the metal-api at all. The masterdata gets persisted in a dedicated PostgreSQL database.
    • metal-console Provides access for users to a machine's serial console via SSH. It can be seen as an optional component.
    • nsq A message queuing system (not developed by the metal-stack) used for decoupling microservices and distributing tasks.

    The following figure shows the relationships between these microservices:

    Metal Control Plane

    Figure 1: The metal control plane deployed in a Kubernetes environment with an ingress-controller exposing additional services via service exposal.

    Some notes on this picture:

    • Users can access the metal-api with the CLI client called metalctl.
    • You can programmatically access the metal-api with client libraries (e.g. metal-go).
    • Our databases are wrapped in a specially built backup-restore-sidecar, which is consistently backing up the databases in external blob storage.
    • The metal-api can be scaled out using replicas when being deployed in Kubernetes.

    Partitions

    A partition is our term for describing hardware in the data center controlled by the metal-stack with all the hardware participating in the same network topology. Being in the same network topology causes the hardware inside a partition to build a failure domain. Even though the network topology for running the metal-stack is required to be redundant by design, you should consider setting up multiple partitions. With multiple partitions it is possible for users to maintain availability of their applications by spreading them across the partitions. Installing partitions in multiple data centers would be even better in regards of fail-safe application performance, which would even tolerate the meltdown of a data center.

    Tip

    In our setups, we encode the name of a region and a zone name into our partition names. However, we do not have dedicated entities for regions and zones in our APIs.

    A region is a geographic area in which data centers are located.

    Zones are geographic locations in a region usually in different fire compartments. Regions can consist of several zones.

    A zone can consist of several partitions. Usually, a partition spans a rack or a group of racks.

    We strongly advise to group your hardware into racks that are specifically assembled for running metal-stack. When using modular rack design, the amount of compute resources of a partition can easily be extended by adding more racks to your partition.

    Info

    The hardware that we currently support to be placed inside a partition is described in the hardware document.

    Info

    How large you can grow your partitions and how the network topology inside a partition looks like is described in the networking document.

    The metal-stack has microservices running on the leaf switches in a partition. For this reason, your leaf switches are required to run a Linux distribution that you have full access to. Additionally, there are a servers not added to the pool of user-allocatable machines, which are instead required for running metal-stack and we call them management servers. We also call the entirety of switches inside a partition the switch plane.

    The microservices running inside a partition are:

    • metal-hammer (runs on a server when not allocated by user, often referred to as discovery image) An initrd, which is booted up in PXE mode, preparing and registering a machine. When a user allocates a machine, the metal-hammer will install the target operating system on this machine and kexec into the new operating system kernel.
    • metal-core (runs on leaf switches) Dynamically configures the leaf switch from information provided by the metal-api. It also proxies requests from the metal-hammer to the metal-api including publishment of machine lifecycle events and machine registration requests.
    • pixiecore (preferably runs on management servers, forked by metal-stack) Provides the capability of PXE booting servers in the PXE boot network.
    • metal-bmc (runs on management servers) Reports the ip addresses that are leased to ipmi devices together with their machine uuids to the metal-api. This provides machine discovery in the partition machines and keeps all IPMI interface access data up-to-date. Also forwards metal-console requests to the actual machine, allowing user access to the machine's serial console. Furthermore it processes firmware updates and power on/off, led on/off, boot order changes.

    Partition

    Figure 2: Simplified illustration of services running inside a partition.

    Some notes on this picture:

    • This figure is slightly simplified. The switch plane consists of spine switches, exit routers, management firewalls and a bastion router with more software components deployed on these entities. Please refer to the networking document to see the full overview over the switch plane.
    • The image-cache is an optional component consisting of multiple services to allow caching images from the public image store inside a partition. This brings increased download performance on machine allocation and increases independence of a partition on the internet connection.

    Complete View

    The following figure shows several partitions connected to a single metal control plane. Of course, it is also possible to have multiple metal control planes, which can be useful for staging.

    metal-stack

    Figure 3: Reduced view on the communication between the metal control plane and multiple partitions.

    Some notes on this picture:

    • By design, a partition only has very few ports open for incoming-connections from the internet. This contributes to a smaller attack surface and higher security of your infrastructure.
    • With the help of NSQ, it is not required to have connections from the metal control plane to the metal-core. The metal-core instances register at the message bus and can then consume partition-specfic topics, e.g. when a machine deletion gets issued by a user.

    Machine Provisioning Sequence

    The following sequence diagram illustrates some of the main principles of the machine provisioning lifecycle.

    provisioning sequence

    Figure 4: Sequence diagram of the machine provisioning sequence.

    Here is a video showing a screen capture of a machine's serial console while running the metal-hammer in "wait mode". Then, a user allocates the machine and the metal-hammer installs the target operating system and the machine boots into the new operating system kernel via the kexec system call.

    -
    + diff --git a/dev/overview/comparison/index.html b/dev/overview/comparison/index.html index f66837beb8..ec17a03e3f 100644 --- a/dev/overview/comparison/index.html +++ b/dev/overview/comparison/index.html @@ -1,2 +1,2 @@ -Comparison · metal-stack
    Edit on GitHub

    Comparison with Commercial Solutions

    As metal-stack is the foundation to build Kubernetes clusters on premise on bare metal, there are several commercial solutions available which offer management of Kubernetes. In this document we describe the differences between some of the most popular solutions. It´s is not a complete list.

    Comparison between Gardener on Metal Stack and Openshift running on VMWare.

    Gardener

    Gardener is a Kubernetes cluster manager to organize a fleet of Kubernetes clusters at scale. It is designed to scale to thousands of clusters at a variety of IaaS Providers regardless where - in the cloud or on premise, virtualized or bare metal. It not only manages the creation and deletion of Kubernetes clusters, it also takes care of updating or upgrading Kubernetes and the operating system of the involved worker nodes in a automatic manner. Gardener is designed cloud-native and as such, it defines clusters, workers and all other components as Kubernetes resources (like pods and deployments) and reconciles these resources to the desired state.

    Kubernetes

    Kubernetes is the de facto open-source standard for container scheduling and orchestration in the data center.

    Openshift

    A fork of Kubernetes with proprietary addons, created by RedHat. For all details see: https://en.wikipedia.org/wiki/OpenShift.

    metal-stack

    Is an IaaS provider for bare metal focused to create Kubernetes cluster on premise. Gardener support is built in.

    VMWare

    The most used virtualization technology in the enterprise data centers.

    Comparison of Gardener on Metal Stack vs. Openshift on VMWare

    FeatureGardener on Metal StackOpenshift on VMWare
    Container Runtimedocker, containerd, gvisorcri-o
    Host Operating SystemUbuntu, Debian , also see OSRHEL, Fedora-Core
    Network PluginsCalico, Cilium(soon)Openshift SDN
    StorageLocal NVME, Lightbits NVMEoTCP, all CSI compatible Solutions, also see StorageCSI compatible
    LoadbalancingBGP built inrequires extra HW like F5, VMWare NSX
    IO at Native SpeedPods run on bare metalall IO must go through the Hypervisor
    Hard MultitenancyWorkers, firewall and load balancers are dedicated for every cluster on bare metalShared virtualization hosts, shared load balancers
    UIGardener DashboardOpenshift Console
    Multi-cluster managementYes (through Gardener)Requires extra licences SW: Redhat Advanced Cluster Manager
    Automatic Kubernetes UpdatesYesYes
    Automatic Worker Nodes UpdatesYesYes
    Supported IaaS ProvidersGCP, AWS, Azure, Alibaba, Openstack, VMWare, metal-stack and moreGCP, AWS, Azure Openstack, VMWare
    Monitoring / Logging StackGrafana/Loki, Kibana/ElasticKibana/Elastic
    GitOPSTool of choice via Helm InstallOpenshift GitOPS
    Container Registryall public accessible registries, private deployed registry of choiceall public accessible registries, in cluster registry
    CI/CDTool of choice via Helm InstallJenkins
    SecurityK8s control plane isolated from tenant, PSP enabled by defaultStrong cluster defaults
    CNCF Kubernetes certifiedYes (Gardener)Yes
    Local developmentminikube, kindminishift
    Proprietary extensionsNoDeploymentConfig and others
    kubectl accessYesYes
    +Comparison · metal-stack
    Edit on GitHub

    Comparison with Commercial Solutions

    As metal-stack is the foundation to build Kubernetes clusters on premise on bare metal, there are several commercial solutions available which offer management of Kubernetes. In this document we describe the differences between some of the most popular solutions. It´s is not a complete list.

    Comparison between Gardener on Metal Stack and Openshift running on VMWare.

    Gardener

    Gardener is a Kubernetes cluster manager to organize a fleet of Kubernetes clusters at scale. It is designed to scale to thousands of clusters at a variety of IaaS Providers regardless where - in the cloud or on premise, virtualized or bare metal. It not only manages the creation and deletion of Kubernetes clusters, it also takes care of updating or upgrading Kubernetes and the operating system of the involved worker nodes in a automatic manner. Gardener is designed cloud-native and as such, it defines clusters, workers and all other components as Kubernetes resources (like pods and deployments) and reconciles these resources to the desired state.

    Kubernetes

    Kubernetes is the de facto open-source standard for container scheduling and orchestration in the data center.

    Openshift

    A fork of Kubernetes with proprietary addons, created by RedHat. For all details see: https://en.wikipedia.org/wiki/OpenShift.

    metal-stack

    Is an IaaS provider for bare metal focused to create Kubernetes cluster on premise. Gardener support is built in.

    VMWare

    The most used virtualization technology in the enterprise data centers.

    Comparison of Gardener on Metal Stack vs. Openshift on VMWare

    FeatureGardener on Metal StackOpenshift on VMWare
    Container Runtimedocker, containerd, gvisorcri-o
    Host Operating SystemUbuntu, Debian , also see OSRHEL, Fedora-Core
    Network PluginsCalico, Cilium(soon)Openshift SDN
    StorageLocal NVME, Lightbits NVMEoTCP, all CSI compatible Solutions, also see StorageCSI compatible
    LoadbalancingBGP built inrequires extra HW like F5, VMWare NSX
    IO at Native SpeedPods run on bare metalall IO must go through the Hypervisor
    Hard MultitenancyWorkers, firewall and load balancers are dedicated for every cluster on bare metalShared virtualization hosts, shared load balancers
    UIGardener DashboardOpenshift Console
    Multi-cluster managementYes (through Gardener)Requires extra licences SW: Redhat Advanced Cluster Manager
    Automatic Kubernetes UpdatesYesYes
    Automatic Worker Nodes UpdatesYesYes
    Supported IaaS ProvidersGCP, AWS, Azure, Alibaba, Openstack, VMWare, metal-stack and moreGCP, AWS, Azure Openstack, VMWare
    Monitoring / Logging StackGrafana/Loki, Kibana/ElasticKibana/Elastic
    GitOPSTool of choice via Helm InstallOpenshift GitOPS
    Container Registryall public accessible registries, private deployed registry of choiceall public accessible registries, in cluster registry
    CI/CDTool of choice via Helm InstallJenkins
    SecurityK8s control plane isolated from tenant, PSP enabled by defaultStrong cluster defaults
    CNCF Kubernetes certifiedYes (Gardener)Yes
    Local developmentminikube, kindminishift
    Proprietary extensionsNoDeploymentConfig and others
    kubectl accessYesYes
    diff --git a/dev/overview/hardware/index.html b/dev/overview/hardware/index.html index 95d5bf450b..6c003686ce 100644 --- a/dev/overview/hardware/index.html +++ b/dev/overview/hardware/index.html @@ -1,2 +1,2 @@ -Hardware Support · metal-stack
    Edit on GitHub

    Hardware Support

    In order to keep the automation and maintenance overhead small, we strongly advise against building highly heterogeneous environments with metal-stack. Having a lot of different vendors and server models in your partitions will heavily increase the time and effort for introducing metal-stack in your infrastructure. From experience we can tell that the interfaces for automating hardware provisioning are usually inconsistent between vendors and even between server models of the same vendor. Therefore, we encourage adopters to start off with only a small amount of machine types. If you want to be on the safe side, you should consider buying the hardware that we officially support.

    We came up with a repository called go-hal, which includes the interface required for metal-stack to support a machine vendor. If you plan to implement support for new vendors, please check out this repository and contribute back your efforts in order to make the community benefit from extended vendor support as well.

    Servers

    At the moment we support the following server types:

    VendorSeriesModelBoard TypeStatus
    SupermicroBig-TwinSYS-2029BT-HNRX11DPT-Bstable
    SupermicroSuperServerSSG-5019D8-TR12PX11SDV-8C-TP8Fstable
    SupermicroSuperServer2029UZ-TN20R25MX11DPUstable
    SupermicroMicrocloud5039MD8-H8TNRX11SDD-8C-Fstable
    LenovoThinkSystemSD530alpha

    Switches

    At the moment we support the following switch types:

    VendorSeriesModelOSStatus
    Edge-CoreAS7700 SeriesAS7712-32XCumulus 3.7.13stable
    Edge-CoreAS7700 SeriesAS7726-32XCumulus 4.1.1stable
    Warning

    On our switches we run Cumulus Linux. The metal-core writes network configuration specifically implemented for this operating system. Please also consider running Cumulus Linux on your switches if you do not want to run into any issues with networking.

    Of course, contributions for supporting other switch vendors and operating systems are highly appreciated.

    Portable metal-stack Setup DIY

    A minimal physical hardware setup may contain at least the following components:

    Warning

    This setup should work as the components are very similar to the currently supported ones but it's currently untested.

    #VendorSeriesModelFunction
    2xEdge-CoreAS5500 SeriesAS5512-54x (10G)Leaf / Exit switches
    1xSupermicroMicrocloudSYS-5039MA16-H12RFTUsable machines
    1xUnifiEdgemaxEdgerouter ProFront router for internet and out-of-band access to servers and switches

    Besides that, a 6HE rack with 1000mm depth and a portable LTE modem is needed.

    This MVP will yield in 12 usable machines, one of them will be reserved as management server.

    +Hardware Support · metal-stack
    Edit on GitHub

    Hardware Support

    In order to keep the automation and maintenance overhead small, we strongly advise against building highly heterogeneous environments with metal-stack. Having a lot of different vendors and server models in your partitions will heavily increase the time and effort for introducing metal-stack in your infrastructure. From experience we can tell that the interfaces for automating hardware provisioning are usually inconsistent between vendors and even between server models of the same vendor. Therefore, we encourage adopters to start off with only a small amount of machine types. If you want to be on the safe side, you should consider buying the hardware that we officially support.

    We came up with a repository called go-hal, which includes the interface required for metal-stack to support a machine vendor. If you plan to implement support for new vendors, please check out this repository and contribute back your efforts in order to make the community benefit from extended vendor support as well.

    Servers

    At the moment we support the following server types:

    VendorSeriesModelBoard TypeStatus
    SupermicroBig-TwinSYS-2029BT-HNRX11DPT-Bstable
    SupermicroSuperServerSSG-5019D8-TR12PX11SDV-8C-TP8Fstable
    SupermicroSuperServer2029UZ-TN20R25MX11DPUstable
    SupermicroMicrocloud5039MD8-H8TNRX11SDD-8C-Fstable
    LenovoThinkSystemSD530alpha

    Switches

    At the moment we support the following switch types:

    VendorSeriesModelOSStatus
    Edge-CoreAS7700 SeriesAS7712-32XCumulus 3.7.13stable
    Edge-CoreAS7700 SeriesAS7726-32XCumulus 4.1.1stable
    Warning

    On our switches we run Cumulus Linux. The metal-core writes network configuration specifically implemented for this operating system. Please also consider running Cumulus Linux on your switches if you do not want to run into any issues with networking.

    Of course, contributions for supporting other switch vendors and operating systems are highly appreciated.

    Portable metal-stack Setup DIY

    A minimal physical hardware setup may contain at least the following components:

    Warning

    This setup should work as the components are very similar to the currently supported ones but it's currently untested.

    #VendorSeriesModelFunction
    2xEdge-CoreAS5500 SeriesAS5512-54x (10G)Leaf / Exit switches
    1xSupermicroMicrocloudSYS-5039MA16-H12RFTUsable machines
    1xUnifiEdgemaxEdgerouter ProFront router for internet and out-of-band access to servers and switches

    Besides that, a 6HE rack with 1000mm depth and a portable LTE modem is needed.

    This MVP will yield in 12 usable machines, one of them will be reserved as management server.

    diff --git a/dev/overview/kubernetes/index.html b/dev/overview/kubernetes/index.html index 5f1aae6da4..533a9d9fb0 100644 --- a/dev/overview/kubernetes/index.html +++ b/dev/overview/kubernetes/index.html @@ -1,2 +1,2 @@ -Kubernetes Integration · metal-stack
    Edit on GitHub

    Kubernetes Integration

    With the help of the Gardener project, metal-stack can be used for spinning up Kubernetes clusters quickly and reliably on bare metal machines.

    To make this happen, we implemented a couple of components, which are described here.

    metal-ccm

    CCM stands for cloud-controller-manager and is the bridge between Kubernetes and a cloud-provider.

    We implemented the cloud provider interface in the metal-ccm repository. With the help of the cloud-controller-controller we provide metal-stack-specific properties for Kubernetes clusters, e.g. load balancer configuration through MetalLB or node properties.

    firewall-controller

    To make the firewalls created with metal-stack easily configurable through Kubernetes resources, we add our firewall-controller to the firewall image. The controller watches special CRDs, enabling users to manage:

    • nftables rules
    • Intrusion-detection with suricata
    • network metric collection

    Please check out the guide on how to use it.

    Gardener components

    There are some Gardener resources that need be reconciled when you act as a cloud provider for the Gardener. This section briefly describes the controllers implemented for deploying Kubernetes clusters through Gardener.

    If you want to learn how to deploy metal-stack with Gardener, please check out the installation section.

    gardener-extension-provider-metal

    The gardener-extension-provider-metal contains of a set of webhooks and controllers for reconciling or mutating Gardener-specific resources.

    The project also contains a validator for metal-type Gardener resources, which you should also deploy in case you want to use metal-stack in combination with Gardener.

    os-metal-extension

    Due to the reason we use ignition in our operating system images for userdata, we had to provide an own extension controller for metal-stack, which you can find at Github in the os-metal-extension repository.

    machine-controller-manager-provider-metal

    Worker nodes are managed through Gardener's machine-controller-manager (MCM). The MCM allows out-of-tree provider implementation via sidecar, which is what we implemented in the machine-controller-manager-provider-metal repository.

    +Kubernetes Integration · metal-stack
    Edit on GitHub

    Kubernetes Integration

    With the help of the Gardener project, metal-stack can be used for spinning up Kubernetes clusters quickly and reliably on bare metal machines.

    To make this happen, we implemented a couple of components, which are described here.

    metal-ccm

    CCM stands for cloud-controller-manager and is the bridge between Kubernetes and a cloud-provider.

    We implemented the cloud provider interface in the metal-ccm repository. With the help of the cloud-controller-controller we provide metal-stack-specific properties for Kubernetes clusters, e.g. load balancer configuration through MetalLB or node properties.

    firewall-controller

    To make the firewalls created with metal-stack easily configurable through Kubernetes resources, we add our firewall-controller to the firewall image. The controller watches special CRDs, enabling users to manage:

    • nftables rules
    • Intrusion-detection with suricata
    • network metric collection

    Please check out the guide on how to use it.

    Gardener components

    There are some Gardener resources that need be reconciled when you act as a cloud provider for the Gardener. This section briefly describes the controllers implemented for deploying Kubernetes clusters through Gardener.

    If you want to learn how to deploy metal-stack with Gardener, please check out the installation section.

    gardener-extension-provider-metal

    The gardener-extension-provider-metal contains of a set of webhooks and controllers for reconciling or mutating Gardener-specific resources.

    The project also contains a validator for metal-type Gardener resources, which you should also deploy in case you want to use metal-stack in combination with Gardener.

    os-metal-extension

    Due to the reason we use ignition in our operating system images for userdata, we had to provide an own extension controller for metal-stack, which you can find at Github in the os-metal-extension repository.

    machine-controller-manager-provider-metal

    Worker nodes are managed through Gardener's machine-controller-manager (MCM). The MCM allows out-of-tree provider implementation via sidecar, which is what we implemented in the machine-controller-manager-provider-metal repository.

    diff --git a/dev/overview/networking/index.html b/dev/overview/networking/index.html index e4d4ba9941..b652dc9ef7 100644 --- a/dev/overview/networking/index.html +++ b/dev/overview/networking/index.html @@ -353,4 +353,4 @@ iface swp1 mtu 9000 bridge-access 4000 -# [...]

    Listing 14: VLAN access setup for bare metal server facing ports on leaves.

    Once a bare metal server is provisioned it is deconfigured from PXE VLAN vlan4000 to avoid accidential or unwanted provisioning.

    During provisioning bare metal servers get internet access via the management network of the exit switches. This is because the exit switches are announced as DHCP gateway to the DHCP clients.

    Management Network

    To manage network switches beside the out-of-band system console access a further management access is required. For this purpose the concept of Management VRF is applied. The Management VRF is a subset of VRF. It provides a separation between out-of-band management network and the in-band data plane network by introducing another routing table mgmt. Cumulus Linux supports eth0 to be used as the management interface.

    To enable and use the Management VRF all switches have to be connected via their eth0 interface to a management-switch. The management switch is connected to a management server. All access is established from within the managment server. Logins to the switch are set into the Management VRF context once the Managment VRF is enabled.

    +# [...]

    Listing 14: VLAN access setup for bare metal server facing ports on leaves.

    Once a bare metal server is provisioned it is deconfigured from PXE VLAN vlan4000 to avoid accidential or unwanted provisioning.

    During provisioning bare metal servers get internet access via the management network of the exit switches. This is because the exit switches are announced as DHCP gateway to the DHCP clients.

    Management Network

    To manage network switches beside the out-of-band system console access a further management access is required. For this purpose the concept of Management VRF is applied. The Management VRF is a subset of VRF. It provides a separation between out-of-band management network and the in-band data plane network by introducing another routing table mgmt. Cumulus Linux supports eth0 to be used as the management interface.

    To enable and use the Management VRF all switches have to be connected via their eth0 interface to a management-switch. The management switch is connected to a management server. All access is established from within the managment server. Logins to the switch are set into the Management VRF context once the Managment VRF is enabled.

    diff --git a/dev/overview/os/index.html b/dev/overview/os/index.html index 3682c38bcd..fbe6ad63ff 100644 --- a/dev/overview/os/index.html +++ b/dev/overview/os/index.html @@ -1,2 +1,2 @@ -Operating Systems · metal-stack
    Edit on GitHub

    Operating Systems

    Our operating system images are built on regular basis from the metal-images repository.

    All images are hosted on GKE at images.metal-stack.io. Feel free to use this as a mirror for your metal-stack partitions if you want. The metal-stack developers continuously have an eye on the supported images. They are updated regularly and scanned for vulnerabilities.

    Supported OS Images

    The operating system images that we build are trimmed down to their bare essentials for serving as Kubernetes worker nodes. Small image sizes make machine provisioning blazingly fast.

    The supported images currently are:

    PlatformDistributionVersion
    LinuxDebian10
    LinuxUbuntu20.04

    Building Your Own Images

    It is fully possible to build your own operating system images and provide them through the metal-stack.

    There are some conventions though that you need to follow in order to make your image installable through the metal-hammer. You should understand the machine provisioning sequence before starting to write your own images.

    1. Images need to be compressed to a tarball using the lz4 compression algorithm
    2. An md5 checksum file with the same name as the image archive needs to be provided in the download path along with the actual os image
    3. A packages.txt containing the packages contained in the OS image should be provided in the download path (not strictly required)
    4. Consider semantic image versioning, which we use in our algorithms to select latest images (e.g. os-major.minor.patch ➡️ ubuntu-19.10.20191018)
    5. Consider installing packages used by the metal-stack infrastructure
      • FRR to enable routing-to-the-host in our network topology
      • go-lldpd to enable checking if the machine is still alive after user allocation
      • ignition for enabling users to run user-specific initialization instructions before bootup. It's pretty small in size, which is why we use it. However, you are free to use other cloud instance initialization tools if you want to.
    6. You have to provide an install.sh script, which applies user-specific configuration in the installed image
      • This script should consume parameters from the install.yaml file that the metal-hammer writes to /etc/metal/install.yaml
      • Please check this contract between image and the metal-hammer here
    7. For the time being, your image must be able to support kexec into the new operating system kernel, the kexec command is issued by the metal-hammer after running the install.sh. We do this because kexec is much faster than rebooting a machine.
    8. We recommend building images from Dockerfiles as it is done in metal-images repository.
    Info

    Building own operating system images is an advanced topic. When you have just started with metal-stack, we recommend using the public operating system images first.

    +Operating Systems · metal-stack
    Edit on GitHub

    Operating Systems

    Our operating system images are built on regular basis from the metal-images repository.

    All images are hosted on GKE at images.metal-stack.io. Feel free to use this as a mirror for your metal-stack partitions if you want. The metal-stack developers continuously have an eye on the supported images. They are updated regularly and scanned for vulnerabilities.

    Supported OS Images

    The operating system images that we build are trimmed down to their bare essentials for serving as Kubernetes worker nodes. Small image sizes make machine provisioning blazingly fast.

    The supported images currently are:

    PlatformDistributionVersion
    LinuxDebian10
    LinuxUbuntu20.04

    Building Your Own Images

    It is fully possible to build your own operating system images and provide them through the metal-stack.

    There are some conventions though that you need to follow in order to make your image installable through the metal-hammer. You should understand the machine provisioning sequence before starting to write your own images.

    1. Images need to be compressed to a tarball using the lz4 compression algorithm
    2. An md5 checksum file with the same name as the image archive needs to be provided in the download path along with the actual os image
    3. A packages.txt containing the packages contained in the OS image should be provided in the download path (not strictly required)
    4. Consider semantic image versioning, which we use in our algorithms to select latest images (e.g. os-major.minor.patch ➡️ ubuntu-19.10.20191018)
    5. Consider installing packages used by the metal-stack infrastructure
      • FRR to enable routing-to-the-host in our network topology
      • go-lldpd to enable checking if the machine is still alive after user allocation
      • ignition for enabling users to run user-specific initialization instructions before bootup. It's pretty small in size, which is why we use it. However, you are free to use other cloud instance initialization tools if you want to.
    6. You have to provide an install.sh script, which applies user-specific configuration in the installed image
      • This script should consume parameters from the install.yaml file that the metal-hammer writes to /etc/metal/install.yaml
      • Please check this contract between image and the metal-hammer here
    7. For the time being, your image must be able to support kexec into the new operating system kernel, the kexec command is issued by the metal-hammer after running the install.sh. We do this because kexec is much faster than rebooting a machine.
    8. We recommend building images from Dockerfiles as it is done in metal-images repository.
    Info

    Building own operating system images is an advanced topic. When you have just started with metal-stack, we recommend using the public operating system images first.

    diff --git a/dev/overview/storage/index.html b/dev/overview/storage/index.html index de2ffa848d..a8f4dbcc5c 100644 --- a/dev/overview/storage/index.html +++ b/dev/overview/storage/index.html @@ -9,4 +9,4 @@ resources: requests: storage: 100Mi - storageClassName: csi-lvm-sc-linear

    The solution does not provide cloud-storage or whatsoever, but it improves the user's accessibility of local storage on bare-metal machines through Kubernetes. Check out the driver's documentation here.

    + storageClassName: csi-lvm-sc-linear

    The solution does not provide cloud-storage or whatsoever, but it improves the user's accessibility of local storage on bare-metal machines through Kubernetes. Check out the driver's documentation here.

    diff --git a/dev/quickstart/index.html b/dev/quickstart/index.html index 5db59049bd..1d78a91e14 100644 --- a/dev/quickstart/index.html +++ b/dev/quickstart/index.html @@ -1,2 +1,2 @@ -Quickstart · metal-stack
    Edit on GitHub

    Getting Started

    Before starting to buy any hardware, you should try out the metal-stack on your notebook and familiarize with the software.

    For this, we made the mini-lab.

    The mini-lab is a fully virtual setup of metal-stack and is supposed to be run locally on a single machine. For this reason, the setup was slightly simplified in comparison to full-blown setups on real hardware. However, the lab should help to understand all ideas behind the metal-stack.

    Get your hands dirty and follow the guide on how to get on with the mini-lab here.

    +Quickstart · metal-stack
    Edit on GitHub

    Getting Started

    Before starting to buy any hardware, you should try out the metal-stack on your notebook and familiarize with the software.

    For this, we made the mini-lab.

    The mini-lab is a fully virtual setup of metal-stack and is supposed to be run locally on a single machine. For this reason, the setup was slightly simplified in comparison to full-blown setups on real hardware. However, the lab should help to understand all ideas behind the metal-stack.

    Get your hands dirty and follow the guide on how to get on with the mini-lab here.

    diff --git a/dev/search/index.html b/dev/search/index.html index ab382a9d83..f192b3559b 100644 --- a/dev/search/index.html +++ b/dev/search/index.html @@ -1,2 +1,2 @@ -Search · metal-stack

    Loading search...

      +Search · metal-stack

      Loading search...