OpenTofu SDK

[anton-yurchenko]

Hey,
Is there a plan to provide an SDK to allow the creation of new applications using OpenTofu?
Example:

package main

import (
    "context"
    "log"
    "github.com/hashicorp/terraform-exec/tfexec"
)

func main() {
    workingDir := "/path/to/your/terraform/configs"
    execPath := "/path/to/terraform/executable"

    tf, err := tfexec.NewTerraform(workingDir, execPath)
    if err != nil {
        log.Fatalf("error setting up Terraform: %s", err)
    }

    err = tf.Init(context.Background(), tfexec.Upgrade(true))
    if err != nil {
        log.Fatalf("error initializing Terraform: %s", err)
    }

    err = tf.Apply(context.Background())
    if err != nil {
        log.Fatalf("error applying Terraform: %s", err)
    }
}

[ghost]

Hi @anton-yurchenko there are two aspects to the question:

1. tfexec launches terraform in the background as a separate application if I remember correctly. It does not, however, compile Terraform directly into your application. Since tfexec is under the MPL license, it should be fairly easy to adapt it for OpenTofu (Please open an issue if this is of interest.)
2. There have been several requests to make it possible to build OpenTofu directly into an application without launching an external binary. Unfortunately, OpenTofu has a lot of internal state that doesn't get cleaned up, so even if we exposed such an API, running apply twice in a row would result in bugs. For the foreseeable future this will not be possible.

[siennathesane]

even if we exposed such an API, running apply twice in a row would result in bugs

I would rather know this and have access to it than not have access at all and have to deal with a separate binary and process. I think this is also a simple workaround in the short-term by re-initializing the internals after every run to mimic the binary's behaviour.

[georgalis]

even if we exposed such an API, running apply twice in a row would result in bugs

I would rather know this and have access to it than not have access at all and have to deal with a separate binary and process. I think this is also a simple workaround in the short-term by re-initializing the internals after every run to mimic the binary's behaviour.

right I was thinking the same thing, mitigation. For example in my shell, I use this alias...

    alias _env='tput sgr0 ; chkerr "$(jobs -l)" \
        && exec env -i TERM="$TERM" COLORTERM="$COLORTERM" \
            SHELL="$SHELL" HOME="$HOME" LOGNAME="$LOGNAME" USER="$USER" \
            SSH_AGENT_PID="$SSH_AGENT_PID" SSH_AUTH_SOCK="$SSH_AUTH_SOCK" \
            '"${SHELL} -l"

where chkerr is a function that returns "$*" to stderr and signal 1, unless it is null and returns signal 0, which prevents the alias if background jobs are running, otherwise anytime I mess with my env, I run _env which leverages env -i to remove any artifacts from my old env and initializes my login env... much better than opening a new terminal.

Even if the env gets out of hand, being able to initialize a controlled environment, program requirements, is an important aspect quality assurance.

[ghost]

@siennathesane I can only reemphasize how large the codebase is. If this was just about a few variables, that would be easy. There is a large maintenance cost to be paid for such a feature where hard to reproduce state bugs need to be tracked down.

On a practical side, there is very little benefit of using OpenTofu as a library over shelling out to it. OpenTofu itself shells out to launch the providers as these are separate binaries and the performance benefit you'd gain from not reinitializing the binary is negligible compared to the other work that's being done.

OpenTofu does not currently have anything close to a clean API you would expect from a library. What you would get from an OpenTofu library right now now is basically equivalent to a shellout.

For the time being I would recommend using TofuDL in conjunction with the exec package.

[marziply]

My concern isn't necessarily with performance but of portability. I want to embed Tofu into my applications so it's not technically needed as a dependency on the host.

[ghost]

@marziply if your concern is portability, you'll have the same problem with providers. Each provider is a separate binary that OpenTofu downloads for the specific platform and architecture from GitHub using the registry as an index. If the author didn't compile the provider for your platform, you can't use it. For example, if you wanted to run OpenTofu on PowerPC or riscv64, both unsupported platforms, just compiling OpenTofu for them won't do the trick, you'll also have to compile and publish each and every provider you intend to use.

As mentioned, providers run entirely as a separate process and OpenTofu talks to them using grpc.

TofuDL essentially does the same for OpenTofu what OpenTofu does for the registry: it downloads the binary for your platform and makes it available to your Go application. You can then use exec to launch it. If you need to support an unsupported platform, TofuDL has built-in tooling to publish your own binaries for your desired platform, including GPG signing them and running a full mirror that is compatible with TofuDL.