Environment merging

[henryiii]

The current situation where environments replace existing environments is not ideal. For example:

[tool.cibuildwheel]
environment = { "A" = "a" }
[tool.cibuildwheel.macos]
environment = { "B" = "b" }
[[tool.cibuildwheel.overrides]]
select = "cp39-macosx*"
environment = { "C" = "c" }

If I get the environment on cp39-macosx_x86_64, I will get "C" = "c", rather than all three. This is further confused by the fact overrides do merge (in definition order), but not for dicts inside, like environment.

I think this behavior is likely much more commonly expected (merging) vs. replacing. The reason replacing was chosen is that if you merge, there's no way to fully unset a variable. That is, I could not specify A = None, B = None in the override if they were merged. I could set them equal to an empty string or "0", but that might have a different effect vs. truly being unset. TOML does not have a "None/nil" type value.

Here are some ideas.

1: Merging as default, provide a way to unset

We could specify a token value, CIBW_UNSET, that would unset an environment variable. This might even be useful for turning off a variable that normally gets set? If it could deactivate environment variables set by your CI for macOS and Windows, for example, it might even have some use outside of this. If not, it still is likely to be very rarely used, so it doesn't really add much to learn, since most users can ignore it.

The biggest downside is that this is change of behavior, but I don't think many users are relying on the replacement feature currently. We could add a feature flag until cibw 3 if that's a potential issue.

Here's an example of setting A and C but not B to illustrate this idea. The following two ideas cannot unset a single value.

[tool.cibuildwheel]
environment = { "A" = "a" }
[tool.cibuildwheel.macos]
environment = { "B" = "b" }
[[tool.cibuildwheel.overrides]]
select = "cp39-macosx*"
environment = { "B" = "CIBW_UNSET", "C" = "c" }

2: Toggle

Building on the feature flag idea from above, we could just provide this as a global toggle.

[tool.cibuildwheel]
merge-environments = true # 1/0 works too

I like the long-term simplicity of option 1 better, but this is also nice and less disruptive. This would not allow partial unsetting; it would be all or nothing.

3: Merging as an option

We could have a special form of environment that would merge. This would be a little ugly, inconsistent, and extra requirements on the users. Example using _merge:

[tool.cibuildwheel]
environment = { "A" = "a" }
[tool.cibuildwheel.macos]
environment_merge = { "B" = "b" }
[[tool.cibuildwheel.overrides]]
select = "cp39-macosx*"
environment_merge = { "C" = "c" }

This requires learning something special for the more common usage, rather than the less common usage. Also, we have to manually handle when a user sets both environment and environment_merge in the same place (TOML normally would handle conflicts between keys, but now they are not the same keys). You could use "+" instead of "_merge", but then TOML requires you surround the key in quotes, and that wouldn't work in the CIBW_ENVIRONMENT version.

Open to other ideas.

[joerick]

One idea I've seen elsewhere (I think it was Xcode build configuration) is the idea of a special value $inherit, which if included in the setting, inserts the previous value.

[tool.cibuildwheel]

environment = {"ASD" = "1"}
before_build = ["./prebuildscript"]

[[tool.cibuildwheel.overrides]]

select = "cp37-*"

environment = {"$inherit" = true, "QWE" = "1"}
before_build = ["$inherit", "./somethingelse"]

This approach could potentially generalise to commands as well, as seen above. Also it's quite nice in that you can control where the previous value is inserted e.g.

[[tool.cibuildwheel.overrides]]
select = "*-aarch64"

before_build = ["./prepare-arm-build", "$inherit"]

Would run ./prepare-arm-build before the default before_build.

[PrimozGodec]

I hope I am not abusing this conversation, but my question suits here.

I want to set different environment variables for Linux arrch64 than other linuxes. I have tried different combinations but nothing since to work. This is the current attempt (and it doesn't do anything):

[tool.cibuildwheel.linux.environment]
AR = "gcc-ar"
NM = "gcc-nm"
RANLIB = "gcc-ranlib"
LDFLAGS = "-flto"
CFLAGS = "-O3 -g1 -pipe -fPIC -flto -march=core2"
STATIC_DEPS = "true"
LIBXML2_VERSION = "2.11.5"
LIBXSLT_VERSION = "1.1.38"

[[tool.cibuildwheel.overrides]]
select = "*aarch64"
[tool.cibuildwheel.overrides.linux]
environment = {CFLAGS = "-O3 -g1 -pipe -fPIC -flto -march=armv8-a -mtune=cortex-a72"}

I want to override only CFLAGS. Other variables should persist. If I understand correctly, I should repeat all variables?
What do I need to change? Do you have any working examples?

[joerick]

Yes, for now, you'll need to repeat all variables inside the environment option. Also, you should get rid of that header, it won't be doing you any good - actually, I'm surprised it's valid TOML at all.

  [[tool.cibuildwheel.overrides]]
  select = "*aarch64"
- [tool.cibuildwheel.overrides.linux]
- environment = {CFLAGS = "-O3 -g1 -pipe -fPIC -flto -march=armv8-a -mtune=cortex-a72"}
+ environment = {CFLAGS = "-O3 -g1 -pipe -fPIC -flto -march=armv8-a -mtune=cortex-a72", AR = "gcc-ar", NM = "gcc-nm", RANLIB = "gcc-ranlib", LDFLAGS = "-flto", STATIC_DEPS = "true", LIBXML2_VERSION = "2.11.5", LIBXSLT_VERSION = "1.1.38" }

[PrimozGodec]

Thanks for your help. It seems it is woking now.

[henryiii]

You don't need it, but you can do inline tables (linux.environment) inside arrays.