Discussing nulls again, sorry

[fizmat]

Just bumped into this limitation in real life, specifically with a list containing nulls

pyproject.toml:

[tool.cxfreeze.bdist_msi.data]
Shortcut = ["MainShortcut", "DesktopFolder", "start_my_app", None, None, None, None, None, None, None, None, None]

"" or {} don't work, they are interpreted as a string/dictionary, not None. This expects exactly 12 values to insert one record into a ?sql? table where all columns except the first one are nullable.

I know the perfect fix would be to improve support of pyproject.toml in cx_Freeze. (As opposed to older setup.py, which obviously can create tuples with Nones easily). Ideally they should probably represent records in msi tables as a dictionaries and not tuples in the first place. But I can see why they went with the easiest implementation for the niche use-case of creating MSI installers, especially as setup.py was the only way to configure projects, so toml limitations were not even on the horison.

But having absolutely no way to represent None/null is still very annoying.

[eksortso]

Although TOML wasn't designed to be a general-purpose data language, there are lots of tools that allow arbitrary data in their configurations. In the case of cx_Freeze's bdist_msi module, the data element allows for the inclusion of SQL-based rows in its data tables using lists, according to its documentation. There are ways to express SQL NULLs in the middle of arrays when the tool's other configuration formats are used. But when pyproject.toml holds the configuration, which the project recommends, the nulls cannot be expressed.

In TOML's early days, we rejected nulls in configuration, because we thought that new projects adopting TOML could design their configurations from day one to avoid nulls. But as existing projects transition to TOML, it is getting more difficult to keep this condition, precisely because of the need for naïve data expression.

I hate suggesting that we add NULL to TOML, but the fact remains that the need for an explicit NULL is not going away. How should we approach this? And should we, if TOML v1.1.0 is on the verge of its first release candidate?

[epage]

How should we approach this? And should we, if TOML v1.1.0 is on the verge of its first release candidate?

imo features should only be removed from TOML 1.1 at this point and anything else should be slated for a 1.2+.

[eksortso]

imo features should only be removed from TOML 1.1 at this point and anything else should be slated for a 1.2+.

@epage Care to clarify? What ought to be removed? Can you file or refer to specific issues, or submit PRs?

[epage]

That wasn't to say features must be removed for 1.1 but what the mindset should be for 1.1: that of pairing down and focusing , rather than letting any lingering (or new) design discussions get in the way of the "core parts" of 1.1 from becoming available. I previously expressed a similar sentiment in #996.

If I were to order and prioritize things

• 1.0.1: Spec clarifications
• 1.1.0 at least inline table changes as that is the one thing I've most heard discussed from end users across different organizations.
  • I could see including the escaping extensions. I've never heard escaping needs come up. The fact that those changes exist means someone wants them (like my comment in that linked PR about wanting an RFC process, it would be nice to have an index of change proposals to see the motivations). They seem minor enough.
• future addition: unicode keys, comments characters

[ChristianSi]

I think this particular use case for NULL is absolutely unconvincing. Modeling a structured type with 12 fields as a 12-tuple is the worst possible way to go about it. How should people ever remember which value goes in the 9th field and which one in the 11th? Of course they won't.

TOML already offers a much better and more intuitive solution for this task: Use a table with keys, omitting the key in cases where it doesn't have a value (corresponding to NULL in SQL). So, in order not to encourage bad design, I would see this particular use case an as argument against rather than in favor of NULL.

In any case, if we decide to add null in the future – and I'm not strictly against, this particular bad use case notwithstanding – we must postpone it to v1.2 at the earliest. We'll be happy if we ever get v1.1 released and should absolutely not burden it with any additional new features, especially not with controversial ones (such as this one).

[eksortso]

I think this particular use case for NULL is absolutely unconvincing. [...]

@ChristianSi You ignored an important condition on this use case. This is an existing project. This isn't a change that we can easily assume that the project team will make. We can't guilt them into using the structured table approach, because they could just as easily fall back to their other configuration methods that permit explicit nulls in lists. And as sloppy as arbitrary data inserts are in theory, they could shoot back with: "we have lots of prewritten code here, we know our own tables, we know our columns, we know what we're doing."

I agree that we shouldn't pay the issue of nulls any more mind until after TOML v1.1.0 is released. How we'll reach the release, or even a hypothetical v1.0.1, is another issue best discussed elsewhere.