This Repo vs Converter Sources

[tajmone]

In the README it says:

The text is extracted from the converter's PPL source code.
This repository has been published to enable people to submit pull requests and start discussions regarding the reference manual.

Does it mean that you have to actually re-integrate manually any third party contributions to the PML sources? i.e. somehow copy and paste them back into the converter sources.

Or does this new repository replace the old system of generating the Ref Man from the converter sources? i.e. the only Ref Man sources are now the files in this repository?

[pml-lang]

you have to actually re-integrate manually any third party contributions to the PML sources?

Yes, unfortunately. I thought already about adding a note in the README file, and suggest people to make pull requests (for changes in the reference manual) in the converter's repository, rather than in this repository, if possible. That would eliminate the need for re-integrating the changes manually.

PS: Thanks a lot for your PRs. I will merge them soon.

[tajmone]

Yes, unfortunately.

That's really a bad downside.

I've finally managed to work out where the Ref Man resides in the converter repo. I see it's not just broken into many snippets associated with each node, but also that the actual text is intertwined with PPL code (macros?) that injects some text too.
E.g. from se_PML_insert_source_code_formal_node.ppl for the Insert To Regex section (code truncated via [...]):

            title = "Insert To Regex"
            description = """If this parameter is not defined [...]

If parameter '{{a_include_to_regex_parameter_id}}' is set to 'yes' [...]"""

I was hoping that there could be some hackerish solution to spare you from having to manually reintegrate contributed changes from this repository — e.g. by inserting some PML comments with special start/end markers that would allow a script to extract the various snippets from the PML source here and inject them in their destination file; but the presence of these {{a_XXX_id}} PPL macros (?) seems to prevent this solution.

Regarding Submitting Changes to pml-lang/converter

I thought already about adding a note in the README file, and suggest people to make pull requests (for changes in the reference manual) in the converter's repository, rather than in this repository, if possible. That would eliminate the need for re-integrating the changes manually.

The problem with this solution is that users won't be able to verify how their changes look like in the final document unless they install the whole PPL language + dependencies — and, honestly, it might be a strong deterrent for the average PML user who only wants to fix a few typos or tweak the text here and there. Furthermore, if people submit changes to both repositories, integration would become even harder and conflicts-prone.

Possible Solutions to the Dual Document-Sources Dilemma

We're faced with a serious dilemma here. In the long term, having to manually integrate changes back and forth between the two repos is going to be a huge burden, unless automated in a reliable manner.

Beside finding a way to programmatically handle integrations and their validation in an automated way, the only other solutions that come to my mind are:

1. Divorcing the PML Ref Man from the converter source code — which I don't think is a good idea since it's a very good choice to keep the documentation source within the app's source code, not just for the added benefits of PPL text injections through language introspection, but because it's good practice.
2. Divorcing this Ref Man from the one generated by pmlc — i.e. this repository would be hosting an alternative Ref Man, community based and maybe a longer version (more examples, links, etc.) compared to the one bundled with (and generated by) the converter.

The downside of the latter solution is that having two different Ref Mans could create confusion (this could be solved by using a different title for the one in this repository), but also it wouldn't fully solve the integration problem, because this Ref Man would still need to integrate changes from the official one, especially concerning syntax changes.

In any case, it's good to have a repository were users can find the entire PML source of the PML Reference Manual (as opposed to having to dig for snippets in the PPL sources of the converter repository) because the index.pml file can always serve as a reference — but then, one doesn't really a repository for this, pmlc could simply generated the Ref Man PML source on demand, or it could just be bundled with the package. The only valid reason to keep this repository up is to allow contributions.

A Middle-Ground "Buffered" Solution

Maybe, until we have a reasonable solution for the integration problem, in this repository there should be two copies of the PML Ref Man: one which is generated from the latest converter code (and which users shouldn't edit), and another where users should add their contributions. E.g.:

• official.pml — the auto-generated Ref Man source, not for editing, for reference only.
• index.pml — the source where users should apply changes to.

Having two files would allow both you and end users to track which proposed changes are still pending and which have already been integrated (e.g. by diffing or visually comparing the two files).

Far from perfect, this solution might introduce some extra integration problems (e.g. if a new node or attribute is added which shows in official.pml but not not in index.pml), but in theory this can be handled by replacing index.pml with a fresh copy of official.pml whenever all changed have been integrated (or discarded). The benefit of this solution is that overriding index.pml is an operation independent of the various PRs that were accepted in the course of time.

It's just an idea to minimize the integration problem, because it's quite possible that you might end up manually integrating some proposed changes while postponing others, which would result in a discrepancy between the edited document in this repo and the auto-generated one.

---

None of the above solutions are ideal (far from that), and probably there's no perfect solution to this problem beside an automated way to integrate changes (which I doubt could ever be fully reliable), but at least they attempt to brainstorm and tackle the problem from different angles.

[pml-lang]

pmlc could simply generate the Ref Man PML source on demand

That's done already. After installing the converter you can create the reference manual with command create_reference_manual.
To get help for this command, type:
pmlc command_info --command create_reference_manual
The PML files are stored in the directory specified by argument temp_directory.

or it could just be bundled with the package

That's done for the HTML files, but not for the PML files.
The HTML files for the user manual and reference manual are stored in subdirectories \docs\ref_manual, and \docs\user_manual.
Maybe the PML files should be included too in the distribution.

For now I will simply make updates manually, and keep the DRY principle. If it becomes unmanageable later, we'll have to find a better solution.

Thanks a lot for your suggestions.