Provide a long-term policy for LLM usage

[PPPDUD]

In 2023, @GarboMuffin made a temporary LLM policy but then removed it a year later. #2258 brought up the possibility that LLM-written code may be able to pass code review with the current state of things, and this would be a bleak reality for our userbase.

This pull request makes a more permanent set of guidelines that were written in the spirit of GarboMuffin's original ideas, but expanded to account for the modern usage of LLMs and the increasing number of image generators.

[Brackets-Coder]

I wouldn't make this official yet, it's not a fine line and my statement in #2258 was non-conclusive. While I personally have opinions about proper responsible use of artificial intelligence, I'd wait on @GarboMuffin's opinion on really anything regarding official contributing guidelines.

[Brackets-Coder]

Tricky issue...

[PPPDUD]

@Brackets-Coder I started a review by accident. Please ignore it.

Problems fixed.

[PPPDUD]

@Brackets-Coder @GarboMuffin What do you think of the latest revision?

[Brackets-Coder]

This is still a tricky issue, I'm going to need @GarboMuffin opinions.

[PPPDUD]

#2286 appears to have been assisted by an LLM. This is getting worse every day that my policy improvements are delayed.

[PPPDUD]

@SharkPool-SP @CubesterYT I want some more perspectives on this. What do you all think?

[GarboMuffin]

i am not presently concerned about the lack of an LLM policy in that document

[Brackets-Coder]

Closed because comment above

[PPPDUD]

I am reopening this because #2327 gives us authority to approve CONTRIBUTING.MD changes ourselves.

Predates new review procedures.

[PPPDUD]

#1828 and #2338 are also disclosed as having used LLMs. @Brackets-Coder @CubesterYT What do you think about approving this now that the new review process is in place?

[GarboMuffin]

I am not convinced that we need an LLM policy. People seem to be largely honest about it. Having a no LLM policy will still result in LLM spam they just won't be honest about it. Have you found any significant functional issues in those extensions?

[Brackets-Coder]

I'm not deeply concerned about an LLM policy, if an extension is poorly written we won't approve it anyway
it it just so happens to be perfect and written with an LLM then it's probably too small or niche to be merged

[PPPDUD]

I am not convinced that we need an LLM policy. People seem to be largely honest about it. Having a no LLM policy will still result in LLM spam they just won't be honest about it. Have you found any significant functional issues in those extensions?

I'm not deeply concerned about an LLM policy, if an extension is poorly written we won't approve it anyway it it just so happens to be perfect and written with an LLM then it's probably too small or niche to be merged

I'm mostly concerned with having some procedure to quickly reject LLM spam without having to treat it with the respect of someone's actual hard work. Currently, there is not a well-defined procedure for handling such extensions, and the PRs will often stay open for several months before getting closed.

As for functional issues, take a look at #2338. Their proposal incorrectly claims that "all strings...include German, Spanish, French, etc., translations within the file". If you review their source code, however, you will notice a complete lack of such functionality. They also added an apparently-hallucinated description field in their getInfo() function that appears to do nothing.

This is the sort of code quality that will slowly destroy any large project, TurboWarp included.

[ScratchFakemon]

I've only made a few small contributions to this repo (such as renaming the "Sound" extension to "URL Playback" and one that's still being reviewed to add some JSON array-related blocks to the Text/strings extension), but I've also made some more ambitious extensions that I haven't submitted, some of which have used AI for some things. These were made at a time when I was a much worse programmer (and I prefer to look at other extensions or ask real people for help with the things I don't know now), but they weren't fully made with AI. My personal views on Artificial Intelligence are that it should be used as a tool for assisting with creation (not with vibe coding a whole extension), and that it should be disclosed if AI is used in any capacity.

One of the extensions I've made (which is more of a single-project thing, so I'm not even considering submitting it) lists ChatGPT under the credits, and I thought another one did, but it actually didn't (time to go add that haha). If you care enough about privacy and glance at the code of a non-gallery custom extension, it's easy enough to notice this, but I feel like most people don't pay much attention to the credits in the gallery. What if the metadata supported a usesAI tag (boolean, of course) that would display a little indicator that says "This extension was made using AI in some capacity." This could use a darkened or translucent version of the accent color and an icon with some stars (usually used to signify AI) to catch people's attention and be more open about the use of AI.

I could probably implement this to both versions of the gallery, but I'm not gonna start working on something if no one else thinks it would be useful.

---

(Also, off topic (and mostlyentirely as a joke), but, since you didn't want to be called Mr. Weber by Brackets-Coder, should we call you "Garbo", "Muffin", "GarboMuffin", "Griffin", and/or "Griffin Weber" instead?)

[PPPDUD]

I've only made a few small contributions to this repo (such as renaming the "Sound" extension to "URL Playback" and one that's still being reviewed to add some JSON array-related blocks to the Text/strings extension), but I've also made some more ambitious extensions that I haven't submitted, some of which have used AI for some things. These were made at a time when I was a much worse programmer (and I prefer to look at other extensions or ask real people for help with the things I don't know now), but they weren't fully made with AI. My personal views on Artificial Intelligence are that it should be used as a tool for assisting with creation (not with vibe coding a whole extension), and that it should be disclosed if AI is used in any capacity.

One of the extensions I've made (which is more of a single-project thing, so I'm not even considering submitting it) lists ChatGPT under the credits, and I thought another one did, but it actually didn't (time to go add that haha). If you care enough about privacy and glance at the code of a non-gallery custom extension, it's easy enough to notice this, but I feel like most people don't pay much attention to the credits in the gallery. What if the metadata supported a usesAI tag (boolean, of course) that would display a little indicator that says "This extension was made using AI in some capacity." This could use a darkened or translucent version of the accent color and an icon with some stars (usually used to signify AI) to catch people's attention and be more open about the use of AI.

That sounds interesting! Perhaps we should add a filter to the extensions list where you can show or hide LLM-generated extensions, which would be hidden by default. I have stronger opinions against vibe coding than you, but I would certainly accept this as a fair compromise. @Brackets-Coder What is your take on this?

[ScratchFakemon]

I've only made a few small contributions to this repo (such as renaming the "Sound" extension to "URL Playback" and one that's still being reviewed to add some JSON array-related blocks to the Text/strings extension), but I've also made some more ambitious extensions that I haven't submitted, some of which have used AI for some things. These were made at a time when I was a much worse programmer (and I prefer to look at other extensions or ask real people for help with the things I don't know now), but they weren't fully made with AI. My personal views on Artificial Intelligence are that it should be used as a tool for assisting with creation (not with vibe coding a whole extension), and that it should be disclosed if AI is used in any capacity.
One of the extensions I've made (which is more of a single-project thing, so I'm not even considering submitting it) lists ChatGPT under the credits, and I thought another one did, but it actually didn't (time to go add that haha). If you care enough about privacy and glance at the code of a non-gallery custom extension, it's easy enough to notice this, but I feel like most people don't pay much attention to the credits in the gallery. What if the metadata supported a usesAI tag (boolean, of course) that would display a little indicator that says "This extension was made using AI in some capacity." This could use a darkened or translucent version of the accent color and an icon with some stars (usually used to signify AI) to catch people's attention and be more open about the use of AI.

That sounds interesting! Perhaps we should add a filter to the extensions list where you can show or hide LLM-generated extensions, which would be hidden by default. I have stronger opinions against vibe coding than you, but I would certainly accept this as a fair compromise. @Brackets-Coder What is your take on this?

I was also sort of thinking that lol

[Brackets-Coder]

I don't think LLM extensions are as big as a problem as needing a filter on the gallery:

• If an extension is entirely LLM generated then the author likely doesn't know enough JS to be adept at writing their own functional extensions. Entirely LLM generated extensions will not be merged
• Extensions which were mostly generated by LLMs are in the grey area
• Extensions which were mostly authored by a real person who used LLMs to enhance and assist the development of their own code are not a problem, especially when they're adapting the LLM-generated code to work contextually within the other code they authored.

[PPPDUD]

I don't think LLM extensions are as big as a problem as needing a filter on the gallery:

• If an extension is entirely LLM generated then the author likely doesn't know enough JS to be adept at writing their own functional extensions. Entirely LLM generated extensions will not be merged
• Extensions which were mostly generated by LLMs are in the grey area
• Extensions which were mostly authored by a real person who used LLMs to enhance and assist the development of their own code are not a problem, especially when they're adapting the LLM-generated code to work contextually within the other code they authored.

It is concerning to believe that an extension could be created with near-zero effort and a few ideas, and then potentially get accepted because it was slightly tweaked by a human.

[Brackets-Coder]

I don't think LLM extensions are as big as a problem as needing a filter on the gallery:

• If an extension is entirely LLM generated then the author likely doesn't know enough JS to be adept at writing their own functional extensions. Entirely LLM generated extensions will not be merged
• Extensions which were mostly generated by LLMs are in the grey area
• Extensions which were mostly authored by a real person who used LLMs to enhance and assist the development of their own code are not a problem, especially when they're adapting the LLM-generated code to work contextually within the other code they authored.

It is concerning to believe that an extension could be created with near-zero effort and a few ideas, and then potentially get accepted because it was slightly tweaked by a human.

I would not approve such extensions, rather the other way around is acceptable:
Mostly written by a human but modified LLM code or assistive code is acceptable..

[PPPDUD]

I don't think LLM extensions are as big as a problem as needing a filter on the gallery:

• If an extension is entirely LLM generated then the author likely doesn't know enough JS to be adept at writing their own functional extensions. Entirely LLM generated extensions will not be merged
• Extensions which were mostly generated by LLMs are in the grey area
• Extensions which were mostly authored by a real person who used LLMs to enhance and assist the development of their own code are not a problem, especially when they're adapting the LLM-generated code to work contextually within the other code they authored.

It is concerning to believe that an extension could be created with near-zero effort and a few ideas, and then potentially get accepted because it was slightly tweaked by a human.

I would not approve such extensions, rather the other way around is acceptable: Mostly written by a human but modified LLM code or assistive code is acceptable..

But there's no policy allowing for a quick close of such a PR, and someone else on our team might be looser about approving stuff.

[Brackets-Coder]

But there's no policy allowing for a quick close of such a PR, and someone else on our team might be looser about approving stuff.

Someone else on our team who believes so strongly on the opposite side (that is, allowing all PRs a chance, even LLM generated ones) wouldn't follow a policy if there was one anyway, so it's best to just leave it up to other reviewers to determine at their wisest discretion when a PR should be closed.