Compile support for special FILES - protobuf, Thrift and Avro

[linux-china]

Protobuf and gRPC are used widely. If a developer use //FILES account.proto in script, it's not just to add file in classpath, and it means to help me compile protobuf and then I can import generated Message class or gRPC class directly. Yes, most developers know how to use Gradle/Maven to compile protobuf, but it's boring work to repeat again. JBang can make this job simple, just //FIES proto file, and JBang will fix everything for you, and you can use Generated classes and interface directly.

Custom resource compile is normal in JavaScript, such as json import, GraphQL import, Yaml import etc, and these resources will be transformed to JS object by plugin during compile, and easy for developer to operate transformed JS object.

import configuration from './config.json'
import queries from './demo.graphql'  
import demo from './demo.yaml'

Following resource types could be considered at first:

• protobuf & grpc: if service keyword detected, compiled with gRPC plugin
• Apache Avro
• Apache Thrift
• JSON Schema: jbang-catalog-schema.json to JbangCatalog.java

For this case, HTTP resource should be considered, such as //FILES demo.proto=https://github.com/user/name/demo.proto. These resources almost act as contract/interface for client/server and are stored on git or registry server.

[maxandersen]

This is treading into full build system territory :)

Something if we get too far users are better of using maven/gradle.

I'm fine exploring it but let's see if we can avoid adding it to jbang core as proto buf isn't the only thing one would want.

Adding it to //FILES does not sound right as that is supposed to be dumb and direct adding resources to the built result.

We have the integration mechanism that lets one have dependency that will be able to hook in after compile but before zipping ...but that is too late for protobuf?

[linux-china]

The generated Code from resource should be transformed before Java compile because the generated code will be used by Java Snippet.

[linux-china]

@maxandersen how about adding before compile for integration? It's like process-resources in Maven, then try build some demo.

// return the generated sources and files 
Map<String, String> beforeCompile(Path builddir, Path scriptFile, Map<String, String>  files,  List<Map.Entry<String, Path>> dependencies)

[maxandersen]

we could but means we would need to introduce some intermediate storage ...like a maven target or gradle build folder...and add copy/move of files. starting to be non trivial.

[linux-china]

esbuild is a famous JS/TS transform tool, and it has good feature called loader, and it can help you transform resource to special object. For example, in Jbang script, I use //FILES demo.csv, and I just want to get content of CSV. You should use getResourceAsStream, then convert InputStream to String, too much code.

With loader introduced, you can use following code:

//FILES:text demo.csv

String csvCode = DemoCSV.text; 

Jbang will help to read demo.csv and generate a new Java file called DemoCSV.java, and demo.cvs's content has been inlined into text field, and you just need to invoke DemoCSV.text, that's all.

We can design loader alike esbuild, but focus on Java only:

• json -> JsonNode or custom class
• xml, yaml , toml -> Jackson Object
• binary: load resource as byte[]
• text: load resource as utf-8 String
• template file, such freemarker, thymeleaf: convert to Template Object

[maxandersen]

Wouldn't that be feasible with a annotation processor or use Quarkus buildtime extension?

[linux-china]

yes, annotation processor is better.

I should take a look at Quarkus buildtime extension. :) It works with Groovy and Kotlin?

[maxandersen]

Kotlin yes. Groovy not yet but there are some working on enabling it.