How to Add Grammar/Json Mode Sampling with llama_sampler_chain_add?

[niral28]

Hi I'm trying to add json grammar to my llama sampler. But have been struggling for a bit. I've been following this example:

https://github.com/ggerganov/llama.cpp/blob/master/examples/llama.android/llama/src/main/cpp/llama-android.cpp#L319C5-L319C28

And using this function in the library: https://github.com/ggerganov/llama.cpp/blob/53ff6b9b9fb25ed0ec0a213e05534fe7c3d0040f/include/llama.h#L1159

And by following some of the source code for llama-cli, this is what I came up with:

    auto sparams = llama_sampler_chain_default_params();
    sparams.no_perf = true;
    llama_sampler * smpl = llama_sampler_chain_init(sparams);
    llama_sampler_chain_add(smpl, llama_sampler_init_grammar(model, json_schema_to_grammar(json::parse("{}")).c_str(), "root")))
    llama_sampler_chain_add(smpl, llama_sampler_init_greedy());

However all my model generates is the opening brace "{" , repeatedly. Am I doing this correctly, can anyone help point out whats wrong?

Also do I need the greedy sampler, it seems to crash without it. https://github.com/ggerganov/llama.cpp/blob/53ff6b9b9fb25ed0ec0a213e05534fe7c3d0040f/include/llama.h#L1037

[ggerganov]

However all my model generates is the opening brace "{" , repeatedly. Am I doing this correctly, can anyone help point out whats wrong?

After sampling the token, make sure to call llama_sampler_accept().

[niral28]

And just for clarification, is the greedy/topk/some additional sampler required after the grammar sampler?

[niral28]

Still no luck as well,

I added llama_sampler_accept(sampler, new_token_id) right after this line: https://github.com/ggerganov/llama.cpp/blob/master/examples/llama.android/llama/src/main/cpp/llama-android.cpp#L414