Logging

[spehl-max]

Hello,

I am wondering if any developers are adding logging to the NeMo chain.

For debugging purposes, I'd find it useful to log,

-every prompt that was fed to an LLM,
-every rail a conversation followed and,
-any triggered InternalSystemActions.

If not, I could try.

Thank you,
Max

[spehl-max]

Follow-up:

I see that when I enabled the verbose option in the LLMRails function with,

app = LLMRails(config, verbose=True)

And ran a test,

print(app.generate(messages=[{ "role": "user", "content": "Hello! What can you do for me?" }]))

The result is,

{'role': 'assistant', 'content': 'I can help you with a variety of banking services. What can I do for you today?'}

Whereas, when I use the --verbose option in the nemoguardrails chat cli,

I obtain a much more detailed log.

I do not understand why the behavior of the verbose option is so different between the app.generate() python call and the cli. Am I misunderstanding how the option is meant to be used?

[drazvan]

Hi @spehl-max !

Indeed, there is an inconsistency in behavior. The difference comes from the fact the the CLI Chat also calls the set_verbose function (https://github.com/NVIDIA/NeMo-Guardrails/blob/develop/nemoguardrails/cli/__init__.py#L63C13-L63C13) which adds a new logging handler. So, if you want to achieve the same behavior from the python API using app.generate, in addition to passing verbose=True you should also call set_verbose(True) before.

We've also just added an explain feature which is used in the Getting Started guide, for example here: https://github.com/NVIDIA/NeMo-Guardrails/blob/develop/docs/getting_started/2_core_colang_concepts/README.md#the-explain-feature.

If you have a time to make a PR that fixes the verbose logging inconsistency, that would be hugely appreciated.
Thanks!

[spehl-max]

Thank you, @drazvan! I will make a PR after December 12th, as I will have more free time then.