Stream text from OpenAI

[gykung]

Would anyone know how we can stream responses from the OpenAI chat.completions endpoint?

[falkoschindler]

Hi @gykung,
What exactly do you want to know? Can you give more information about the endpoint you're referring to? How is this related to NiceGUI?
For reference: This question has been cross-posted on reddit.

[gykung]

Hi @falkoschindler , thanks for reading this. I mean a chatbot using the chat completions API, with text streaming, such as in this example: docs.streamlit.io/develop/tutorials/chat-and-llm-apps/build-conversational-apps#write-the-app.

I'm familiar with the OpenAI API, but do not have enough front-end knowledge or familiarity with nicegui to replicate the streaming behaviour in nicegui.

For reference, this is the chat completions API: https://platform.openai.com/docs/api-reference/chat

[rodja]

Have a look at our OpenAI assistant example:

nicegui/examples/openai_assistant/main.py

Lines 36 to 50 in 78a5035

	stream = await client.beta.threads.runs.create(
	assistant_id=assistant.id,
	thread_id=thread.id,
	stream=True,
	)
	async for chunk in stream:
	if isinstance(chunk, ThreadMessageInProgress):
	spinner.delete()
	# NOTE: the stream contains a lot of different types so we need to filter out the ones we don't need
	if not isinstance(chunk.data, MessageDeltaEvent) or not chunk.data.delta.content:
	continue
	content = chunk.data.delta.content[0]
	if not isinstance(content, TextDeltaBlock) or content.text is None or content.text.value is None:
	continue
	response.content += content.text.value

Also the langchain example might be helpful:

nicegui/examples/chat_with_ai/main.py

Lines 23 to 28 in c8c3dd7

	response = ''
	async for chunk in llm.astream(question, config={'callbacks': [NiceGuiLogElementCallbackHandler(log)]}):
	response += chunk.content
	response_message.clear()
	with response_message:
	ui.html(response)

[gykung]

@rodja thanks! This works. Next I would try to figure out how to make it show the previous turns (history from user/bot), UI as speech bubbles ('bot' aligned left, 'user' aligned right), and the input box to always stick to the bottom (similar to most chat apps)...

I think NiceGUI is not well known enough - I did not manage to find this example via Google, perhaps I was using the wrong keywords.

[rodja]

Next I would try to figure out how to make it show the previous turns (history from user/bot), UI as speech bubbles ('bot' aligned left, 'user' aligned right), and the input box to always stick to the bottom (similar to most chat apps)...

Have you launched the examples? Most of what you write is already implement...

I think NiceGUI is not well known enough - I did not manage to find this example via Google, perhaps I was using the wrong keywords.

Yes. SEO and marketing are not our strong suits. But try the search on our website https://nicegui.io. It should show you the examples you are looking for when you search "chat app".