Implemented low level websocket client in java, not able to get any events back apart from Results.

[rostwal95]

{
"type": "Configure",
"features": {
"asr": {
"language": "en-US",
"interim_results": true,
"vad_events": true,
"model": "nova-3",
"encoding": "mulaw",
"sample_rate": 8000,
"channels":1,
"utterance_end_ms": "1000",
"endpointing": 300,
"punctuate": true
}
}
}

I am using deepgrams STT feature.
Tried request on both nova-2 and nova-3 model.

SessionId: 46df0a24-9aef-42f0-8bda-8a3a3f130b9a - here is the sessionId for reference.

        I dont see any error back on the websocket. if I try to change the message type to config, I do get schema error, which essentially means the message structure is correct, I am sending this message after opening the websocket connection and subsequently audio stream flows. Is there anything I am missing here ??

[deepgram-community[bot]]

Thanks for asking your question. Please be sure to reply with as much detail as possible so the community can assist you efficiently.
_{Consider joining our Discord community for more opportunity to engage with your fellow Deepgram users. You can earn points which can be redeemed for cool stuff by being active in our communities!}

[deepgram-community[bot]]

Hey there! It looks like you haven't connected your GitHub account to your Deepgram account. You can do this at https://community.deepgram.com - being verified through this process will allow our team to help you in a much more streamlined fashion.

[deepgram-community[bot]]

It looks like we're missing some important information to help debug your issue. Would you mind providing us with the following details in a reply?

• The deepgram product you are using (e.g Speech to Text, Agent API)
• A request ID that triggered your error or issue.

[deepgram-community[bot]]

Are you using any of our SDKs to do this?

Can you provide more of the code that isn't working for you beyond just this configuration example?

Also if you are getting RequestIDs from our API that would be helpful to share. I looked up the Session ID you provide, and all I see is:

The request's usage data is still processing,Check back later for more information.

This message was sent by John Vajda from Deepgram, via our community automation.

[rostwal95]

I am not able to use sdk, I tried the python sdk and things are working perfectly fine for me, but I need to implement the client in java due to few constraints.

import jakarta.websocket.*;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;

import java.io.IOException;
import java.net.URI;
import java.net.URISyntaxException;
import java.nio.ByteBuffer;
import java.util.Collections;
import java.util.List;
import java.util.Map;
import java.util.concurrent.CountDownLatch;
import java.util.concurrent.Executors;
import java.util.concurrent.ScheduledExecutorService;
import java.util.concurrent.TimeUnit;

@ClientEndpoint
public class DeepgramWebSocketClient extends Endpoint {

    private static final Logger logger = LoggerFactory.getLogger(DeepgramWebSocketClient.class);

    private Session session;
    private final CountDownLatch closeLatch = new CountDownLatch(1);
    private final DeepgramTranscriptionListener transcriptionListener;

    private static final long KEEP_ALIVE_INTERVAL_SECONDS = 5;
    private ScheduledExecutorService keepAliveExecutor;

    private final String apiKey;
    private final String language;
    private final int sampleRate;
    private final int channels;
    private final String encoding;
    private final boolean interimResults;

    public DeepgramWebSocketClient(DeepgramTranscriptionListener transcriptionListener,
                                   String apiKey,
                                   String language,
                                   int sampleRate,
                                   int channels,
                                   String encoding,
                                   boolean interimResults) {
        this.transcriptionListener = transcriptionListener;
        this.apiKey = apiKey;
        this.language = language;
        this.sampleRate = sampleRate;
        this.channels = channels;
        this.encoding = encoding;
        this.interimResults = interimResults;
    }

    @Override
    public void onOpen(Session session, EndpointConfig config) {
        this.session = session;
        logger.info("Connected to Deepgram WebSocket. SessionId: {}", session.getId());

        session.addMessageHandler(String.class, new MessageHandler.Partial<>() {
            private final StringBuilder messageBuffer = new StringBuilder();

            @Override
            public void onMessage(String message, boolean last) {
                messageBuffer.append(message);
                if (last) {
                    String completeMessage = messageBuffer.toString();
                    messageBuffer.setLength(0);
                    handleTextMessage(completeMessage);
                } else {
                    logger.info("Received partial message chunk: {}", message);
                }
            }
        });

        session.addMessageHandler(ByteBuffer.class, this::handleBinaryMessage);

        sendConfigureMessage();
        startKeepAliveScheduler();
    }

    private void startKeepAliveScheduler() {
        keepAliveExecutor = Executors.newSingleThreadScheduledExecutor();
        keepAliveExecutor.scheduleAtFixedRate(() -> {
            if (session != null && session.isOpen()) {
                sendKeepAliveMessage();
            }
        }, KEEP_ALIVE_INTERVAL_SECONDS, KEEP_ALIVE_INTERVAL_SECONDS, TimeUnit.SECONDS);
        logger.debug("KeepAlive scheduler started ({} seconds interval)", KEEP_ALIVE_INTERVAL_SECONDS);
    }

    private void stopKeepAliveScheduler() {
        if (keepAliveExecutor != null && !keepAliveExecutor.isShutdown()) {
            keepAliveExecutor.shutdownNow();
            logger.info("KeepAlive scheduler stopped.");
        }
    }

    private void handleTextMessage(String message) {
        logger.info("Received complete text message: {}", message);
        if (transcriptionListener != null) {
            transcriptionListener.onTranscription(message);
        }
    }

    private void handleBinaryMessage(ByteBuffer message) {
        logger.info("Received binary message of size: {} bytes", message.remaining());
    }

    @Override
    @OnError
    public void onError(Session session, Throwable throwable) {
        logger.error("WebSocket encountered an error", throwable);
    }

    @Override
    @OnClose
    public void onClose(Session session, CloseReason closeReason) {
        logger.info("WebSocket closed. Reason: {}", closeReason.getReasonPhrase());
        stopKeepAliveScheduler();
        closeLatch.countDown();
    }


    public void sendConfigureMessage() {
        String configureMessage = String.format("""
            {
                "type": "Configure",
                "features": {
                    "asr": {
                        "language": "%s",
                        "interim_results": %b,
                        "vad_events": true,
                        "model": "nova-3",
                        "encoding": "%s",
                        "sample_rate": %d,
                        "channels": %d,
                        "utterance_end_ms": "1000",
                        "endpointing": 300,
                        "punctuate": true
                    }
                }
            }
            """, language, interimResults, encoding, sampleRate, channels);
        sendTextMessage(configureMessage);
        logger.info("Sent Configure message: {}", configureMessage);
    }

    public void sendKeepAliveMessage() {
        String keepAliveMessage = "{\"type\": \"KeepAlive\"}";
        sendTextMessage(keepAliveMessage);
        logger.debug("Sent KeepAlive message.");
    }

    public void sendFinalizeMessage() {
        String finalizeMessage = "{\"type\": \"Finalize\"}";
        sendTextMessage(finalizeMessage);
        logger.info("Sent Finalize message.");
    }

    public void sendCloseStreamMessage() {
        String closeStreamMessage = "{\"type\": \"CloseStream\"}";
        sendTextMessage(closeStreamMessage);
        logger.info("Sent CloseStream message.");
    }

    public void sendTextMessage(String message) {
        if (session != null && session.isOpen()) {
            logger.debug("Sending text message: {}", message);
            session.getAsyncRemote().sendText(message, result -> {
                if (result.getException() != null) {
                    logger.error("Failed to send text message: {}", result.getException().getMessage());
                } else {
                    logger.debug("Text message sent successfully.");
                }
            });
        } else {
            logger.warn("Session is not open. Unable to send text message.");
        }
    }

    public void sendAudio(byte[] audioData) {
        if (session != null && session.isOpen()) {
            logger.debug("Sending audio data of size: {} bytes", audioData.length);
            ByteBuffer buffer = ByteBuffer.wrap(audioData);
            session.getAsyncRemote().sendBinary(buffer, result -> {
                if (result.getException() != null) {
                    logger.error("Failed to send audio data: {}", result.getException().getMessage());
                } else {
                    logger.debug("Audio data sent successfully.");
                }
            });
        } else {
            logger.warn("Session is not open. Unable to send audio.");
        }
    }

    public boolean awaitClose(long timeout, TimeUnit unit) {
        try {
            return closeLatch.await(timeout, unit);
        } catch (InterruptedException e) {
            Thread.currentThread().interrupt();
            logger.error("Interrupted while waiting for WebSocket to close", e);
            return false;
        }
    }

    public void close() {
        if (session != null && session.isOpen()) {
            try {
                session.close();
            } catch (Exception e) {
                logger.error("Error while closing WebSocket session", e);
            }
        }
    }

    /**
     * Returns true if the WebSocket session is open.
     */
    public boolean isOpen() {
        return session != null && session.isOpen();
    }


    public void connect() throws DeepgramConnectException, DeploymentException, IOException, URISyntaxException {
        String endpoint = System.getProperty("deepgram.endpoint", "wss://api.deepgram.com/v1/listen");
        URI uri = new URI(endpoint);
        ClientEndpointConfig clientConfig = ClientEndpointConfig.Builder.create()
                .configurator(new ClientEndpointConfig.Configurator() {
                    @Override
                    public void beforeRequest(Map<String, List<String>> headers) {
                        headers.put("Sec-WebSocket-Protocol", Collections.singletonList("token," + apiKey));
                    }
                })
                .build();
        WebSocketContainer container = ContainerProvider.getWebSocketContainer();
        container.connectToServer(this, clientConfig, uri);
    }
}

Here is the code. I will share the requestId in some time.

using below libraries -

    <dependency>
        <groupId>jakarta.websocket</groupId>
        <artifactId>jakarta.websocket-api</artifactId>
        <version>2.2.0</version>
    </dependency>

    <dependency>
        <groupId>org.glassfish.tyrus.bundles</groupId>
        <artifactId>tyrus-standalone-client</artifactId>
        <version>2.2.0</version>
    </dependency>

for implementing websocket client.

[rostwal95]

{"message":"Connected to Deepgram WebSocket. SessionId: 5a6be801-be7f-4ed8-9312-550197dd52d5"}
{"message":"Sent Configure message: {\n "type": "Configure",\n "features": {\n "asr": {\n "language": "en-US",\n "interim_results": true,\n "vad_events": true,\n "encoding": "mulaw",\n "sample_rate": 8000,\n "channels": 1,\n "utterance_end_ms": "1000",\n "endpointing": 300,\n "punctuate": true\n }\n }\n}\n"}
{"message":"Received complete text message: {"type":"Results","channel_index":[0,1],"duration":3.48,"start":0.0,"is_final":true,"speech_final":true,"channel":{"alternatives":[{"transcript":"my phone number is one two three four five","confidence":0.99853516,"words":[{"word":"my","start":0.9069318,"end":1.1435227,"confidence":0.99902344},{"word":"phone","start":1.1435227,"end":1.3801136,"confidence":0.99853516},{"word":"number","start":1.3801136,"end":1.7744318,"confidence":0.9995117},{"word":"is","start":1.7744318,"end":2.16875,"confidence":0.99853516},{"word":"one","start":2.16875,"end":2.405341,"confidence":0.9995117},{"word":"two","start":2.405341,"end":2.6419318,"confidence":0.9995117},{"word":"three","start":2.6419318,"end":2.8785226,"confidence":0.9980469},{"word":"four","start":2.8785226,"end":3.1151135,"confidence":0.8989258},{"word":"five","start":3.1151135,"end":3.48,"confidence":0.9980469}]}]},"metadata":{"request_id":"306d2977-9220-4354-ae2b-65582c95e29d","model_info":{"name":"general","version":"2024-01-26.8851","arch":"base"},"model_uuid":"1ed36bac-f71c-4f3f-a31f-02fd6525c489"},"from_finalize":false}"}
{"message":"Processing Deepgram event of type: Results"}
{message":"Transcript received: 'my phone number is one two three four five' with is_final: true and speech_final: true"}

even if I am sending all the details in the configure message, I dont see any events back.

also if you see the results message. it has arch as base not nova-3 and name as general.

sessionId is nothing but the requestId, could you please check ?

[rostwal95]

Please let me know if there is any more info needed from my end. This is bit critical for us, need your help ASAP.

also from the documentation the message flow is not very much clear to me, if you could help with the message formats it will be easy for us to take things forward.