transcribestream still doesn't work

[ambitslix]

Describe the bug

aws-doc-sdk-examples fails with

./get_transcript
Starting...
ERROR: A complete signal was sent without the preceding empty frame.
Failed to create a stream
Failed to send an empty frame

Expected Behavior

Test should pass

Current Behavior

Exception:
ERROR: A complete signal was sent without the preceding empty frame.

Reproduction Steps

Run the

aws-doc-sdk-examples get_transcript test

Possible Solution

No response

Additional Information/Context

No response

AWS CPP SDK version used

1.10.48

Compiler and Version used

11.3.0

Operating System and version

Ubuntu

[jmklix]

Are you following this example?

[ambitslix]

Are you following this example?

Yes, definitely.

[jmklix]

Which version of Ubuntu are you using?
Can you enable logging and include them as well.

[ambitslix]

Sure,

lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 22.04.1 LTS
Release: 22.04
Codename: jammy

aws_sdk_2023-01-12-06.log

[jmklix]

I'm able to successfully run the sample with ubuntu 22.04.1 on an amazon ec2 instance. I see that the error you are getting is preceded by a `BadRequestException'. Have you set up your credentials correctly with the right permissions to use this service?

[github-actions]

Greetings! It looks like this issue hasn’t been active in longer than a week. We encourage you to check if this is still an issue in the latest release. Because it has been longer than a week since the last update on this, and in the absence of more information, we will be closing this issue soon. If you find that this is still a problem, please feel free to provide a comment or add an upvote to prevent automatic closure, or if the issue is already closed, please feel free to open a new one.

[ambitslix]

Hello,

I just re-checked my permissions in IAM. I have full access to AWSPolly and AWSTranscribe. Furthermore I have been testing AWSPolly without any issues. The same credentials should work for AWSTranscribe so I assume the issue must be elsewhere and not with credentials. I have also tested the online AWS transcription via the browser and it worked although I think the browser uses separate logon. However the aws-doc-sdk-example get_transcript still doesn't work, nor does my application using the transcription.

[ambitslix]

I'm able to successfully run the sample with ubuntu 22.04.1 on an amazon ec2 instance. I see that the error you are getting is preceded by a `BadRequestException'. Have you set up your credentials correctly with the right permissions to use this service?

Still getting this after checking all permissions are good. Just giving you a heads up.

[jmklix]

Hey when looking through your logs the first error I see is

[ERROR] 2023-01-12 06:02:31.074 CurlHttpClient [140663358203456] Curl returned error code 55 - Failed sending data to the peer

This is fairly a runtime dependent error and comes around because of dependencies, Are you running this from your laptop? are you running this from a EC2 instance? can you replicate your environment where this occurs in a docker image?
The other issues I see is

[ERROR] 2023-01-12 06:02:40.085 CurlHttpClient [140663358203456] Curl returned error code 28 - Timeout was reached

We set the curl option

curl_easy_setopt(handle, CURLOPT_LOW_SPEED_LIMIT, m_lowSpeedLimit)

Which is described by curl as

Pass a long as parameter. It contains the average transfer speed in bytes per second that the transfer should be below during CURLOPT_LOW_SPEED_TIME seconds for libcurl to consider it to be too slow and abort.
by default in the SDK this is 1, and will terminate your connection if you drop below 1 byte per second, if your runtime is constrained and you will allow for it to drop to lower than 1 byte per second you can disable this with

ClientConfiguration clientConfiguration;
clientConfiguration.lowSpeedLimit = 0;

Which will never terminate the connection. I would suggest trying this out.

[ambitslix]

This is running on a powerful desktop, my connection is 1Gb up and down. I tried the lowSpeedLimit and re-compiled but the exact same error occurs. Again AWS-polly works fine. I checked and get_transcript is linked with local version of curl which is very recent. The first error is actually what is below in the [DEBUG] section line 464.
aws_sdk_2023-03-10-21.log

Then the first [ERROR] section appears later:

[TRACE] 2023-03-10 21:47:57.585 ConcurrentStreamBuf [140078001153600] Stream characters in buffer: 1024
[DEBUG] 2023-03-10 21:47:57.586 CURL [140078001153600] (SSLDataOut) 5bytes
[DEBUG] 2023-03-10 21:47:57.586 CURL [140078001153600] (Text) Send failure: Broken pipe

[DEBUG] 2023-03-10 21:47:57.586 CURL [140078001153600] (Text) [CONN-0-0][CF-TCP] send(len=1053) -> -1, err=55

[DEBUG] 2023-03-10 21:47:57.586 CURL [140078001153600] (Text) OpenSSL SSL_write: Broken pipe, errno 32

[TRACE] 2023-03-10 21:47:57.586 ConcurrentStreamBuf [140078001153600] Stream characters in buffer: 1024
[DEBUG] 2023-03-10 21:47:57.586 CURL [140078001153600] (Text) Closing connection 0

[DEBUG] 2023-03-10 21:47:57.586 CURL [140078001153600] (Text) [CONN-0-0][CF-TCP] recv(len=5) -> 5, err=0

[DEBUG] 2023-03-10 21:47:57.586 CURL [140078001153600] (SSLDataIn) 5bytes
[DEBUG] 2023-03-10 21:47:57.586 CURL [140078001153600] (Text) [CONN-0-0][CF-TCP] recv(len=161) -> 161, err=0

[DEBUG] 2023-03-10 21:47:57.586 CURL [140078001153600] (Text) [CONN-0-0][CF-TCP] cf_socket_close(20) active

[DEBUG] 2023-03-10 21:47:57.586 CURL [140078001153600] (Text) socket_close(20)

[ERROR] 2023-03-10 21:47:57.586 CurlHttpClient [140078001153600] Curl returned error code 55 - Failed sending data to the peer
[DEBUG] 2023-03-10 21:47:57.587 CurlHandleContainer [140078001153600] Destroy curl handle: 0x7f6658008d50
[DEBUG] 2023-03-10 21:47:57.587 CurlHandleContainer [140078001153600] Created replacement handle and released to pool: 0x7f665805ef70
[DEBUG] 2023-03-10 21:47:57.587 AWSClient [140078001153600] Request returned error. Attempting to generate appropriate error codes from response
[ERROR] 2023-03-10 21:47:57.587 AWSJsonClient [140078001153600] HTTP response code: -1
Resolved remote host IP address: 54.212.20.242
Request ID: 
Exception name: 
Error message: curlCode: 55, Failed sending data to the peer
14 response headers:
content-type : application/vnd.amazon.eventstream
date : Fri, 10 Mar 2023 21:47:49 GMT
transfer-encoding : chunked
x-amzn-request-id : dc20554c-7d63-46f8-9a9a-248179fdade2
x-amzn-requestid : dc20554c-7d63-46f8-9a9a-248179fdade2
x-amzn-transcribe-enable-channel-identification : false
x-amzn-transcribe-enable-partial-results-stabilization : false
x-amzn-transcribe-identify-language : false
x-amzn-transcribe-identify-multiple-languages : false
x-amzn-transcribe-language-code : en-US
x-amzn-transcribe-media-encoding : pcm
x-amzn-transcribe-sample-rate : 8000
x-amzn-transcribe-session-id : 20fdc5a1-a621-4ae3-a7da-05e2ee123cc6
x-amzn-transcribe-show-speaker-label : false
[WARN] 2023-03-10 21:47:57.587 AWSClient [140078001153600] If the signature check failed. This could be because of a time skew. Attempting to adjust the signer.
[DEBUG] 2023-03-10 21:47:57.587 AWSClient [140078001153600] Server time is Fri, 10 Mar 2023 21:47:49 GMT, while client time is Fri, 10 Mar 2023 21:47:57 GMT
[WARN] 2023-03-10 21:47:57.587 AWSClient [140078001153600] Request failed, now waiting 0 ms before attempting again.
[DEBUG] 2023-03-10 21:47:57.588 AWSAuthEventStreamV4Signer [140078001153600] Canonical Header String: amz-sdk-invocation-id:C10385C0-326F-4318-9BF2-D6255B5E0A3E
amz-sdk-request:ttl=20230310T214752Z; attempt=2; max=11
content-type:application/vnd.amazon.eventstream
host:transcribestreaming.us-west-2.amazonaws.com
x-amz-api-version:2017-10-26
x-amz-content-sha256:STREAMING-AWS4-HMAC-SHA256-EVENTS
x-amz-date:20230310T214757Z
x-amzn-transcribe-language-code:en-US
x-amzn-transcribe-media-encoding:pcm
x-amzn-transcribe-sample-rate:8000

[jmklix]

Are you behind a corporate firewall or have some special network settings that might be blocking some of the requests?

[ambitslix]

Are you behind a corporate firewall or have some special network settings that might be blocking some of the requests?

I'm not, just personal firewall but even if I was I'm not aware of any special port requirements to use this AWS service? AWSPolly works. Or should I run the python version of this test to rule this out?

[jmklix]

Trying to repro it with python would be a good idea. I'm still unsure what might be causing this and if it does(or doesn't) happen with another sdk that would be helpful to know

[ambitslix]

Okay I've tested the following python transcription example:

https://github.com/awsdocs/aws-doc-sdk-examples/tree/main/python/example_code/transcribe

$python transcribe_basics.py

It works without any issues. Are you sure that you've tested the

aws-doc-sdk-examples/cpp/example_code/transcribe/get_transcript

./get_transcript 
Starting...
ERROR: A complete signal was sent without the preceding empty frame.
Failed to create a stream
Failed to send an empty frame
ERROR: A complete signal was sent without the preceding empty frame.
ERROR: A complete signal was sent without the preceding empty frame.

and it worked for you under:

lsb_release -a
No LSB modules are available.
Distributor ID:	Linuxmint
Description:	Linux Mint 21
Release:	21
Codename:	vanessa

[sbiscigl]

Yes we have tested and verified it with several platforms, one of them being AL2, specific docker file:

# Using offical Amazon Linux 2 image from public ECR
FROM public.ecr.aws/amazonlinux/amazonlinux:2

#Install java
RUN amazon-linux-extras enable corretto8 && yum install -y java-1.8.0-amazon-corretto

#Install g++
RUN yum groupinstall "Development Tools" -y

#Install required dependencies
RUN yum install -y curl-devel openssl-devel ninja-build maven cmake3 awscli jq clang gcc10 gcc10-c++ bison rsync

#Link cmake3 correctly
RUN ln -s /usr/bin/cmake3 /usr/bin/cmake

#Build from source
RUN git clone --recurse-submodules https://github.com/aws/aws-sdk-cpp && \
    cd aws-sdk-cpp && \
    mkdir build && \
    cd build && \
    cmake .. -G "Unix Makefiles" -DBUILD_ONLY="transcribestreaming;transcribe" && \
    make install

I see that you are using mint/ubuntu earlier. Are you able to replicate your dependencies in a dockerfile? that way we could replicate it as well?

[ambitslix]

Well I could replicate my dependencies in a docker but let me check and run your docker test to see if that works? Would I be able to run yours in my workstation?

[sbiscigl]

yeah, everything there you can/should be able to use, that would really be appreciated if you could give that a shot! give me a shout if that works or not. if it does work we can start narrowing down why your deps dont work.

[sbiscigl]

apologies, i spent a little more time on it and i can provide a more complete set of docker file and instructions now, the latter lacked how to run it. so given the dockerfile

# Using offical Amazon Linux 2 image from public ECR
FROM public.ecr.aws/amazonlinux/amazonlinux:2

#Install java
RUN amazon-linux-extras enable corretto8 && yum install -y java-1.8.0-amazon-corretto

#Install g++
RUN yum groupinstall "Development Tools" -y

#Install required dependencies
RUN yum install -y curl-devel openssl-devel ninja-build maven cmake3 awscli jq clang gcc10 gcc10-c++ bison rsync

#Link cmake3 correctly
RUN ln -s /usr/bin/cmake3 /usr/bin/cmake

#Build sdk from source
RUN git clone --recurse-submodules https://github.com/aws/aws-sdk-cpp && \
    cd aws-sdk-cpp && \
    mkdir build && \
    cd build && \
    cmake .. -G "Unix Makefiles" -DBUILD_ONLY="transcribestreaming;transcribe" && \
    make install

#Build transcribe samples
RUN git clone https://github.com/awsdocs/aws-doc-sdk-examples.git && \
    cd aws-doc-sdk-examples/cpp/example_code/transcribe && \
    mkdir build && \
    cd build && \
    cmake .. -G "Unix Makefiles" && \
    make

once you have that file from the dir you can run

docker build -t transcribe-issue .

followed by

docker run \
-e AWS_ACCESS_KEY_ID=your-access-key \
-e AWS_SECRET_ACCESS_KEY=your-secret-key \
-e AWS_SESSION_TOKEN=your-session-token \
--name transcribe-issue \
-it transcribe-issue /aws-doc-sdk-examples/cpp/example_code/transcribe/build/get_transcript

to actually run the example. let me know how that works for you.

[ambitslix]

That worked:

Starting...
[partial] But what
[partial] But what if somebody
[partial] But what if somebody decides?
[partial] But what if somebody decides to break it?
[partial] But what if somebody decides to break it? Be careful.
[partial] But what if somebody decides to break it? Be careful that you can
[partial] But what if somebody decides to break it? Be careful that you keep
[partial] But what if somebody decides to break it? Be careful that you keep adequate.
[partial] But what if somebody decides to break it? Be careful that you keep adequate coverage.
[partial] But what if somebody decides to break it? Be careful that you keep adequate coverage. But look for
[partial] But what if somebody decides to break it? Be careful that you keep adequate coverage, But look for places.
[partial] But what if somebody decides to break it? Be careful that you keep adequate coverage, but look for places to save.
[partial] But what if somebody decides to break it? Be careful that you keep adequate coverage but look for places to save money.
[partial] But what if somebody decides to break it? Be careful that you keep adequate coverage. But look for places to save money. Maybe it's
[partial] But what if somebody decides to break it? Be careful that you keep adequate coverage. But look for places to save money. Maybe it's taking long.
[partial] But what if somebody decides to break it? Be careful that you keep adequate coverage. But look for places to save money. Maybe it's taking longer to
[partial] But what if somebody decides to break it? Be careful that you keep adequate coverage. But look for places to save money. Maybe it's taking longer to get thing.
[partial] But what if somebody decides to break it? Be careful that you keep adequate coverage. But look for places to save money. Maybe it's taking longer to get things square.
[partial] But what if somebody decides to break it? Be careful that you keep adequate coverage. But look for places to save money. Maybe it's taking longer to get things squared away.
[partial] But what if somebody decides to break it? Be careful that you keep adequate coverage. But look for places to save money. Maybe it's taking longer to get things squared away.
[partial] But what if somebody decides to break it? Be careful that you keep adequate coverage. But look for places to save money. Maybe it's taking longer to get things squared away. Then the bankers
[partial] But what if somebody decides to break it? Be careful that you keep adequate coverage. But look for places to save money. Maybe it's taking longer to get things squared away. Then the bankers expect
[partial] But what if somebody decides to break it? Be careful that you keep adequate coverage. But look for places to save money. Maybe it's taking longer to get things squared away. Then the bankers expected
File: only 880 could be read
[partial] But what if somebody decides to break it? Be careful that you keep adequate coverage. But look for places to save money. Maybe it's taking longer to get things squared away. Then the bankers expected hiring the wife.
Successfully sent the empty frame
[partial] But what if somebody decides to break it? Be careful that you keep adequate coverage. But look for places to save money. Maybe it's taking longer to get things squared away. Then the bankers expected during the why, for once.
[partial] But what if somebody decides to break it? Be careful that you keep adequate coverage. But look for places to save money. Maybe it's taking longer to get things squared away. Then the bankers expected during the Y for one's company.
[Final] But what if somebody decides to break it? Be careful that you keep adequate coverage. But look for places to save money. Maybe it's taking longer to get things squared away. Then the bankers expected during the Y for one's company.
Done

I've looked and the containers version of curl is:

RUN yum info libcurl
 ---> Running in 0efd28e75b60
Loaded plugins: ovl, priorities
Installed Packages
Name        : libcurl
Arch        : x86_64
Version     : 7.88.1
...
Name        : openssl
Arch        : x86_64
Epoch       : 1
Version     : 1.0.2k

While the curl version I'm linking is master from github v8.0.0-dev and I'm linking with OpenSSL_1_1_1-stable branch. I would guess either of those two is the culprit?

[ambitslix]

...Update: I built openssl 1.0.2k and curl v7.88.1 for linking with and the same issue manifests. BTW I have two machines that exhibit the same issue one is Linux Mint, the other Ubuntu.

[sbiscigl]

I can run it on ubuntu with the following docker image (same instructions as above).

FROM public.ecr.aws/lts/ubuntu:22.04_stable

RUN apt-get update && \
  apt-get install build-essential cmake git libcurl4-openssl-dev zlib1g-dev libssl-dev -y

#Build sdk from source
RUN git clone --recurse-submodules https://github.com/aws/aws-sdk-cpp && \
    cd aws-sdk-cpp && \
    mkdir build && \
    cd build && \
    cmake .. -G "Unix Makefiles" -DBUILD_ONLY="transcribestreaming;transcribe" && \
    make install

#Build transcribe samples
RUN git clone https://github.com/awsdocs/aws-doc-sdk-examples.git && \
    cd aws-doc-sdk-examples/cpp/example_code/transcribe && \
    mkdir build && \
    cd build && \
    cmake .. -G "Unix Makefiles" && \
    make

i know that that is not the version of curl and open ssl you are referring to, but with this setup it it will work on ubuntu. If you could update this docker file to reflect your deps to reproduce it, it will be a lot easier for us to debug it as we will have a baseline of exactly what deps are wrong.

[ambitslix]

Ok I will try to build a docker image with the issue manifested. BTW I tried to build with the stock system openssl and curl and it also has the issue. So for now the versions of openssl and curl I've used originally to locally link has the issue, the systemwide(apt) libs has the issue and even if I link the same versions as in your AL2 docker libs also has the issue. So it seems that it works in the AL2 docker container but not when linked on my machine. This seems to suggest the issue is not with these two libraries. So it's either another dependency or it's a gcc bug in my version. But now if it works in your Ubuntu 22 docker I'm really lost as to what it could be...? I will try to build over the Ubuntu 22 docker.

[ambitslix]

...Ok it works after all when linked with the stock system libraries. What must have happened was that it was still picking up something from the locally built curl/openssl. So I think the issue is when I locally build curl/openssl something isn't quite right with the resulting libraries or linking sequence to aws-sdk. Actually I often see the following:

[ 95%] Linking CXX executable transcribestreaming-gen-tests
./transcribestreaming-gen-tests: error while loading shared libraries: libssl.so.1.0.0: cannot open shared object file: No such file or directory

Even though libssl.so is there in the local directory and CMAKE_PREFIX_PATH points to that lib dir. I will replicate this now in a container.

[ambitslix]

Here is the failing docker test case with some notes: I tried to wget the same versions of libs as in aptitude.

FROM public.ecr.aws/lts/ubuntu:22.04_stable

RUN apt-get update && \
# apt-get install build-essential cmake git libcurl4-openssl-dev zlib1g-dev libssl-dev -y
  apt-get install build-essential cmake git zlib1g-dev wget tar -y

RUN wget -c https://www.openssl.org/source/old/3.0/openssl-3.0.7.tar.gz && \
  tar xvzf openssl-3.0.7.tar.gz && \
  cd openssl-3.0.7 && \
  ./config shared && \
  make VERBOSE=1 -j8 && \
  make install

RUN wget -c https://curl.se/download/curl-7.88.1.tar.gz && \
  tar xvzf curl-7.88.1.tar.gz && \
  cd curl-7.88.1 && \
  ./configure --with-openssl=/usr/local --enable-versioned-symbols && \
  make VERBOSE=1 -j8 && \
  make install

#
#
# HACK: Necessary so aws-sdk finds openssl libs later
#
#
RUN cp -r /usr/local/lib64/* /usr/local/lib

#Build sdk from source
RUN git clone --recurse-submodules https://github.com/aws/aws-sdk-cpp
RUN cd aws-sdk-cpp;rm -rf build;mkdir build

#
# Something to try to force aws-sdk to find openssl without the "HACK" but it doesn't work
# RUN export OPENSSL_ROOT_DIR=/usr/local && cd aws-sdk-cpp/build && cmake .. -G "Unix Makefiles" -DCMAKE_PREFIX_PATH=/usr/local -DOPENSSL_ROOT_DIR=/usr/local/openssl -DBUILD_ONLY="transcribestreaming;transcribe"
RUN cd aws-sdk-cpp/build && cmake .. -G "Unix Makefiles" -DCMAKE_PREFIX_PATH=/usr/local -DBUILD_ONLY="transcribestreaming;transcribe"
RUN cd aws-sdk-cpp/build && make VERBOSE=1 -j8
RUN cd aws-sdk-cpp/build && make install

#Build transcribe samples
RUN git clone https://github.com/awsdocs/aws-doc-sdk-examples.git && \
    cd aws-doc-sdk-examples/cpp/example_code/transcribe && \
    mkdir build && \
    cd build && \
    cmake .. -G "Unix Makefiles" && \
    make

The workaround for now is to link with systemwide curl/openssl.

[sbiscigl]

ERROR: A complete signal was sent without the preceding empty frame.
Failed to create a stream
Failed to send an empty frame
ERROR: A complete signal was sent without the preceding empty frame.

thank you so much for providing this, happy to say i can replicate. we can start a investigation into why these deps and these versions cause this.

[ambitslix]

Sure, my guess is that the versions of those deps should not matter much, it's some kind of linkage problem.

[ambitslix]

...Update, I debugged my application with the workaround and I was perplexed to find that it still had this bug. Then I ran the get_transcript test from within my application and it worked. As I re-checked my logic I saw that I had prematurely called Aws::TranscribeStreamingService::Model::AudioStream::Close(), after with the my code was still writing to it. This has the same symptom as the bug. I fixed my code and now my application works. However I'm wondering if the bug isn't some kind of race condition which closes the stream prematurely. This can happen if the semaphores aren't initialized for example.

[jmklix]

Did you need any more help with transcribe streaming while using this sdk?

[ambitslix]

Not at the moment, however I would not consider this resolved. I'm simply using a workaround to make it work for now. The issue still exists as I demonstrated with the docker setup when the SDK is built locally from sources.

[gjasny]

The issue goes away if you build your local curl with http2 support (by adding nghttp2).

[jmklix]

Closing this issue as it seems to be working correctly now if you use the correct version of curl/openssl:

The workaround for now is to link with systemwide curl/openssl.

Please let us know if you are still running into any problems with transcribestream.

[github-actions]

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see.
If you need more assistance, please either tag a team member or open a new issue that references this one.
If you wish to keep having a conversation with other community members under this issue feel free to do so.