Putting objects from macOS is slower than Windows

[ShimYama]

Problem

I'm working on a software for mac OS that reads text files in S3 bucket, processes them, and uploads and updates them periodically. However, the time required for uploading is too slow to keep up with the frequency of updates.
When uploading a file of similar size on Windows, the upload finishes sooner than macOS.
I think the problem stems from the different HTTP clients used by the OS (Windows: WinHttpClient, macOS: curl). However, so far, changing various parameters in ClientConfiguration has had almost no effect. (use transfer acceleration, set disableExpectHeader true, and change maxConnections etc.)

I modified put_object sample source and got these results.
Avarage Time to upload a text file (size 6.7 KB)

• Windows: about 120 ms
• macOS: about 300 ms

However, uploading files with curl library (not to S3 bucket) on mac is not so slow as aws sdk. (It takes about 100-150 ms to upload the same text file)
So I think my internet speed from my mac is not too slow.

Is there any idea to improve upload speed?

Enviroment

• common
  • SDK version: 1.11.155
• Windows
  • Windows 10 Pro 22H2
  • Visual Studio 2015 Community / Build Engine version 14.0.25420.1
• mac
  • MacBookPro (13-inch, 2019)
  • macOS 12.6.7
  • cmake version 3.24.2

Code sample

modified put_object.cpp with measuring upload time.
I didn't edit AwsDoc::S3::PutObject()

int64_t currentTimeMillis() {
    static int64_t correction = 0;
#if defined(_WIN32)
    struct _timeb t;
    _ftime(&t);
    correction = (((int64_t) t.time) * 1000 + t.millitm);
#else
    struct timeval tv;
    struct timezone tz;
    gettimeofday(&tv, &tz);
    correction = (((int64_t) tv.tv_sec) * 1000 + tv.tv_usec / 1000);
#endif
    return correction;
}

class ElapsedTime {
public:
    ElapsedTime() { reset(); }
    virtual ~ElapsedTime() {}
    void reset() { mStartMS = currentTimeMillis(); }
    int64_t	get() const { return currentTimeMillis() - mStartMS; }
private:
    int64_t	mStartMS;
};

int main() {
    Aws::SDKOptions options;
    Aws::InitAPI(options);
    {
        //TODO(user): Change bucket_name to the name of a bucket in your account.
        const Aws::String bucket_name = "<my-bucket-name>";
        //TODO(user): Create a file called "my-file.txt" in the local folder where your executables are built to.
        const Aws::String object_name = "sample.txt";

        Aws::Client::ClientConfiguration clientConfig;
        // Optional: Set to the AWS Region in which the bucket was created (overrides config file).
        clientConfig.region = "<my-bucket-region>";

        ElapsedTime timer;
        AwsDoc::S3::PutObject(bucket_name, object_name, clientConfig);
        std::cout << "Upload Time: " << timer.get() << " ms" << std::endl;
    }

    Aws::ShutdownAPI(options);

    return 0;
}

[yasminetalby]

@github-actions proposed-answer
Hello @ShimYama ,

Thank you very much for your submission.

First of all if you would like to see improvement in speed performances for object upload I would highly recommend you using the S3 CRT Client (see). This might be a good workaround for you.

There are multiple reasons why you can witness speed performance differences that would be proper to your environment. I'll attempt to reproduce this to see if I can notice a difference in speed performances on my end.

I also wanted to share this general documentation to troubleshoot slow speeds when downloading/uploading to Amazon s3 : https://repost.aws/knowledge-center/s3-troubleshoot-slow-downloads-uploads

Best regards,

Yasmine

[github-actions[bot]]

Hello! A team member has marked the above comment as the likely answer to this discussion thread.

• If you agree, please upvote that comment, or click on Mark as answer. I will automatically mark the comment as the answer next time I check.

• If this answer does not help you, please downvote the answer instead and let us know why it was not helpful. I will add a label to this discussion to gain attention from the team.

[ShimYama]

I replaced S3Client with S3CrtClient and the performance improves as Windows one.
Thank you for your answer!

[github-actions[bot]]

Hello! Reopening this discussion to make it searchable.