$client->uploadDirectory() always uploads every file, even when no changes

[curtiscarlson]

Please fill out the sections below to help us address your issue.

Version of AWS SDK for PHP?

3.133.5

Version of PHP (php -v)?

PHP 7.2.21

What issue did you see?

when running uploadDirectory(), it creates a new version of every file in the folder

Steps to reproduce

If you have a runnable example, please include it as a snippet or link to a repository/gist for larger code examples.

No, just the basic function with a local source folder and a bucket destination, no other options or arguments

Additional context

Any additional information relevant to the issue. Examples include any framework you may be using (e.g. Laravel, Wordpress) in conjunction with the AWS SDK for PHP, or PHP/environment config settings if the issue is related to memory or performance.

windows / xampp

using slim/pimple and env, my setup looks like this

<?php

namespace App\Providers;

use Pimple\Container;
use Pimple\ServiceProviderInterface;
use Aws\S3\S3Client;

class S3ServiceProvider implements ServiceProviderInterface
{
     public function register(Container $container) {

        // set up s3
        $container['s3client'] = function() {
            $client = new S3Client([
                'credentials' => [
                    'key'    => $_ENV['AWS_ACCESS_KEY_ID'],
                    'secret' => $_ENV['AWS_SECRET_ACCESS_KEY'],
                ],
                'region' => $_ENV['AWS_DEFAULT_REGION'],
                'version' => 'latest',
            ]);

            return $client;
        };

   }
}

Symfony console command:

<?php

namespace App\Commands;

use Symfony\Component\Console\Command\Command;
use Symfony\Component\Console\Input\InputInterface;
use Symfony\Component\Console\Output\OutputInterface;
use Symfony\Component\Console\Question\Question;
use curtiscarlson\phplibs\ServiceLocator;

Class BackupStorage extends Command {

  protected function configure() {
    $this->setName('storage:backup')
      ->setDescription('Syncs the /storage folder to S3')
      ->setHelp('Syncs the /storage folder to S3');
  }

  public function execute(InputInterface $input, OutputInterface $output) {


      $client = ServiceLocator::get('s3client');

      $client->uploadDirectory(ROOT . '/storage', 'mybucketname');

  }

}

[curtiscarlson]

could it be a permissions issue?

[curtiscarlson]

To answer the question above, it is not a permissions issue. I added administrator rights to the user and the outcome is the same.

[diehlaws]

Hi @curtiscarlson, thanks for reaching out to us. It sounds like you're expecting uploadDirectory() to behave in a manner similar to the s3 sync command in the AWS CLI; unfortunately the AWS SDK for PHP does not have a function similar to this CLI command.

Instead, uploadDirectory() intentionally uploads every file in the provided directory regardless of the contents of the destination prefix in the destination S3 Bucket. If you have versioning enabled on the bucket, this will result in new versions of existing objects being created when local files are re-uploaded to the bucket. Depending on your use case you could configure a lifecycle policy for the prefix to which your files are being re-uploaded in order to minimize the number of versions for these objects.

[curtiscarlson]

@diehlaws Thank you for your response. Are you sure though?

This page: https://docs.aws.amazon.com/aws-sdk-php/v2/guide/service-s3.html states "The uploadDirectory() method of a client will compare the contents of the local directory to the contents in the Amazon S3 bucket and only transfer files that have changed. "

The page also explains that the function takes an options array, one of which is 'force', defined as "Set to true to upload every file, even if the file is already in Amazon S3 and has not changed." Which implies that the default is false. I have tried setting this option to false manually and it doesn't do anything.

Finally, the page explains that "The uploadDirectory() method is an abstraction layer over the much more powerful Aws\S3\Sync\UploadSyncBuilder". Which means to imply that it is indeed similar to the s3 sync command. (It is also all under the "Syncing data with Amazon S3" heading).

Thanks again.

[diehlaws]

No problem! The documentation page you mentioned is for version 2 of the AWS SDK for PHP. It looks like this function was changed with the release of version 3 of the AWS SDK for PHP so that, instead of iterating through existing objects in the destination bucket/prefix to only copy new and changed files from the local directory, it copies the full contents of the given directory regardless of the contents of the destination. Unfortunately it does not look like a new function was ever created in V3 to implement the functionality present in the V2 S3Client's UploadDirectory() function.

That being said, you can provide an iterator to the S3 Transfer's constructor method to then call its transfer() function so that only new/changed files are transferred to the destination bucket. Alternately, if you'd like to see a sync function implemented in the S3 Client for V3 we can mark this as a feature request for future implementation - unfortunately we have a number of higher priority items at this time so implementation for this may be slow without external contributions.

[curtiscarlson]

@diehlaws I think both of your suggestions are probably over my head. I tried to find the V2 code on github but I would have to click for hours to get back that far. I want to say that I think this functionality is extremely important and that you should reconsider its priority, for the following reasons

• There is no purpose whatsoever to uploading every file every time and making a new version in place (that I can see)
• It's bad for the environment, bad for users bills, bad for internet traffic, and bad for site availability
• It's already been documented to work this way

If you point me in the right direction to the original code I might be able to make the contribution but as I said I might be over my head. Thank you.

[diehlaws]

Thanks for the feedack @curtiscarlson! I've marked the issue as a feature request and will bring it up with the rest of the team in the next team sprint.

You can review the code for V2 of the SDK by switching branches in this repo from master to 2.8. Please keep in mind that there are some pretty significant changes both to the core SDK and the S3 client between v2 and v3 of the AWS SDK for PHP, so simply porting the code for v2's uploadDirectory to a new function in v3 likely won't work as expected.

[alexwai]

Any update on this issue

[curtiscarlson]

@alexwai
I found a sync package on packagist and updated it to meet my needs.

[danielerne]

Very disappointed that this was removed in v3. We upload hundreds of files every 30 minutes.

[stobrien89]

Hi all,

We've had a number of conversations about this as a team and we've decided that the PHP SDK will not be implementing this feature. At least until the SDKs have settled on a shared specification of a sync-like customization. Closing for now, but happy to answer any questions you might have.

[github-actions]

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see.
If you need more assistance, please either tag a team member or open a new issue that references this one.
If you wish to keep having a conversation with other community members under this issue feel free to do so.