Scan interval with documentdb source

[ralimardanov]

Hello everyone,
I don't see any option to change scan interval with documentdb as a source. Can anyone help me out, please?
I tried such configuration, but it doesn't work:

version: "2"
documentdb-pipeline:
  source:
    documentdb:
      interval: "30s"
      acknowledgments: true
      host: "<documentbd_db_url>
      port: 27017
      authentication:
        username: "${{aws_secrets:secret:username}}"
        password: "${{aws_secrets:secret:password}}"
      aws:
        sts_role_arn: "" 
      s3_bucket: "test" 
      s3_region: ""
      s3_prefix: "test-prefix" 
      collections:
        - collection: "" 
          export: false
          stream: true
    s3: 
      aws:
        region: ""
        sts_role_arn: ""
      scan:
        scheduling:
          interval: "30s"
        buckets:
          - bucket:
              name: "test"
              filter:
                include_suffix:
                  - "test-prefix"
  sink:
    - opensearch:
        hosts:
          - ""

        index: ""
        index_type: custom
        document_id: "${getMetadata(\"primary_key\")}"
        action: "${getMetadata(\"opensearch_action\")}"
        aws:
          sts_role_arn: ""
          region: ""
        dlq:
          s3:
            bucket: ""
            key_path_prefix: ""
            region: "eu-west-1"
            sts_role_arn: "
        max_retries: 10
        bulk_size: 5
extension:
  aws:
    secrets:
      secret:
        secret_id: ""
        region: ""
        sts_role_arn: ""
        refresh_interval: PT1H
 

I still keep getting this:

2024-08-29T10:21:30.546 [s3-source-scan-1] INFO org.opensearch.dataprepper.plugins.source.s3.S3ScanPartitionCreationSupplier - Skipping scan as the interval of 60 seconds has not been reached yet |  

[dinujoh]

@ralimardanov Thanks for your interest in Data Prepper and trying out documentdb source plugin.

We currently don't support configuration option to change the scan interval as part of the documentdb source plugin.

The documentdb pipeline configuration is transformed to two pipelines: (1) DocumentDB source and S3 Sink (2) S3 Source and Opensearch Sink.

The transformation template is defined here

You can see the 2nd pipeline has S3 scan source with configuration scheduling: interval: "60s".

This is not currently configurable in DocumentDB source. However you can change the above setting in the template file and build the source and run data prepper.