cloudstorage: fix a bug that may cause storage sink get stuck (#12142)

[ti-chi-bot]

This is an automated cherry-pick of #12142

What problem does this PR solve?

Issue Number: close #9162

What is changed and how it works?

Check List

Tests

• Unit test
• Manual test

1. I created a container on Azure.
   

2. I modified the WriteFile function as follows to make it easier to throw a context timeout error.

func (s *extStorageWithTimeout) WriteFile(ctx context.Context, name string, data []byte) error {
	timeout := s.timeout
	if counter.Load()%10 == 0 {
		// Make it more likely to timeout
		timeout = 5 * time.Millisecond
	}
	ctx, cancel := context.WithTimeout(ctx, timeout)
	counter.Add(1)
	defer cancel()
	return s.ExternalStorage.WriteFile(ctx, name, data)
}

3. I compiled the modified CDC and created a changefeed to synchronize data to Azure Blob Storage.

./cdc cli changefeed create -c test --sink-uri="azure://dongmen-test/cdc?protocol=canal-json&account-name=${}account-key=${}" 

4. I wrote data to TiDB.

./workload -database-host 127.0.0.1 -database-port 4000 -database-db-name "test" -table-count 4 -workload-type large_row -total-row-count 10000 -action prepare -thread 1

5. I observed the following warning message from the changefeed.

./cdc cli changefeed list
[
  {
    "id": "test",
    "namespace": "default",
    "summary": {
      "state": "warning",
      "tso": 457412143773646869,
      "checkpoint": "2025-04-17 19:20:54.117",
      "error": {
        "time": "2025-04-17T19:21:20.841014+08:00",
        "addr": "127.0.0.1:8300",
        "code": "CDC:ErrProcessorUnknown",
        "message": "Failed to write azure blob file, file info: bucket(container)='dongmen-test', key='cdc/test/large_row_1/457412143747432452/2025-04-17/CDC00000000000000000001.json': context deadline exceeded"
      }
    }
  }
]

6. The changefeed retried the operation in the sink module when encountering this error and quickly recovered.

> ./cdc cli changefeed list
[
  {
    "id": "test",
    "namespace": "default",
    "summary": {
      "state": "normal",
      "tso": 457412177498472454,
      "checkpoint": "2025-04-17 19:23:02.767",
      "error": null
    }
  }
]

7. I checked the Azure Blob Storage, and all the data was synchronized correctly.

The above test process verifies that the underlying calls of the WriteFile function respect the context with a timeout, which validates the effectiveness of the fix.

Moreover, this test verifies that the changefeed can quickly recover from similar errors.

Questions

Will it cause performance regression or break compatibility?

Do you need to update user documentation, design documentation or monitoring documentation?

Release note

Fix a bug that may cause changefeed with storage sink getting stuck.

[ti-chi-bot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: asddongmen

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [asddongmen]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment