bug: delegation token parameter should be renamed when using WebHDFS as storage

[jason-heo]

Search before asking

• I had searched in the issues and found no similar issues.

Version

v1.2.615-a8da519a63(rust-1.81.0-nightly-2024-08-19T03:29:16.412442905Z)

What's Wrong?

Hello, Im new to Databend.

I'm trying to test Databend with my data on secure hadoop.

I've setup Query node as describe in Deploying with HDFS using WebHDFS

My current setting is like this:

[storage.webhdfs]
endpoint_url = "https://<hostname>:<port>"
root = "/path/to"
# if your webhdfs needs authentication, uncomment and set with your value
delegation = "<token>"

When I run INSERT INTO I got an error:

error: APIError: ResponseError with 3002: PermissionDenied (persistent) at Writer::close, context: {
  uri: https://<hostname>:<port>/webhdfs/v1/path/to/xxx.parquet?op=CREATE&...&delegation_token=<token>,
  response: Parts {
    status: 401,
    version: HTTP/1.1,
    headers: {
      ...
    }
  }
}

I think delegation_token should be renamed to delegation.

You can see not delegation_token but delegation on WebHDFS REST API -> Authentication

I've tested using curl and confirm that following url worked well.

$ curl -i "https://<hostname>:<port>/webhdfs/v1/path/to?op=LISTSTATUS&delegation=<token>"

How to Reproduce?

Described above.

Are you willing to submit PR?

• Yes I am willing to submit a PR!

[jason-heo]

It seems that this issue is related to OpenDAL.

After checking reproducable via OpenDAL API, let me report to OpenDAL.

[Xuanwo]

Thank you @jason-heo for this. Maybe we can also accpet delegation during parsing the url.

[Xuanwo]

This change should be simple that parse delegation as delegation_token.

[jason-heo]

Hello, @Xuanwo

If my understanding is correct, to resolve this isssue one needs to replace delegation_token to delegation in url parameter in some codes.

Could you point me out which source code should be modified for this issue?

I tried to find delegation_token but it seems that only test codes have delegation_token.

git grep -i delegation_token
src/meta/proto-conv/tests/it/user_proto_conv.rs:                delegation: "<delegation_token>".to_string(),
src/meta/proto-conv/tests/it/user_stage.rs:                delegation: "<delegation_token>".to_string(),
src/meta/proto-conv/tests/it/v030_user_stage.rs:                delegation: "<delegation_token>".to_string(),
src/meta/proto-conv/tests/it/v031_copy_max_file.rs:                delegation: "<delegation_token>".to_string(),

Although my background is Java, Scala and Python, I'll try to resolve this is if simple.

Sorry, my team's task priority has been changed.

Thanks.