[Bug] - posix "link tmpfile" multipart invalid cross-device link with zfs

[lunixbochs]

Describe the bug
versitygw creates multipart temp files at bucket/.sgwtmp/multipart and tries to hardlink them into place
on zfs it's common to have nested datasets e.g. pool/dataset and pool/dataset/inner, which are separate mounts
This means versitygw will attempt a cross-device link during multipart uploads

To Reproduce
Create a posix bucket where the root is on a different filesystem mount than a subdir, then do a multipart upload to the subdir

Expected behavior
The multipart upload succeeds.

Server Version
$ versitygw -version && uname -a
Version : v1.0.7
Build : 162944d
BuildTime: 2024-10-27_07:28:49PM
Linux nas1 5.15.0-97-generic #107-Ubuntu SMP Wed Feb 7 13:26:48 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux

Additional context
s3 client is boto3

[benmcclelland]

I think the reason for the failure is because we are using O_TMPFILE to fill out all the object data and then linking it into the namespace once the data/metadata are in place. And if the object parent directory is a different mount than the tmpfile file descriptor, then this will fail. I would expect non multipart uploads to fail in a similar way too? I think we can fix this by using the object parent dir (or closest existing ancestor dir) for generating the tmpfile file descriptor.