Open
Conversation
Jo-stfc
pushed a commit
that referenced
this pull request
Feb 18, 2025
* Buffer implementation for XrdCeph * Better error return code values * Add timing into BufferIO * Add timing into BufferSimple * Utils code area * Update raw data access and copy * Adding Extents * ReadV simple logic * Add to own files the readV implementations * Add to own files the readV implementations; cmake updated * Logging improvements and write buffer updates * Add IOadapter with blocking aio access * Use IOadapter with blocking aio access * Small logging update * Reduce logging information; fix timeing to ms * Reduce logging information; * Reduced logging, and better use of aggregated metrics * comment clean and typo fixes * Remove uncessary file close * Additional logging in case of problems * Additional logging in case of problems * allow option for buffering with IO or AIO buffer Co-authored-by: james <james.walder@stfc.ac.uk> Co-authored-by: root <root@host-172-16-112-239.nubes.stfc.ac.uk> merge variable rpm name into bufferedIO (#19) * variable rpm name * Update xrootd-ceph.spec.in * Update makesrpm.sh * Update makesrpm.sh Fixes to remove warnings from devtoolset-9 compilation Master buffered ceph io (#20) * Buffer implementation for XrdCeph * Better error return code values * Add timing into BufferIO * Add timing into BufferSimple * Utils code area * Update raw data access and copy * Adding Extents * ReadV simple logic * Add to own files the readV implementations * Add to own files the readV implementations; cmake updated * Logging improvements and write buffer updates * Add IOadapter with blocking aio access * Use IOadapter with blocking aio access * Small logging update * Reduce logging information; fix timeing to ms * Reduce logging information; * Reduced logging, and better use of aggregated metrics * comment clean and typo fixes * Remove uncessary file close * Additional logging in case of problems * Additional logging in case of problems * allow option for buffering with IO or AIO buffer * fix conflicts * Allow for finite retries on EBUSY, else fail with EIO. It is possible for a read/write from the buffer to return EBUSY due to an underlying issue. In these cases, if the -EBUSY is returned out of XrdCeph, a large number of retries can originate. It is better at this point for the transfer to be flagged as failed, and retried properly. The code allows for 5 retries with a 1s sleep between them. If this doesn't work - which it might not - then an -EIO error is returned to xrootd. Other error messages are not affected. * Better summary stats output for CephIOAdapterRaw * Comment out a comment Co-authored-by: james <james.walder@stfc.ac.uk> Co-authored-by: root <root@host-172-16-112-239.nubes.stfc.ac.uk> variable version/release for template (#21) Update bufferedIO with updates from master (#26) * variable rpm name (#17) * variable rpm name * Update xrootd-ceph.spec.in * Update makesrpm.sh * Update makesrpm.sh * Master cephnamelib (#16) * Allow ceph.namelib to take params and apply translation to full path * Reduce logging Remove extraneous logging messages * simplify parsing of namelib and added a log line for any remapped file Co-authored-by: James <James.walder@stfc.ac.uk> * XRD-22 Fix ensuring the correct filename is passed to the CephFile instance. (#24) A regression in previous commit meant that the filename was not correctly passed to the CephFile instance. This fix ensures that the filename is set correctly. Co-authored-by: james <james.walder@stfc.ac.uk> * re-introduce variable names to spec input (#27) Co-authored-by: Jo-stfc <71326101+Jo-stfc@users.noreply.github.com> Co-authored-by: James <James.walder@stfc.ac.uk> Decreased logging for bufferedIO operations. (#25) Reduced printouts. Only summary stats now produced, rather than the logging per read. Co-authored-by: James Walder <james.walder@stfc.ac.uk> Updates from master to buffered io needed for 550 2 (#32) * XRD-12 Add timestamp information for ceph logging methods Update the logwrapper method to print out the current timestamp in the initial section of output. * Return permission denied on write attempt on existing file with EXCL set (#31) Co-authored-by: James Walder <james.walder@stfc.ac.uk> * disable posc (#30) posc is disabled for proxies, but not for a unified setup. XrdCeph does not support the posc flag as it misinterprets objects as folders Co-authored-by: James Walder <james.walder@stfc.ac.uk> Co-authored-by: Jo-stfc <71326101+Jo-stfc@users.noreply.github.com> Buffered io multibuffers (#38) * Add multiple buffer support for reads in case of simultaneous threads reading the same file. * Further refinements to the simultaneous file reads code - Ensure all relevent read / write methods will create a buffer if needed - Validty check on close that a buffer was actually created (or bypass code if not) - Bugfix in case of odd read sizes combined with multi/split buffer reads (critical) - Clean of comments included for development * Enhanced logging for cluster metrics and readV layer improvments (#35) - dumpCLusterInfo to check on the rados connection info - extra logging in a delete to give info on delete times - update the readV basic alg to do a simple bulk request Co-authored-by: James Walder <james.walder@stfc.ac.uk> * Add time taken to unlink a file in the logging message - Logging an unlink now includes the time taken, in cases of (un)successful deletes - Remove some extraneous comments * - Fix issue with buffer passthrough read - Add maximum number of simultaneous buffers for a given file Once a given number of opens have been made against the same file, don't create a large buffer, and only create a 1MiB buffer for each new file. This should avoid issues with small paged reads, but would normally hope the pasthrough mode would be triggered in each read. * Additional statistics on buffered reading added. - Will report bytes read from ceph, bytes read but bypassed the cache, and the cache hit fraction --------- Co-authored-by: James Walder <james.walder@stfc.ac.uk> Bug fix for writes with bufferedIO when extending over buffer range. (#40) * Bug fix for writes with bufferedIO when extending over buffer range. - Fix for case where multiple writes to the buffer are needed for a given xrd write request - Previously threw an error; now will correctly perform the multiple writes as required. - Set the Simple Data buffer capacity to the input size, rather than the capacity of the vector, which could be larger. --------- Co-authored-by: James Walder <james.walder@stfc.ac.uk> variable rpm name (#17) * variable rpm name * Update xrootd-ceph.spec.in * Update makesrpm.sh * Update makesrpm.sh re-introduce variable names to spec input (#27)
Jo-stfc
pushed a commit
that referenced
this pull request
Feb 19, 2025
* Buffer implementation for XrdCeph * Better error return code values * Add timing into BufferIO * Add timing into BufferSimple * Utils code area * Update raw data access and copy * Adding Extents * ReadV simple logic * Add to own files the readV implementations * Add to own files the readV implementations; cmake updated * Logging improvements and write buffer updates * Add IOadapter with blocking aio access * Use IOadapter with blocking aio access * Small logging update * Reduce logging information; fix timeing to ms * Reduce logging information; * Reduced logging, and better use of aggregated metrics * comment clean and typo fixes * Remove uncessary file close * Additional logging in case of problems * Additional logging in case of problems * allow option for buffering with IO or AIO buffer Co-authored-by: james <james.walder@stfc.ac.uk> Co-authored-by: root <root@host-172-16-112-239.nubes.stfc.ac.uk> merge variable rpm name into bufferedIO (#19) * variable rpm name * Update xrootd-ceph.spec.in * Update makesrpm.sh * Update makesrpm.sh Fixes to remove warnings from devtoolset-9 compilation Master buffered ceph io (#20) * Buffer implementation for XrdCeph * Better error return code values * Add timing into BufferIO * Add timing into BufferSimple * Utils code area * Update raw data access and copy * Adding Extents * ReadV simple logic * Add to own files the readV implementations * Add to own files the readV implementations; cmake updated * Logging improvements and write buffer updates * Add IOadapter with blocking aio access * Use IOadapter with blocking aio access * Small logging update * Reduce logging information; fix timeing to ms * Reduce logging information; * Reduced logging, and better use of aggregated metrics * comment clean and typo fixes * Remove uncessary file close * Additional logging in case of problems * Additional logging in case of problems * allow option for buffering with IO or AIO buffer * fix conflicts * Allow for finite retries on EBUSY, else fail with EIO. It is possible for a read/write from the buffer to return EBUSY due to an underlying issue. In these cases, if the -EBUSY is returned out of XrdCeph, a large number of retries can originate. It is better at this point for the transfer to be flagged as failed, and retried properly. The code allows for 5 retries with a 1s sleep between them. If this doesn't work - which it might not - then an -EIO error is returned to xrootd. Other error messages are not affected. * Better summary stats output for CephIOAdapterRaw * Comment out a comment Co-authored-by: james <james.walder@stfc.ac.uk> Co-authored-by: root <root@host-172-16-112-239.nubes.stfc.ac.uk> variable version/release for template (#21) Update bufferedIO with updates from master (#26) * variable rpm name (#17) * variable rpm name * Update xrootd-ceph.spec.in * Update makesrpm.sh * Update makesrpm.sh * Master cephnamelib (#16) * Allow ceph.namelib to take params and apply translation to full path * Reduce logging Remove extraneous logging messages * simplify parsing of namelib and added a log line for any remapped file Co-authored-by: James <James.walder@stfc.ac.uk> * XRD-22 Fix ensuring the correct filename is passed to the CephFile instance. (#24) A regression in previous commit meant that the filename was not correctly passed to the CephFile instance. This fix ensures that the filename is set correctly. Co-authored-by: james <james.walder@stfc.ac.uk> * re-introduce variable names to spec input (#27) Co-authored-by: Jo-stfc <71326101+Jo-stfc@users.noreply.github.com> Co-authored-by: James <James.walder@stfc.ac.uk> Decreased logging for bufferedIO operations. (#25) Reduced printouts. Only summary stats now produced, rather than the logging per read. Co-authored-by: James Walder <james.walder@stfc.ac.uk> Updates from master to buffered io needed for 550 2 (#32) * XRD-12 Add timestamp information for ceph logging methods Update the logwrapper method to print out the current timestamp in the initial section of output. * Return permission denied on write attempt on existing file with EXCL set (#31) Co-authored-by: James Walder <james.walder@stfc.ac.uk> * disable posc (#30) posc is disabled for proxies, but not for a unified setup. XrdCeph does not support the posc flag as it misinterprets objects as folders Co-authored-by: James Walder <james.walder@stfc.ac.uk> Co-authored-by: Jo-stfc <71326101+Jo-stfc@users.noreply.github.com> Buffered io multibuffers (#38) * Add multiple buffer support for reads in case of simultaneous threads reading the same file. * Further refinements to the simultaneous file reads code - Ensure all relevent read / write methods will create a buffer if needed - Validty check on close that a buffer was actually created (or bypass code if not) - Bugfix in case of odd read sizes combined with multi/split buffer reads (critical) - Clean of comments included for development * Enhanced logging for cluster metrics and readV layer improvments (#35) - dumpCLusterInfo to check on the rados connection info - extra logging in a delete to give info on delete times - update the readV basic alg to do a simple bulk request Co-authored-by: James Walder <james.walder@stfc.ac.uk> * Add time taken to unlink a file in the logging message - Logging an unlink now includes the time taken, in cases of (un)successful deletes - Remove some extraneous comments * - Fix issue with buffer passthrough read - Add maximum number of simultaneous buffers for a given file Once a given number of opens have been made against the same file, don't create a large buffer, and only create a 1MiB buffer for each new file. This should avoid issues with small paged reads, but would normally hope the pasthrough mode would be triggered in each read. * Additional statistics on buffered reading added. - Will report bytes read from ceph, bytes read but bypassed the cache, and the cache hit fraction --------- Co-authored-by: James Walder <james.walder@stfc.ac.uk> Bug fix for writes with bufferedIO when extending over buffer range. (#40) * Bug fix for writes with bufferedIO when extending over buffer range. - Fix for case where multiple writes to the buffer are needed for a given xrd write request - Previously threw an error; now will correctly perform the multiple writes as required. - Set the Simple Data buffer capacity to the input size, rather than the capacity of the vector, which could be larger. --------- Co-authored-by: James Walder <james.walder@stfc.ac.uk> variable rpm name (#17) * variable rpm name * Update xrootd-ceph.spec.in * Update makesrpm.sh * Update makesrpm.sh re-introduce variable names to spec input (#27)
Jo-stfc
pushed a commit
that referenced
this pull request
Feb 21, 2025
* Buffer implementation for XrdCeph * Better error return code values * Add timing into BufferIO * Add timing into BufferSimple * Utils code area * Update raw data access and copy * Adding Extents * ReadV simple logic * Add to own files the readV implementations * Add to own files the readV implementations; cmake updated * Logging improvements and write buffer updates * Add IOadapter with blocking aio access * Use IOadapter with blocking aio access * Small logging update * Reduce logging information; fix timeing to ms * Reduce logging information; * Reduced logging, and better use of aggregated metrics * comment clean and typo fixes * Remove uncessary file close * Additional logging in case of problems * Additional logging in case of problems * allow option for buffering with IO or AIO buffer Co-authored-by: james <james.walder@stfc.ac.uk> Co-authored-by: root <root@host-172-16-112-239.nubes.stfc.ac.uk> merge variable rpm name into bufferedIO (#19) * variable rpm name * Update xrootd-ceph.spec.in * Update makesrpm.sh * Update makesrpm.sh Fixes to remove warnings from devtoolset-9 compilation Master buffered ceph io (#20) * Buffer implementation for XrdCeph * Better error return code values * Add timing into BufferIO * Add timing into BufferSimple * Utils code area * Update raw data access and copy * Adding Extents * ReadV simple logic * Add to own files the readV implementations * Add to own files the readV implementations; cmake updated * Logging improvements and write buffer updates * Add IOadapter with blocking aio access * Use IOadapter with blocking aio access * Small logging update * Reduce logging information; fix timeing to ms * Reduce logging information; * Reduced logging, and better use of aggregated metrics * comment clean and typo fixes * Remove uncessary file close * Additional logging in case of problems * Additional logging in case of problems * allow option for buffering with IO or AIO buffer * fix conflicts * Allow for finite retries on EBUSY, else fail with EIO. It is possible for a read/write from the buffer to return EBUSY due to an underlying issue. In these cases, if the -EBUSY is returned out of XrdCeph, a large number of retries can originate. It is better at this point for the transfer to be flagged as failed, and retried properly. The code allows for 5 retries with a 1s sleep between them. If this doesn't work - which it might not - then an -EIO error is returned to xrootd. Other error messages are not affected. * Better summary stats output for CephIOAdapterRaw * Comment out a comment Co-authored-by: james <james.walder@stfc.ac.uk> Co-authored-by: root <root@host-172-16-112-239.nubes.stfc.ac.uk> variable version/release for template (#21) Update bufferedIO with updates from master (#26) * variable rpm name (#17) * variable rpm name * Update xrootd-ceph.spec.in * Update makesrpm.sh * Update makesrpm.sh * Master cephnamelib (#16) * Allow ceph.namelib to take params and apply translation to full path * Reduce logging Remove extraneous logging messages * simplify parsing of namelib and added a log line for any remapped file Co-authored-by: James <James.walder@stfc.ac.uk> * XRD-22 Fix ensuring the correct filename is passed to the CephFile instance. (#24) A regression in previous commit meant that the filename was not correctly passed to the CephFile instance. This fix ensures that the filename is set correctly. Co-authored-by: james <james.walder@stfc.ac.uk> * re-introduce variable names to spec input (#27) Co-authored-by: Jo-stfc <71326101+Jo-stfc@users.noreply.github.com> Co-authored-by: James <James.walder@stfc.ac.uk> Decreased logging for bufferedIO operations. (#25) Reduced printouts. Only summary stats now produced, rather than the logging per read. Co-authored-by: James Walder <james.walder@stfc.ac.uk> Updates from master to buffered io needed for 550 2 (#32) * XRD-12 Add timestamp information for ceph logging methods Update the logwrapper method to print out the current timestamp in the initial section of output. * Return permission denied on write attempt on existing file with EXCL set (#31) Co-authored-by: James Walder <james.walder@stfc.ac.uk> * disable posc (#30) posc is disabled for proxies, but not for a unified setup. XrdCeph does not support the posc flag as it misinterprets objects as folders Co-authored-by: James Walder <james.walder@stfc.ac.uk> Co-authored-by: Jo-stfc <71326101+Jo-stfc@users.noreply.github.com> Buffered io multibuffers (#38) * Add multiple buffer support for reads in case of simultaneous threads reading the same file. * Further refinements to the simultaneous file reads code - Ensure all relevent read / write methods will create a buffer if needed - Validty check on close that a buffer was actually created (or bypass code if not) - Bugfix in case of odd read sizes combined with multi/split buffer reads (critical) - Clean of comments included for development * Enhanced logging for cluster metrics and readV layer improvments (#35) - dumpCLusterInfo to check on the rados connection info - extra logging in a delete to give info on delete times - update the readV basic alg to do a simple bulk request Co-authored-by: James Walder <james.walder@stfc.ac.uk> * Add time taken to unlink a file in the logging message - Logging an unlink now includes the time taken, in cases of (un)successful deletes - Remove some extraneous comments * - Fix issue with buffer passthrough read - Add maximum number of simultaneous buffers for a given file Once a given number of opens have been made against the same file, don't create a large buffer, and only create a 1MiB buffer for each new file. This should avoid issues with small paged reads, but would normally hope the pasthrough mode would be triggered in each read. * Additional statistics on buffered reading added. - Will report bytes read from ceph, bytes read but bypassed the cache, and the cache hit fraction --------- Co-authored-by: James Walder <james.walder@stfc.ac.uk> Bug fix for writes with bufferedIO when extending over buffer range. (#40) * Bug fix for writes with bufferedIO when extending over buffer range. - Fix for case where multiple writes to the buffer are needed for a given xrd write request - Previously threw an error; now will correctly perform the multiple writes as required. - Set the Simple Data buffer capacity to the input size, rather than the capacity of the vector, which could be larger. --------- Co-authored-by: James Walder <james.walder@stfc.ac.uk> variable rpm name (#17) * variable rpm name * Update xrootd-ceph.spec.in * Update makesrpm.sh * Update makesrpm.sh re-introduce variable names to spec input (#27)
…hecksum. Settings for streamed checksum allow calculation only, calc+log, and calc+log+store as extended attribute. Record streamed and readback checksums in a central checksums.log file.
…tialised checksum value when client reads file. Move recording of readback checksum value into XrdCephPosix instead of Set() method in XrdCephXAttr.cc. More checks for writable checksum record file
…ing zlib directly Move from using zlib adler32() function to XrdCks/XrdCksCalcadler32. The superclass, XrdCksCalc, provides a uniform interface to all supported checksum algorithms, and is the preferred method in XRootD. Remove an unneccessary guard condition in the ceph_posix_*write() methods.
Make XrdCeph plugin approach to calculating streamed checksums more acceptable by using the more generic XrdCksCalc class, subclass XrdCksCalcadler32, rather than calling the zlib adler32 function directly.
The XrdCksCalc object in the CephFileRef structure needs a matching delete in ceph_posix_close()
Clang-tidy warns that XrdOucPup::Unpack(&dP, eodP, &idP, idSz) may assign a nullptr to idP and return false, in which case the left side of the expression in line 211 evaluates to true and skips the pointer check done afterwards, leading to a potential null pointer dereference later in line 225.
The while loop should also be run only when str != nullptr. Fix also a -Wvolatile warning while at it, since the volatile keyword is deprecated since C++20.
This change is to fix the following error from static analysis:
/usr/lib/gcc/x86_64-pc-linux-gnu/14/include/g++-v14/bits/unique_ptr.h:91:16:
error: implicit instantiation of undefined template 'XrdCl::Operation<true>'
91 | static_assert(sizeof(_Tp)>0,
| ^
/usr/lib/gcc/x86_64-pc-linux-gnu/14/include/g++-v14/bits/unique_ptr.h:399:4:
note: in instantiation of member function 'std::default_delete<XrdCl::Operation<true>>::operator()' requested here
399 | get_deleter()(std::move(__ptr));
| ^
xrootd/src/XrdCl/XrdClOperations.hh:79:7: note: in instantiation of member function
'std::unique_ptr<XrdCl::Operation<true>>::~unique_ptr' requested here
79 | PipelineHandler()
| ^
xrootd/src/XrdCl/XrdClOperations.hh:49:32: note: template is declared here
49 | template<bool HasHndl> class Operation;
| ^
1 error generated.
Error while processing xrootd/src/XrdCl/XrdClFileStateHandler.cc.
The problem is that PipelineHandler has a std::unique_ptr<XrdCl::Operation<true>>
as a member, but the type XrdCl::Operation<true> is not yet fully known by the compiler.
Moving the class definition for Operation before PipelineHandler makes the type known to
the compiler by the time it reaches the member declaration in PipelineHandler. However,
since class Operation uses PipelineHandler as well, some methods had to be outlined and
moved towards the end of the header file.
…dSfsFile
Needed because class Stream needs a std::unique_ptr<XrdSfsFile>, which is only
forward declared. This change addresses the static analysis error below.
[ 70%] Building CXX object src/XrdHttpTpc/CMakeFiles/XrdHttpTPC-5.dir/XrdHttpTpcStream.cc.o
/usr/lib/gcc/x86_64-pc-linux-gnu/14/include/g++-v14/bits/unique_ptr.h:91:16:
error: invalid application of 'sizeof' to an incomplete type 'XrdSfsFile'
91 | static_assert(sizeof(_Tp)>0,
| ^~~~~~~~~~~
/usr/lib/gcc/x86_64-pc-linux-gnu/14/include/g++-v14/bits/unique_ptr.h:399:4:
note: in instantiation of member function 'std::default_delete<XrdSfsFile>::operator()' requested here
399 | get_deleter()(std::move(__ptr));
| ^
xrootd/src/XrdHttpTpc/XrdHttpTpcStream.hh:24:5:
note: in instantiation of member function 'std::unique_ptr<XrdSfsFile>::~unique_ptr' requested here
24 | Stream(std::unique_ptr<XrdSfsFile> fh, size_t max_blocks, size_t buffer_size, XrdSysError &log)
| ^
src/xrootd/src/XrdHttpTpc/XrdHttpTpcStream.hh:18:7: note: forward declaration of 'XrdSfsFile'
18 | class XrdSfsFile;
| ^
1 error generated.
Error while processing xrootd/src/XrdHttpTpc/XrdHttpTpcStream.cc.
xrootd/tests/stress/curl_stress.cc:164:2: warning: embedding a directive
within macro arguments has undefined behavior [-Wembedded-directive]
164 | #else
| ^
Move operations from a manager should succed after redirection
- Add tests for rename operation with HTTP
- MOVE operation is currently fails
after redirection
- Thus the tests assert against http code 401;
a fix would change it to 201 instead
- For redirection from meta-manager; move failes
for a manager node
…peration; this allows rename after redirections
uint16_t to time_t. Since time_t can be 64 bits the Python bindings need to use unsigned long long.
We test for overflow, but now timeouts use time_t, which may be 64 bits wide, so use 2^65 to force an overflow in that case too.
Reverts commit eb4295e as the hack introduced there is no longer needed after the current fix. Fixes: xrootd#2357
This adds support to get/put void* values in the client environment. Since pointers are only defined during runtime, no support for shell import is provided for pointers. Issue: xrootd#2522
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
10 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.