Refactor XML handling

[EmmasBox]

We can speed up the API and simplify the code base by switching to Rapid XML

[yinjie-z]

I ran a benchmark comparing our current manual approach against RapidXML and pugixml across two scenarios: a small trait set and a long trait list (10,000 iterations each).

Here's the result with small trait :

[MANUAL (StringStream)]
  Time (10k ops): 13.8659 ms
  Avg per op:     1.38659 microseconds
  String Capacity: 1024 bytes

[RapidXML (Pool)]
  Time (10k ops): 30.8162 ms
  Avg per op:     3.08162 microseconds
  String Capacity: 480 bytes

[pugixml (DOM)]
  Time (10k ops): 18.9056 ms
  Avg per op:     1.89056 microseconds
  String Capacity: 319 bytes

Here's another result with long trait list :

[MANUAL (StringStream)]
  Time (10k ops): 89.5363 ms
  Avg per op:     8.95363 microseconds
  String Capacity: 2048 bytes

[RapidXML (Pool)]
  Time (10k ops): 112.185 ms
  Avg per op:     11.2185 microseconds
  String Capacity: 1920 bytes

[pugixml (DOM)]
  Time (10k ops): 66.5336 ms
  Avg per op:     6.65336 microseconds
  String Capacity: 1078 bytes

In conclusion :

Method	Small Set (Avg μs)	Long Set (Avg μs)	Memory (Long Set)
Manual (StringStream)	1.38	8.95	2048 bytes
pugixml (DOM)	1.89	6.65	1078 bytes
RapidXML (Pool)	3.08	11.21	1920 bytes

Performance Scaling: While Manual is slightly faster for tiny objects, pugixml outperforms Manual string concatenation by ~25% as the XML complexity increases.

Memory Efficiency: pugixml consistently uses significantly less memory (nearly 50% less than the Manual capacity for large sets).

RapidXML Overhead: RapidXML's memory pool seems to struggle with scaling in this specific implementation compared to pugixml's DOM management.

Given that pugixml is faster, more memory-efficient, and generally has a cleaner code, would you be open to using pugixml ?

Note on Environment: These benchmarks were performed on Linux. I have not yet validated the performance on zos, though pugixml is known for being highly portable.

View Benchmark Code

#include <chrono>
#include <iostream>
#include <vector>
#include <nlohmann/json.hpp>
#include "test_manual.cpp"
#include "test_rapidxml.cpp"
#include "test_pugixml.cpp"

void runBenchmark() {
    nlohmann::json traits = {{"base:name", "ELIJAH SWIFT"}, {"omvs:uid", "24"}, {"omvs:home_directory", "/home.ESWIFT"}};
    std::string admin = "user";
    const int iterations = 10000;

    auto benchmark = [&](auto func, std::string label) {
        auto start = std::chrono::high_resolution_clock::now();
        size_t memory_sample = 0;
        
        for(int i = 0; i < iterations; ++i) {
            std::string result = func(admin, traits);
            if (i == 0) memory_sample = result.capacity(); // Sample capacity for memory footprint
        }
        
        auto end = std::chrono::high_resolution_clock::now();
        std::chrono::duration<double, std::milli> elapsed = end - start;
        
        std::cout << "[" << label << "]\n";
        std::cout << "  Time (10k ops): " << elapsed.count() << " ms\n";
        std::cout << "  Avg per op:     " << (elapsed.count() / iterations) * 1000 << " microseconds\n";
        std::cout << "  String Capacity: " << memory_sample << " bytes\n\n";
    };

    benchmark(ManualXMLGenerator::build, "MANUAL (StringStream)");
    benchmark(RapidXMLGenerator::build,  "RapidXML (Pool)");
    benchmark(PugiXMLGenerator::build,   "pugixml (DOM)");
}

int main() {
    runBenchmark();
    return 0;
}

View Manual Test

#include <string>
#include <sstream>
#include <nlohmann/json.hpp>

class ManualXMLGenerator {
public:
    static std::string escape(std::string data) {
        std::string buffer;
        buffer.reserve(data.size());
        for (size_t pos = 0; pos != data.size(); ++pos) {
            switch (data[pos]) {
                case '&':  buffer.append("&amp;");  break;
                case '\"': buffer.append("&quot;"); break;
                case '\'': buffer.append("&apos;"); break;
                case '<':  buffer.append("&lt;");   break;
                case '>':  buffer.append("&gt;");   break;
                default:   buffer.append(1, data[pos]); break;
            }
        }
        return buffer;
    }

    static std::string build(const std::string& admin_type, const nlohmann::json& traits) {
        std::string xml;
        xml.reserve(1024); // Optimization: pre-allocate memory

        // Meta and Root
        xml += "<?xml version=\"1.0\" encoding=\"IBM-1047\" ?>";
        xml += "<securityrequest xmlns=\"http://www.ibm.com/systems/zos/saf\" ";
        xml += "xmlns:racf=\"http://www.ibm.com/systems/zos/racf\">";

        std::string true_type = escape(admin_type);
        xml += "<" + true_type + " requestid=\"" + true_type + "_request\">";

        if (!traits.empty()) {
            for (auto& [key, value] : traits.items()) {
                std::string k = escape(key);
                std::string v = escape(value.is_string() ? value.get<std::string>() : value.dump());
                xml += "<" + k + ">" + v + "</" + k + ">";
            }
            xml += "</" + true_type + ">";
        } else {
            xml += "/>"; 
        }

        xml += "</securityrequest>";
        return xml;
    }
};

View RapidXML Test

#include "rapidxml.hpp"
#include "rapidxml_print.hpp"

class RapidXMLGenerator {
public:
    static std::string build(const std::string& admin_type, const nlohmann::json& traits) {
        using namespace rapidxml;
        xml_document<> doc;
        
        // Declaration
        xml_node<>* decl = doc.allocate_node(node_declaration);
        decl->append_attribute(doc.allocate_attribute("version", "1.0"));
        decl->append_attribute(doc.allocate_attribute("encoding", "IBM-1047"));
        doc.append_node(decl);

        // Root
        xml_node<>* root = doc.allocate_node(node_element, "securityrequest");
        root->append_attribute(doc.allocate_attribute("xmlns", "http://www.ibm.com/systems/zos/saf"));
        root->append_attribute(doc.allocate_attribute("xmlns:racf", "http://www.ibm.com/systems/zos/racf"));
        doc.append_node(root);

        // Admin Node
        char* node_name = doc.allocate_string(admin_type.c_str());
        xml_node<>* admin = doc.allocate_node(node_element, node_name);
        
        std::string req_id = admin_type + "_request";
        admin->append_attribute(doc.allocate_attribute("requestid", doc.allocate_string(req_id.c_str())));
        root->append_node(admin);

        for (auto& [key, value] : traits.items()) {
            char* k = doc.allocate_string(key.c_str());
            std::string v_str = value.is_string() ? value.get<std::string>() : value.dump();
            char* v = doc.allocate_string(v_str.c_str());
            admin->append_node(doc.allocate_node(node_element, k, v));
        }

        std::string out;
        print(std::back_inserter(out), doc, print_no_indenting);
        return out;
    }
};

View pugixml Test

#include "pugixml.hpp"

class PugiXMLGenerator {
public:
    static std::string build(const std::string& admin_type, const nlohmann::json& traits) {
        pugi::xml_document doc;
        
        auto decl = doc.prepend_child(pugi::node_declaration);
        decl.append_attribute("version") = "1.0";
        decl.append_attribute("encoding") = "IBM-1047";

        auto root = doc.append_child("securityrequest");
        root.append_attribute("xmlns") = "http://www.ibm.com/systems/zos/saf";
        root.append_attribute("xmlns:racf") = "http://www.ibm.com/systems/zos/racf";

        auto admin = root.append_child(admin_type.c_str());
        admin.append_attribute("requestid") = (admin_type + "_request").c_str();

        for (auto& [key, value] : traits.items()) {
            std::string v_str = value.is_string() ? value.get<std::string>() : value.dump();
            admin.append_child(key.c_str()).text().set(v_str.c_str());
        }

        std::stringstream ss;
        doc.save(ss, "", pugi::format_raw | pugi::format_no_declaration);
        return ss.str();
    }
};

Lines to add in rapidxml_print.hpp for current version cpp

namespace rapidxml
{

    ///////////////////////////////////////////////////////////////////////
    // Printing flags

    const int print_no_indenting = 0x1;   //!< Printer flag instructing the printer to suppress indenting of XML. See print() function.

    ///////////////////////////////////////////////////////////////////////
    // Internal

    //! \cond internal
    namespace internal
    {
        // Lines to add with modern cpp
        template<class OutIt, class Ch> inline OutIt print_node(OutIt out, const xml_node<Ch> *node, int flags = 0, int indent = 0);
        template<class OutIt, class Ch> inline OutIt print_children(OutIt out, const xml_node<Ch> *node, int flags = 0, int indent = 0);
        template<class OutIt, class Ch> inline OutIt print_element_node(OutIt out, const xml_node<Ch> *node, int flags = 0, int indent = 0);
        template<class OutIt, class Ch> inline OutIt print_data_node(OutIt out, const xml_node<Ch> *node, int flags = 0, int indent = 0);
        template<class OutIt, class Ch> inline OutIt print_cdata_node(OutIt out, const xml_node<Ch> *node, int flags = 0, int indent = 0);
        template<class OutIt, class Ch> inline OutIt print_declaration_node(OutIt out, const xml_node<Ch> *node, int flags = 0, int indent = 0);
        template<class OutIt, class Ch> inline OutIt print_comment_node(OutIt out, const xml_node<Ch> *node, int flags = 0, int indent = 0);
        template<class OutIt, class Ch> inline OutIt print_doctype_node(OutIt out, const xml_node<Ch> *node, int flags = 0, int indent = 0);
        template<class OutIt, class Ch> inline OutIt print_pi_node(OutIt out, const xml_node<Ch> *node, int flags = 0, int indent = 0);

...

[EmmasBox]

Wow good work, I am impressed. I am solution agnostic, whatever works best is what we should use. Attachment is the root of all evils.