Verify enveloped XML signatures on Win32 (OpenSSL)

[oblaser]

I'm trying to verify some enveloped XML signatures with xmlsec in a Win32 application, but I struggle to do so. I created a test project (see my GitHub repo (tag "old-prebuilt-binaries")) from scratch and added the prebuilt binaries from here: https://www.zlatkovic.com/libxml.en.html

All the examples (verify1-4) fail to verify successfully. In my application only the verify1 and 2 example are of interest, I think, so I created a test report here:
https://github.com/oblaser/poc-xmlsec/blob/main/tests.md

For the verify1 example it seems that the canonical form could not be created. I could not figure out how to set the "dsigCtx->c14nMethod" pointer correctly.
Using the verify2 example ran through without any API error, but it results saying that the signature in the XML file is invalid. Which I find very strange, since I didn't change the example XML files.

I already tried it with the MS-crypto and openssl version of the library, which didn't help. Do you have any hint for me what else I could do, to at least run the examples successfully?

[lsh123]

First, the Windows binaries on www.zlatkovic.com are very old. I would not recommend to use those.

I always start from trying out the xmlsec1 command line utility:

$ xmlsec1 verify --pubkey-pem /tmp/rsapub.pem /tmp/xml-01_signed.xml 
OK
SignedInfo References (ok/all): 1/1
Manifests References (ok/all): 0/0

I also just tried the examples/verify1.c example as-is without any modifications and worked as well:

$ ./verify1 /tmp/build-openssl/xml-01_signed.xml  /tmp/build-openssl/rsapub.pem 
Signature is OK

Both of the above are using the latest released version on my Ubuntu (xmlsec 1.2.34 is a couple years old):

$ xmlsec1 --version
xmlsec1 1.2.34 (openssl)

[lsh123]

BTW, just an idea: what you observe sounds like an issue with difference in memory layout between compiled libraries and your program. I am not really sure how these binaries were compiled. I would make sure that XMLSEC_NO_SIZE_T is set (or not set) in both xmlsec libraries and the program that uses them. But again, the pre-compiled Windows binaries are very old (2011!!!).