Fix RSA-OAEP to allow zero-length plaintext per RFC 8017 by MarkAtwood · Pull Request #10012 · wolfSSL/wolfssl

MarkAtwood · 2026-03-19T02:11:50Z

Summary

RsaPublicEncryptEx() rejects inLen == 0 with BAD_FUNC_ARG, but RFC 8017 Section 7.1.1 (RSAES-OAEP-ENCRYPT) permits zero-length messages. The only length constraint is mLen <= k - 2*hLen - 2, which mLen = 0 always satisfies.
RsaPrivateDecryptEx() converts a zero-length decryption result to RSA_BUFFER_E (unless WOLFSSL_RSA_DECRYPT_TO_0_LEN is defined). RFC 8017 Section 7.1.2 (RSAES-OAEP-DECRYPT) produces the original message M which may be empty.

Both OpenSSL and BoringSSL accept empty OAEP plaintexts.

Changes

Encrypt side (RsaPublicEncryptEx, line ~3321):

Changed inLen == 0 rejection to only apply for non-OAEP padding types
Added inline comment referencing RFC 8017 Section 7.1.1

Decrypt side (RsaPrivateDecryptEx, line ~3716):

When WOLFSSL_RSA_DECRYPT_TO_0_LEN is not defined, the ret == 0 rejection now uses constant-time masking (ctMaskEq) to allow zero-length results for WC_RSA_OAEP_PAD while preserving the existing behavior for other padding types
Added inline comment referencing RFC 8017 Section 7.1.2

Spec references

RFC 8017 Section 7.1.1 — RSAES-OAEP-ENCRYPT: Step 1b constrains mLen <= k - 2*hLen - 2; mLen = 0 is valid
RFC 8017 Section 7.1.2 — RSAES-OAEP-DECRYPT: Step 3g outputs M which may be the empty string

Testing

Verified with wolfcrypt-ring RSA-OAEP round-trip tests using SHA-1, SHA-256, SHA-384, and SHA-512 with zero-length plaintext.

🤖 Generated with Claude Code

Copilot

Pull request overview

This PR updates RSA-OAEP handling to permit zero-length plaintexts on both encryption and decryption, aligning behavior with RFC 8017 and other TLS/crypto libraries.

Changes:

Relaxed RsaPublicEncryptEx() argument validation to allow inLen == 0 for OAEP padding.
Adjusted RsaPrivateDecryptEx() constant-time error handling to permit ret == 0 results for OAEP padding.
Added RFC 8017 references in inline comments to document the behavioral change.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

You can also share your feedback on Copilot code review. Take the survey.

wolfcrypt/src/rsa.c

+     * a zero-length input is invalid. */
+    if (in == NULL || (inLen == 0 && pad_type != WC_RSA_OAEP_PAD)) {


wolfcrypt/src/rsa.c

+     * permitted: the spec requires mLen <= k - 2*hLen - 2, and mLen = 0
+     * satisfies this for all supported key sizes. For other padding types,
+     * a zero-length input is invalid. */


wolfcrypt/src/rsa.c

+                int zeroOk = ctMaskEq(pad_type, WC_RSA_OAEP_PAD);
+                ret = ctMaskSelInt(ctMaskNotEq(ret, 0) | zeroOk, ret,


RsaPublicEncryptEx() rejected inLen==0 unconditionally with BAD_FUNC_ARG. RFC 8017 Section 7.1.1 (RSAES-OAEP-ENCRYPT) permits zero-length messages: the only length constraint is mLen <= k - 2*hLen - 2, which mLen=0 always satisfies. RsaPrivateDecryptEx() converted a zero-length decryption result to RSA_BUFFER_E (unless WOLFSSL_RSA_DECRYPT_TO_0_LEN was defined). RFC 8017 Section 7.1.2 (RSAES-OAEP-DECRYPT) produces the original message M which may be empty. The fix uses constant-time masking to allow ret==0 when pad_type is WC_RSA_OAEP_PAD, preserving the existing timing-safe behavior for other padding types. Both OpenSSL and BoringSSL accept empty OAEP plaintexts. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> Found via Wycheproof test vectors.

Copilot

Pull request overview

Updates wolfCrypt RSA-OAEP handling to align with RFC 8017 by permitting zero-length plaintexts on encrypt and decrypt paths.

Changes:

Adjusts RsaPublicEncryptEx() argument validation to allow inLen == 0 for OAEP padding.
Adjusts RsaPrivateDecryptEx() handling of ret == 0 to allow zero-length OAEP results (under #ifndef WOLFSSL_RSA_DECRYPT_TO_0_LEN) using constant-time masking.
Adds inline RFC 8017 references in both code paths.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot · 2026-04-12T05:26:08Z

wolfcrypt/src/rsa.c

+            /* RFC 8017 Section 7.1.2: OAEP decryption may produce a valid
+             * zero-length message. Only reject ret==0 for non-OAEP types. */
+            {
+                int zeroOk = ctMaskEq(pad_type, WC_RSA_OAEP_PAD);
+                ret = ctMaskSelInt(ctMaskNotEq(ret, 0) | zeroOk, ret,
+                                   WC_NO_ERR_TRACE(RSA_BUFFER_E));
+            }


Allowing ret==0 for OAEP here makes OAEP padding errors indistinguishable from a valid empty plaintext: RsaUnPad_OAEP() returns 0 on error by setting idx=pkcsBlockLen (see wolfcrypt/src/rsa.c around the "Return 0 data length on error" logic), so invalid OAEP blocks would now be treated as successful decryptions of an empty message. To support empty messages safely, OAEP unpadding needs a separate validity indicator / negative error return (selected constant-time), and this layer should continue to reject the error case even when the message length is 0.

Suggested change

/* RFC 8017 Section 7.1.2: OAEP decryption may produce a valid

* zero-length message. Only reject ret==0 for non-OAEP types. */

{

int zeroOk = ctMaskEq(pad_type, WC_RSA_OAEP_PAD);

ret = ctMaskSelInt(ctMaskNotEq(ret, 0) | zeroOk, ret,

WC_NO_ERR_TRACE(RSA_BUFFER_E));

}

/* Reject zero-length results unless the dedicated build option

* explicitly enables them. In particular, OAEP unpadding may

* return 0 on error, so allowing ret == 0 here would make an

* invalid OAEP block indistinguishable from a valid empty

* plaintext. */

ret = ctMaskSelInt(ctMaskNotEq(ret, 0), ret,

WC_NO_ERR_TRACE(RSA_BUFFER_E));

Copilot · 2026-04-12T05:26:08Z

wolfcrypt/src/rsa.c

+     * a zero-length input is invalid. */
+    if (in == NULL || (inLen == 0 && pad_type != WC_RSA_OAEP_PAD)) {


This new argument validation still rejects in==NULL even when inLen==0 with OAEP. If the intent is to allow empty OAEP plaintexts, consider permitting a NULL input pointer when inLen==0 (common C API convention and safe here because OAEP padding copies 0 bytes).

Suggested change

* a zero-length input is invalid. */

if (in == NULL || (inLen == 0 && pad_type != WC_RSA_OAEP_PAD)) {

* a zero-length input is invalid. Permit a NULL input pointer only for

* the empty OAEP case, which is safe because no input bytes are read. */

if ((in == NULL && (inLen != 0 || pad_type != WC_RSA_OAEP_PAD)) ||

(inLen == 0 && pad_type != WC_RSA_OAEP_PAD)) {

Copilot · 2026-04-12T05:26:08Z

wolfcrypt/src/rsa.c

+     * permitted: the spec requires mLen <= k - 2*hLen - 2, and mLen = 0
+     * satisfies this for all supported key sizes. For other padding types,
+     * a zero-length input is invalid. */


The comment claims mLen=0 "satisfies this for all supported key sizes", but OAEP also requires k > 2*hLen+2 for the chosen hash; small keys (or large hashes) can still be invalid even with mLen=0. Suggest rewording to only state that mLen=0 satisfies the message-length constraint from RFC 8017 step 1b, without implying all key sizes are valid.

Suggested change

* permitted: the spec requires mLen <= k - 2*hLen - 2, and mLen = 0

* satisfies this for all supported key sizes. For other padding types,

* a zero-length input is invalid. */

* permitted with respect to the step 1b message-length check:

* mLen <= k - 2*hLen - 2. For other padding types, a zero-length input

* is invalid. */

Copilot · 2026-04-12T05:26:08Z

wolfcrypt/src/rsa.c

+                int zeroOk = ctMaskEq(pad_type, WC_RSA_OAEP_PAD);
+                ret = ctMaskSelInt(ctMaskNotEq(ret, 0) | zeroOk, ret,


ctMaskEq()/ctMaskNotEq() return a byte mask (0x00/0xFF) and ctMaskSelInt() takes a byte mask. Declaring zeroOk as byte (and keeping the combined mask as byte) would better match the CT helpers and avoid potential compiler warnings about implicit int->byte truncation.

Suggested change

int zeroOk = ctMaskEq(pad_type, WC_RSA_OAEP_PAD);

ret = ctMaskSelInt(ctMaskNotEq(ret, 0) | zeroOk, ret,

byte zeroOk = ctMaskEq(pad_type, WC_RSA_OAEP_PAD);

byte validRetMask = (byte)(ctMaskNotEq(ret, 0) | zeroOk);

ret = ctMaskSelInt(validRetMask, ret,

Copilot · 2026-04-12T05:26:09Z

wolfcrypt/src/rsa.c

+            /* RFC 8017 Section 7.1.2: OAEP decryption may produce a valid
+             * zero-length message. Only reject ret==0 for non-OAEP types. */
+            {
+                int zeroOk = ctMaskEq(pad_type, WC_RSA_OAEP_PAD);
+                ret = ctMaskSelInt(ctMaskNotEq(ret, 0) | zeroOk, ret,
+                                   WC_NO_ERR_TRACE(RSA_BUFFER_E));


There are existing OAEP padding tests (e.g., rsa_oaep_padding_test() in wolfcrypt/test/test.c) but they don’t cover a successful OAEP encrypt/decrypt round-trip of an empty message nor do they assert that invalid OAEP padding yields a negative error (as opposed to a 0-length "success"). Adding those cases would prevent regressions around the ret==0 handling being changed here.

Copilot AI review requested due to automatic review settings March 19, 2026 02:11

Copilot AI reviewed Mar 19, 2026

View reviewed changes

dgarske assigned MarkAtwood Mar 19, 2026

MarkAtwood force-pushed the fix/rsa-oaep-allow-empty-plaintext branch from eefa39a to a0752c3 Compare March 23, 2026 14:43

MarkAtwood assigned wolfSSL-Bot Mar 23, 2026

Copilot AI review requested due to automatic review settings April 12, 2026 05:20

MarkAtwood force-pushed the fix/rsa-oaep-allow-empty-plaintext branch from a0752c3 to e8e7cd1 Compare April 12, 2026 05:20

Copilot started reviewing on behalf of MarkAtwood April 12, 2026 05:22 View session

Copilot AI reviewed Apr 12, 2026

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fix RSA-OAEP to allow zero-length plaintext per RFC 8017#10012

Fix RSA-OAEP to allow zero-length plaintext per RFC 8017#10012
MarkAtwood wants to merge 1 commit intowolfSSL:masterfrom
MarkAtwood:fix/rsa-oaep-allow-empty-plaintext

MarkAtwood commented Mar 19, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

Copilot AI left a comment

Uh oh!

Copilot AI Apr 12, 2026

Uh oh!

Copilot AI Apr 12, 2026

Uh oh!

Copilot AI Apr 12, 2026

Uh oh!

Copilot AI Apr 12, 2026

Uh oh!

Copilot AI Apr 12, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

		* a zero-length input is invalid. */
		if (in == NULL \|\| (inLen == 0 && pad_type != WC_RSA_OAEP_PAD)) {

		int zeroOk = ctMaskEq(pad_type, WC_RSA_OAEP_PAD);
		ret = ctMaskSelInt(ctMaskNotEq(ret, 0) \| zeroOk, ret,

-                int zeroOk = ctMaskEq(pad_type, WC_RSA_OAEP_PAD);
-                ret = ctMaskSelInt(ctMaskNotEq(ret, 0) | zeroOk, ret,
+                byte zeroOk = ctMaskEq(pad_type, WC_RSA_OAEP_PAD);
+                byte validRetMask = (byte)(ctMaskNotEq(ret, 0) | zeroOk);
+                ret = ctMaskSelInt(validRetMask, ret,

Conversation

MarkAtwood commented Mar 19, 2026

Summary

Changes

Spec references

Testing

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Copilot AI Apr 12, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Apr 12, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Apr 12, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Apr 12, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Apr 12, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants