Encryption severely broken for large databases

[rokclimb15]

Unfortunately there are a couple of serious problems with openssl smime that render the encryption option useless and dangerous when dumping a database over a few GB.

Regardless of input size, openssl smime will always produce a 1.9GB file on disk, which indicates that the input is truncated. This happens silently, which is very dangerous for a DB backup. Additionally, it is not possible to decrypt any smime message that large with OpenSSL due to internal limitations.

Reference https://rt.openssl.org/Ticket/Display.html?id=4651 for issues (guest/guest is login).

I recommend implementing gpg/gpg2 encryption with a passphrase file. gpg2 supports AES-NI with a new enough version of libgcrypt for AES acceleration. I can work on a patch if desired, but wanted to file this immediately so users with large DB exports can stop using encryption. It's very unsafe for backup as it cannot be restored.

[cytopia]

@rokclimb15 Thanks a lot for pointing this out.
I made it visible at the top of the Readme, so that everybody is aware about this.

Do you know the exact database size, when this problem will occur?

[rokclimb15]

The problem occurs between 1500 and 1600MB of input. That won't translate exactly to a database size, but anything over 1.5GB of data (minus indexes) is a candidate for this problem. Files that large can be encoded but not decoded.

The problem with input being clipped seems to happen at 1.9GB, but due to the previous limitation, that issue is irrelevant right now.

[cytopia]

I made a temporary adjustment, which will issue a warning to stderr in case of databases > 1200 Megabytes with enabled encryption.

[cytopia]

Updated version also pushed to homebrew, webpage and announced on twitter.

[jasperjorna]

Thanks for the heads up, @rokclimb15! Much appreciated.

[cytopia]

@rokclimb15 it seems that a more recent OpenSSL version with -stream (as pointed out by Dr Stephen N. Henson in the OpenSSL ticket) works as expected.

# Old OpenSSL Version
$ openssl version
OpenSSL 0.9.8zh 14 Jan 2016

$ openssl \
smime -encrypt -binary -text -aes256 -in sample.txt \
-out sample.txt.enc  -outform DER /etc/mysqldump-secure.pub.pem

# More recent OpenSSL Version
$ /usr/local/Cellar/openssl101/1.0.1t_1/bin/openssl version
OpenSSL 1.0.1t  3 May 2016

$ /usr/local/Cellar/openssl101/1.0.1t_1/bin/openssl \
smime -encrypt -stream -binary -text -aes256 -in sample.txt \
-out sample.txt.stream.enc  -outform DER /etc/mysqldump-secure.pub.pem

# with and without stream
$ ls -laph |grep sample
-rw-r--r--    1 cytopia 1286676289 3.0G Aug 18 23:43 sample.txt
-rw-r--r--    1 cytopia 1286676289 1.9G Aug 18 23:44 sample.txt.enc
-rw-r--r--    1 cytopia 1286676289 3.1G Aug 18 23:50 sample.txt.stream.enc

Can you verify this on your end?

[rokclimb15]

That does appear to fix the encryption problem. But now try to decrypt it ;)

It's unintentional ransomware if you ever lost your data and needed to restore a backup. The data size warning should probably remain until streaming decryption or bigger mem bufs are introduced in OpenSSL.

On Aug 18, 2016, at 5:54 PM, cytopia notifications@github.com wrote:

@rokclimb15 it seems that a more recent OpenSSL version with -stream (as pointed out by Dr Stephen N. Henson in the OpenSSL ticket) works as expected.

Old OpenSSL Version

$ openssl version
OpenSSL 0.9.8zh 14 Jan 2016

$ openssl
smime -encrypt -binary -text -aes256 -in sample.txt
-out sample.txt.enc -outform DER /etc/mysqldump-secure.pub.pem

More recent OpenSSL Version

$ /usr/local/Cellar/openssl101/1.0.1t_1/bin/openssl version
OpenSSL 1.0.1t 3 May 2016

$ /usr/local/Cellar/openssl101/1.0.1t_1/bin/openssl
smime -encrypt -stream -binary -text -aes256 -in sample.txt
-out sample.txt.stream.enc -outform DER /etc/mysqldump-secure.pub.pem

with and without stream

$ ls -laph |grep sample
-rw-r--r-- 1 cytopia 1286676289 3.0G Aug 18 23:43 sample.txt
-rw-r--r-- 1 cytopia 1286676289 1.9G Aug 18 23:44 sample.txt.enc
-rw-r--r-- 1 cytopia 1286676289 3.1G Aug 18 23:50 sample.txt.stream.enc
Can you verify this on your end?

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or mute the thread.

[cytopia]

I see, decryption throws this error:

Error reading S/MIME message
140735184199760:error:07069041:memory buffer routines:BUF_MEM_grow_clean:malloc failure:buffer.c:150:
140735184199760:error:0D06B041:asn1 encoding routines:ASN1_D2I_READ_BIO:malloc failure:a_d2i_fp.c:239:

Too bad.

[rokclimb15]

Here's what I came up with for workable encryption. Command should be same or similar for gpg. I used v2 because it uses libgcrypt with AES-NI if new enough. the passphrase file should be chmod 600 for security and contain the passphrase for encryption/decryption. This reads from STDIN and writes to STDOUT by default.

gpg2 --compress-algo none --cipher-algo AES256 --symmetric --batch --passphrase-file /root/mysql-backup-passphrase.txt

[cytopia]

@rokclimb15 Thanks for this.
I still have to see how exactly I need to implement the encryption (symmetric or asymmetric)

[rokclimb15]

My two cents is to use symmetric. This is a backup tool, so in general the user will maintain control of the files throughout their lifecycle. Public/private key crypto probably isn't needed for that reason. Additionally, a careless user might not back up their private key and if the whole system was lost, they could very well remember their encryption passphrase. Most backup products use a passphrase to encrypt and decrypt.