Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

SIGSEGV on pk_decrypt() #98

Open
dmagyar opened this issue Aug 21, 2017 · 9 comments
Open

SIGSEGV on pk_decrypt() #98

dmagyar opened this issue Aug 21, 2017 · 9 comments

Comments

@dmagyar
Copy link

dmagyar commented Aug 21, 2017

First of, thanks for creating this! I have tried to do a simple public-key decryption (cleartext is encrypted with the private key) and I run into a SIGSEGV :(

Lua 5.3.4  Copyright (C) 1994-2017 Lua.org, PUC-Rio
> pkey = require"openssl.pkey"
> u = require"util"
> 
> enc_msg_b64 = "QBmiROTW/ptrTTnhrEa1f3J0RiKKMo62ThV7gMCukiCoLFID7cSDKiRWRRM+HMxUCwpitasr7M5cyXHoEAdnmgZ9rvakJSAXH3BxsvyruFrLwCJzgAWyry6N7zqLIf5ZYq9OLcdfMNVMt6buyQLfgZaba4WY41FobOlaGTiJ8DU="
> pk = pkey.new("-----BEGIN PUBLIC KEY-----\nMIGfMA0GCSqGSIb3DQEBAQUAA4GNADCBiQKBgQDG526ikLUwCMRZfmzSjT5nYhWj\nVbnL5T8j29b0s+gZg1/SEqOeRnpGqEpbExcLOJ3Qk/SVBjdx4swDFDNen5vzUSb2\nepH4qyRFJpS+qFGKuqxpEV/B5qJ7NVg/Kv4HY3ZED/9rM8NC7ZogWM+RSadNfSc6\nGyhWJptupptJgCfqFQIDAQAB\n-----END PUBLIC KEY-----","PEM")
> 
> enc_msg = u.b64dec(enc_msg_b64)
> pk.decrypt(pk,enc_msg)
Segmentation fault (core dumped)

I checked and the b64 decoded message matches what I have and it also decrypts using the same pubkey and openssl rsautl.

@dmagyar
Copy link
Author

dmagyar commented Aug 22, 2017

I did some ltraces but can not figure out why it crashes:

readline("> "> pk.decrypt(pk,enc_msg)
)                                                                                                  = "pk.decrypt(pk,enc_msg)"
strlen("pk.decrypt(pk,enc_msg)")                                                                                = 22
realloc(0, 47)                                                                                                  = 0x19a9fa0
memcpy(0x19a9fb8, "pk.decrypt(pk,enc_msg)", 22)                                                                 = 0x19a9fb8
free(0x1979d20)                                                                                                 = <void>
strchr("return %s;", '%')                                                                                       = "%s;"
realloc(0, 32)                                                                                                  = 0x197ad20
memcpy(0x197ad38, "return ", 7)                                                                                 = 0x197ad38
strlen("pk.decrypt(pk,enc_msg)")                                                                                = 22
memcmp(0x19a9fb8, 0x19a9fb8, 22, 8)                                                                             = 0
strchr(";", '%')                                                                                                = nil
strlen(";")                                                                                                     = 1
realloc(0, 26)                                                                                                  = 0x19a9f70
memcpy(0x19a9f88, ";", 1)                                                                                       = 0x19a9f88
memcpy(0x7ffe0887aa60, "return ", 7)                                                                            = 0x7ffe0887aa60
memcpy(0x7ffe0887aa67, "pk.decrypt(pk,enc_msg)", 22)                                                            = 0x7ffe0887aa67
memcpy(0x7ffe0887aa7d, ";", 1)                                                                                  = 0x7ffe0887aa7d
realloc(0, 55)                                                                                                  = 0x19a9fe0
memcpy(0x19a9ff8, "return pk.decrypt(pk,enc_msg);", 30)                                                         = 0x19a9ff8
strlen("return pk.decrypt(pk,enc_msg);")                                                                        = 30
_setjmp(0x7ffe0887a9f8, 1, 0x7ffe0887ab30, 0x7ffe0887a9f0)                                                      = 0
realloc(0, 40)                                                                                                  = 0x19aa020
realloc(0, 56)                                                                                                  = 0x19aa050
realloc(0, 120)                                                                                                 = 0x19aa090
strcmp("=stdin", "=stdin")                                                                                      = 0
memcmp(0x426879, 0x1956258, 4, 0x195e820)                                                                       = 0
realloc(0, 32)                                                                                                  = 0x19aa110
realloc(0, 64)                                                                                                  = 0x19aa140
memcmp(0x19aa110, 0x19565b8, 6, 5)                                                                              = 0
realloc(0, 32)                                                                                                  = 0x19aa190
memcmp(0x19aa110, 0x19a0d58, 2, 1)                                                                              = 0
realloc(0, 64)                                                                                                  = 0x19aa1c0
free(0x19aa190)                                                                                                 = <void>
realloc(0, 64)                                                                                                  = 0x19aa210
realloc(0, 16)                                                                                                  = 0x1979d20
realloc(0, 16)                                                                                                  = 0x1977dc0
memcmp(0x19aa110, 0x199f5d8, 7, 6)                                                                              = 0
realloc(0, 128)                                                                                                 = 0x19aa260
free(0x19aa1c0)                                                                                                 = <void>
memcmp(0x19aa110, 0x19a0d58, 2, 1)                                                                              = 0
memcmp(0x19aa110, 0x19a7058, 7, 6)                                                                              = 0
realloc(0x1979d20, 32)                                                                                          = 0x19aa190
realloc(0x1977dc0, 32)                                                                                          = 0x19aa2f0
realloc(0x19aa190, 28)                                                                                          = 0x19aa190
realloc(0x19aa2f0, 28)                                                                                          = 0x19aa2f0
realloc(0x19aa210, 48)                                                                                          = 0x19aa210
free(0)                                                                                                         = <void>
free(0)                                                                                                         = <void>
realloc(0x19aa140, 16)                                                                                          = 0x19aa140
realloc(0, 32)                                                                                                  = 0x19aa160
free(0x19aa110)                                                                                                 = <void>
free(0)                                                                                                         = <void>
free(0)                                                                                                         = <void>
free(0)                                                                                                         = <void>
add_history(0x19a9fb8, -16, 0x19a7740, 0x19a7740)                                                               = 0
__sysv_signal(2, 0x4040e0, 0x19a7730, 0x19a7730)                                                                = 0
_setjmp(0x7ffe0887aa88, 1, 0x7ffe0887abc0, 0x7ffe0887aa80)                                                      = 0
strcmp("EVP_PKEY*", "EVP_PKEY*")                                                                                = 0
strcmp("EVP_PKEY*", "EVP_PKEY*")                                                                                = 0
realloc(0, 48)                                                                                                  = 0x19aa320
realloc(0, 56)                                                                                                  = 0x19aa360
strcmp("__gc", "path")                                                                                          = -17
strcmp("__gc", "getVerify")                                                                                     = -8
strlen("__gc")                                                                                                  = 4
memcmp(0x7fa74219c73c, 0x1955e38, 4, 1852)                                                                      = 0
realloc(0, 32)                                                                                                  = 0x19aa110
--- SIGSEGV (Segmentation fault) ---
+++ killed by SIGSEGV +++

@dmagyar
Copy link
Author

dmagyar commented Aug 22, 2017

After recompiling openssl with symbols and debug enabled I got this:

Program received signal SIGSEGV, Segmentation fault.
RSA_eay_private_decrypt (flen=<optimized out>, from=<optimized out>, to=0x68c740 "", rsa=0x6893d0, padding=1) at rsa_eay.c:562
562	            BN_with_flags(d, rsa->d, BN_FLG_CONSTTIME);
(gdb) bt
#0  RSA_eay_private_decrypt (flen=<optimized out>, from=<optimized out>, to=0x68c740 "", rsa=0x6893d0, padding=1) at rsa_eay.c:562
#1  0x00007ffff689bcb1 in pkey_rsa_decrypt (ctx=0x68c680, out=0x68c740 "", outlen=0x7fffffffe080, 
    in=0x65cba8 "@\031\242D\344\326\376\233kM9\341\254F\265\177rtF\"\212\062\216\266N\025{\200\300\256\222 \250,R\003\355\304\203*$VE\023>\034\314T\v\nb\265\253+\354\316\\\311q\350\020\ag\232\006}\256\366\244% \027\037pq\262\374\253\270Z\313\300\"s\200\005\262\257.\215\357:\213!\376Yb\257N-\307_0\325L\267\246\356\311\002\337\201\226\233k\205\230\343Qhl\351Z\031\070\211\360\065", inlen=128) at rsa_pmeth.c:461
#2  0x00007ffff6e7f834 in pk_decrypt (L=0x637018) at /root/luaossl/src/openssl.c:3533
#3  0x0000000000408b24 in luaD_precall ()
#4  0x000000000041430b in luaV_execute ()
#5  0x0000000000408def in luaD_call ()
#6  0x0000000000408e41 in luaD_callnoyield ()
#7  0x000000000040825f in luaD_rawrunprotected ()
#8  0x000000000040915d in luaD_pcall ()
#9  0x00000000004065af in lua_pcallk ()
#10 0x00000000004040a7 in docall ()
#11 0x0000000000404474 in doREPL ()
#12 0x0000000000404c48 in pmain ()
#13 0x0000000000408b24 in luaD_precall ()
#14 0x0000000000408de3 in luaD_call ()
#15 0x0000000000408e41 in luaD_callnoyield ()
#16 0x000000000040825f in luaD_rawrunprotected ()
#17 0x000000000040915d in luaD_pcall ()
#18 0x00000000004065af in lua_pcallk ()
#19 0x0000000000403e85 in main ()
(gdb) 

@daurnimator
Copy link
Collaborator

daurnimator commented Aug 26, 2017

Replicated with this C program:

#include <stdio.h>
#include <openssl/bio.h>
#include <openssl/buffer.h>
#include <openssl/evp.h>
#include <openssl/err.h>
#include <openssl/pem.h>
#include <openssl/rsa.h>

static void decrypt(EVP_PKEY *key, const char *str, size_t inlen, int rsaPadding) {
	size_t outlen;
	EVP_PKEY_CTX *ctx;
	BIO *bio;
	BUF_MEM *buf;
	int base_type = EVP_PKEY_base_id(key);

	bio = BIO_new(BIO_s_mem());
	if (!bio)
		goto sslerr;
	BIO_reset(bio);
	BIO_get_mem_ptr(bio, &buf);

	if (!(ctx = EVP_PKEY_CTX_new(key, NULL)))
		goto sslerr;

	if (EVP_PKEY_decrypt_init(ctx) <= 0)
		goto sslerr;

	if (base_type == EVP_PKEY_RSA && !EVP_PKEY_CTX_set_rsa_padding(ctx, rsaPadding))
		goto sslerr;

	if (EVP_PKEY_decrypt(ctx, NULL, &outlen, (const unsigned char *)str, inlen) <= 0)
		goto sslerr;

	if (!BUF_MEM_grow_clean(buf, outlen))
		goto sslerr;

	if (EVP_PKEY_decrypt(ctx, (unsigned char *)buf->data, &outlen, (const unsigned char *)str, inlen) <= 0)
		goto sslerr;

	EVP_PKEY_CTX_free(ctx);
	ctx = NULL;

	fwrite(buf->data, 1, outlen, stdout);

	BIO_reset(bio);
	BIO_free(bio);
	return;
sslerr:
	exit(4);
}

static const char pem[] = "-----BEGIN PUBLIC KEY-----\nMIGfMA0GCSqGSIb3DQEBAQUAA4GNADCBiQKBgQDG526ikLUwCMRZfmzSjT5nYhWj\nVbnL5T8j29b0s+gZg1/SEqOeRnpGqEpbExcLOJ3Qk/SVBjdx4swDFDNen5vzUSb2\nepH4qyRFJpS+qFGKuqxpEV/B5qJ7NVg/Kv4HY3ZED/9rM8NC7ZogWM+RSadNfSc6\nGyhWJptupptJgCfqFQIDAQAB\n-----END PUBLIC KEY-----";

static const char str[] = {
  0x40, 0x19, 0xa2, 0x44, 0xe4, 0xd6, 0xfe, 0x9b, 0x6b, 0x4d, 0x39, 0xe1,
  0xac, 0x46, 0xb5, 0x7f, 0x72, 0x74, 0x46, 0x22, 0x8a, 0x32, 0x8e, 0xb6,
  0x4e, 0x15, 0x7b, 0x80, 0xc0, 0xae, 0x92, 0x20, 0xa8, 0x2c, 0x52, 0x03,
  0xed, 0xc4, 0x83, 0x2a, 0x24, 0x56, 0x45, 0x13, 0x3e, 0x1c, 0xcc, 0x54,
  0x0b, 0x0a, 0x62, 0xb5, 0xab, 0x2b, 0xec, 0xce, 0x5c, 0xc9, 0x71, 0xe8,
  0x10, 0x07, 0x67, 0x9a, 0x06, 0x7d, 0xae, 0xf6, 0xa4, 0x25, 0x20, 0x17,
  0x1f, 0x70, 0x71, 0xb2, 0xfc, 0xab, 0xb8, 0x5a, 0xcb, 0xc0, 0x22, 0x73,
  0x80, 0x05, 0xb2, 0xaf, 0x2e, 0x8d, 0xef, 0x3a, 0x8b, 0x21, 0xfe, 0x59,
  0x62, 0xaf, 0x4e, 0x2d, 0xc7, 0x5f, 0x30, 0xd5, 0x4c, 0xb7, 0xa6, 0xee,
  0xc9, 0x02, 0xdf, 0x81, 0x96, 0x9b, 0x6b, 0x85, 0x98, 0xe3, 0x51, 0x68,
  0x6c, 0xe9, 0x5a, 0x19, 0x38, 0x89, 0xf0, 0x35
};

int main() {
	EVP_PKEY *pub;

	BIO *bio = BIO_new_mem_buf((void *)pem, sizeof pem);
	if (!bio)
		goto sslerr;

	/*
	 * BIO_reset is a rewind for read-only
	 * memory buffers. See mem_ctrl in
	 * crypto/bio/bss_mem.c of OpenSSL source.
	 */
	BIO_reset(bio);
	pub = PEM_read_bio_PUBKEY(bio, NULL, 0, "");
	if (!pub)
		goto sslerr;

	decrypt(pub, str, sizeof str, RSA_PKCS1_PADDING);
	return 0;

sslerr:
	ERR_print_errors_fp(stderr);
	return 1;
}

Looking at the problem from a higher level: you're trying to decrypt using a public key. We probably need to add a check for this; but there's no way for this operation to work in the first place.

@dmagyar
Copy link
Author

dmagyar commented Aug 28, 2017

Was able to perform just that - but I had to expose RSA_public_decrypt and use that instead of EVP_PKEY_decrypt. It works perfectly with this.

static int pk_rsa_pdecrypt(lua_State *L) {
	size_t outlen, inlen;
	EVP_PKEY *key = checksimple(L, 1, PKEY_CLASS);
	const char *str = luaL_checklstring(L, 2, &inlen);
	int rsaPadding = RSA_PKCS1_PADDING; /* default for `openssl rsautl` */
	int base_type = EVP_PKEY_base_id(key);
	RSA *rsakey;
	unsigned char *to = NULL;

	if (lua_istable(L, 3)) {
		if (base_type == EVP_PKEY_RSA) {
			lua_getfield(L, 3, "rsaPadding");
			rsaPadding = luaL_optinteger(L, -1, rsaPadding);
			lua_pop(L, 1);
		}
	}


	rsakey=EVP_PKEY_get1_RSA(key);
	if (!rsakey) goto sslerr;

	to = (unsigned char *)malloc(RSA_size(rsakey));

	base_type = RSA_public_decrypt(inlen, (const unsigned char *)str, to, rsakey, rsaPadding);
	if (base_type < 0) goto sslerr; 
	lua_pushlstring(L, (const char *)to, (size_t)base_type);
	if (to) free(to);

	return 1;

sslerr:
	if (to) free(to);
	return auxL_error(L, auxL_EOPENSSL, "pkey:rsa_pdecrypt");
} /* pk_rsa_pdecrypt() */

@daurnimator
Copy link
Collaborator

daurnimator commented Aug 29, 2017

So you're trying to extract the digest from a signed message?
What are you doing with that afterwards? Perhaps you're looking for pk:verify()?

@daurnimator
Copy link
Collaborator

I had a look into preventing the segfault: as far as I can see it requires manual upfront validation. Which needs to happen on a per key-type basis.
e.g.

       union {
               RSA *rsa;
               DH *dh;
               DSA *dsa;
#ifndef OPENSSL_NO_EC
               EC_KEY *ec;
#endif
       } base_key = { EVP_PKEY_get0(key) };

       /* Validate that required parameters are present, openssl doesn't do this itself */
       switch (EVP_PKEY_base_id(key)) {
       case EVP_PKEY_RSA: {
               const BIGNUM *d = NULL;
               RSA_get0_key(base_key.rsa, NULL, NULL, &d);
               luaL_argcheck(L, d != NULL, 1, "private key missing");
       }
               break;
       default:
               /* do nothing */
               break;
       }

This brings up questions about how this can happen in the first place: in this case it's importing a key from a PEM without the private portion. One option I considered was to just fill in the private portion with dummy values on import. But that seems like a poor choice: keys could come from elsewhere in an application. e.g. out of an SSL session. This led me to the conclusion that setParameters should have a way to clear a value, so that it's easy to create all key structures (particularly for testing).

@dmagyar
Copy link
Author

dmagyar commented Sep 4, 2017

I'm not decoding a digest - in this application a secret blob is encrypted with the private key and whoever has the public key can decrypt the blob. It works fine with RSA_public_decrypt now. Recommend to expose the RSA_ commands as well, here is the patch that exposes RSA_public_decrypt:

diff -Nur luaossl/src/openssl.c luaossl-patched/src/openssl.c
--- luaossl/src/openssl.c	2017-08-23 08:32:54.000000000 -0400
+++ luaossl-patched/src/openssl.c	2017-08-23 08:33:41.000000000 -0400
@@ -3550,6 +3552,41 @@
 
 	return auxL_error(L, auxL_EOPENSSL, "pkey:decrypt");
 } /* pk_decrypt() */
+
+static int pk_rsa_pdecrypt(lua_State *L) {
+	size_t outlen, inlen;
+	EVP_PKEY *key = checksimple(L, 1, PKEY_CLASS);
+	const char *str = luaL_checklstring(L, 2, &inlen);
+	int rsaPadding = RSA_PKCS1_PADDING; /* default for `openssl rsautl` */
+	int base_type = EVP_PKEY_base_id(key);
+	RSA *rsakey;
+	unsigned char *to = NULL;
+
+	if (lua_istable(L, 3)) {
+		if (base_type == EVP_PKEY_RSA) {
+			lua_getfield(L, 3, "rsaPadding");
+			rsaPadding = luaL_optinteger(L, -1, rsaPadding);
+			lua_pop(L, 1);
+		}
+	}
+
+
+	rsakey=EVP_PKEY_get1_RSA(key);
+	if (!rsakey) goto sslerr;
+
+	to = (unsigned char *)malloc(RSA_size(rsakey));
+
+	base_type = RSA_public_decrypt(inlen, (const unsigned char *)str, to, rsakey, rsaPadding);
+	if (base_type < 0) goto sslerr; 
+	lua_pushlstring(L, (const char *)to, (size_t)base_type);
+	if (to) free(to);
+
+	return 1;
+
+sslerr:
+	if (to) free(to);
+	return auxL_error(L, auxL_EOPENSSL, "pkey:rsa_pdecrypt");
+} /* pk_decrypt() */
 #endif
 
 #if HAVE_EVP_PKEY_CTX_NEW
@@ -4333,6 +4370,7 @@
 #if HAVE_EVP_PKEY_CTX_NEW
 	{ "decrypt",       &pk_decrypt },
 	{ "encrypt",       &pk_encrypt },
+	{ "rsa_pdecrypt", &pk_rsa_pdecrypt },
 #endif
 	{ "sign",          &pk_sign },
 	{ "verify",        &pk_verify },

@stephengaito
Copy link

Hello,

I have recreated the same seg-fault in a slightly different way on the current code base (e4cdcc5 on 13 Aug 2018).

My test does create a public key which has no corresponding private key part (as discussed in comment 98 above).

By hacking openssl.c in pk_decrypt (which was were I found the problem first) I found that the seg-fault (for me) happens on the call to 'BIO_get_mem_ptr(bio, &buf);' (line 3864).

My work around was to call toPEM("public") on each key just after it is created. I suspect this might perform something like the work around suggested in comment 98 above).

Since I have (what I think is a robust) work around, this is currently not a high priority issue.

My test scripts and short analysis can be found in my branch:
https://github.com/stephengaito/luaossl/tree/RSA-encrypt/decrypt-seg-fault

Many thanks for writing and maintaining a very useful Lua module.

Regards,
Stephen Gaito

@daurnimator
Copy link
Collaborator

My work around was to call toPEM("public") on each key just after it is created. I suspect this might perform something like the work around suggested in comment 98 above).

Oh? that's interesting.... you think there's something inside of PEM_write_bio_PUBKEY?

My test scripts and short analysis can be found in my branch:
https://github.com/stephengaito/luaossl/tree/RSA-encrypt/decrypt-seg-fault

Note that this projects has tests in the regress subdirectory.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

3 participants