Friday, August 8, 2008

OpenSSL and RSACryptoServiceProvider

We had a need to interoperate with OpenSSL from within our application recently. More specifically, we needed to be able to sign some data with OpenSSL on our server (a linux machine), and then verify the data within Mojo. This was quite easy to do in the Mac version using SSCrypto. However, accomplishing the same feat on Windows in our C# app proved to be a nightmare. We eventually got it working, and this post is being written in the hope of sparing others the same hell we went through.

To fully explain the situation, I'll detail a bit of the setup first.

We used OpenSSL to create a private key:
openssl genrsa -out private.pem 1024

We then extracted the public key from the generated private key:
openssl rsa -in private.pem -out public.pem -pubout

You can then easily sign data with the private key like this:
echo -n "secret" | openssl rsautl -sign -inkey private.pem -out secret.txt

And then you can verify the data with the public key like so:
openssl rsautl -verify -in secret.txt -inkey public.pem -pubin

I have to pause here to point something out. Signing is not exactly encrypting with the private key. Because all you need to read the data is the public key, and after all, the public key is ... public! It's called sign/verify because all it really does is prove that the data indeed came from the person with the private key.

But this is not a tutorial about public key architecture. Wikipedia is an excellent source is you need an introduction or a refresher on public key cryptography.

So like I was saying, we could easily sign the data on our server. Now how do we verify the data in our C# app?

Our first thought was that we could use the RSACryptoServiceProvider class from .Net. It comes with a few sign/verify methods that may work. But first we need to read in the public key. Which is in PEM format. Which doesn't appear to be supported in the .Net API. And why would it? It's only the default format in OpenSSL - one of the world's most popular cryptography API's. Nevertheless - how do we read a PEM file in C#?

We found a way using some code from JavaScience: OpenSSLKey

And it worked great. Just not on every public key we had. We needed to fix a little bug first:

I was getting an exception in the PeekChar method. A little research told me all I had to do to fix the problem was initialize the BinaryReader instances with ASCII encoding.

You can verify OpenSSLKey is working by dumping the key modulus in your code, and then comparing it to what OpenSSL says the key modulus is:
openssl rsa -in public.pem -pubin -text

And since OpenSSLKey kindly gives us a RSACryptoServiceProvider object, we can attempt to verify the data from OpenSSL. And we see that it does NOT work!

After hours of mucking with it, we decided to take a different approach. We'll use OpenSSL directly from within our application.

You can download a windows installer of OpenSSL from here.

This not only allows you to use OpenSSL on the windows command line, but it also comes with the OpenSSL libraries in dll format. And once we figured out the cryptic OpenSSL API's, we were able to verify the data via the libeay32.dll. And we successfully tested this technique on many computers.


Shortly after releasing the new version, we started getting a bunch of complaints. Many users were having the same problem: .Net continually failed to load libeay32.dll. (This was version 0.9.8g of the dll on Win XP and Vista machines.) We tried just about everything to fix this problem, but came up empty handed. We had no idea why some (many) computers couldn't load this dll on their machine. So it was back to the drawing board...

After much searching on the Internet for an answer, we realized we weren't alone. It seems that many others were having similar problems, with not many people reporting success. We kept trying to get RSACryptoServiceProvider to do what we wanted, but no matter of reversing bytes or anything seemed to work.

So, in a fit of desperation, we finally emailed the developer of an RSA commercial library. We asked the developer why their RSA library didn't work with OpenSSL. And we got an excellent response:
OpenSSL: Can only sign small bits of data that fit within a single block. The data is padded and signed. The reverse is called "verify" and in that case the data is "unsigned" and then unpadded and the original data is returned.

[Windows]: Can sign any amount of data. The Sign* methods first hash the data and then the hash is padded and signed. The Verify* methods expect three inputs: the original data, a hash algorithm name, and the signature data. The original data is hashed and the result of unsigning/unpadding is compared with the hash of the original data.

In addition to this, the developer promised to add OpenSSL compatible methods to the RSA library, and delivered on that promise within 48 hours. The Chilkat RSA library can be used to interoperate with OpenSSL from within C#. We now use it, it works great, our nightmare is over, and we can finally get back to our regular development work.

We've created a project with sample code demonstrating how to accomplish this task using all 3 methods:
1. Parsing the PEM key with OpenSSLKey, and using RSACryptoServiceProvider (doesn't work)
2. Using OpenSSL directly via libeay32.dll (works, but has deployment issues)
3. Using Chilkat commercial library (works)

Download Sample Project

We hope this helps someone.


Pavel Chuchuva said...

Actually there is a way to parse public key in PEM format into RSACryptoServiceProvider. See PemPublic by Michel Gallant.

Robbie Hanson said...

The PemPublic class you refer to comes from the same place as the OpenSSLKey class I referred to - JavaScience. And in fact, the PemPublic class does the exact same thing that the DecodeX509PublicKey method does within OpenSSLKey. It's the exact same source code. And they both return a RSACryptoServiceProvider instance.

However, the problem wasn't parsing a public PEM key. It was the difference in techniques between signing/verifying in OpenSSL vs in .Net.

John said...

You might want to revise your statements, I have code that has been verifying signed hashed signatures with the RSACryptoServiceProvider class for a while now. And a lot of the signed signatures where done from OpenSSL libraries.

And your claim about it being OpenSSL vs ".Net" is incorrect also, as most(.Net 3.5 is doing more in managed code than CAPI) of the Crypto code in .Net is done by calling the underling CAPI unmanaged functions.

And if what the developer that you quoted earlier is correct, it sounds like the padding that OpenSSL is doing(for some reason) is breaking rules and not being RFC compliant, as other RFC compliant Crypto APIs are not able to verify it.

Robbie Hanson said...

Hi John,

I posted code demonstrating the incompatibility. If you have code that you claim works, please post the code so the community can take a look at it.

Also, CAPI != OpenSSL.

Anonymous said...

When I needed to do this I converted the certificate file to PCKS12 (openssl has a method for doing it), imported it into the windows certificate store and then used it from there and didn't get any mismatches.

It was near impossible to go from an X509Certificate2 to an RSACryptoServiceProvider but once I figured out the right CSPParameters it was fine.

Didn't encounter the same signature length issues, but that could be because the data I'm signing is very short.

I *did* find that the output was reversed compared to the crypto API though when encrypting/decrypting, which seems very strange.

Wolf Eyes said...

When I gen pair key by opelssl,headder's public key is "
but i can't load by Chilkat with LoadOpenSslPemFile method. I try load the key with headder's public key is "-----BEGIN PUBLIC KEY-----"
then everything OK. How can i use first key.
Please help me! thanks

Mikael Henriksson said...

This does not work for me on Windows 7 x64. It does however work for some one my colleagues on Windows XP x86.

Still trying to find the cause of the problem.

Anonymous said...


Can anyone post a code on how to Signdata() or Decrypt() with only knowing n,d and e created using oppenssl. Microsoft's RSA implementatio demands providing P, Q as well but its not known to me :(

Danny Scheelings said...


Great post. Thanks!
I'm wodering: is this still up-to-date?


Felipe said...

Thank you very much, I have wasted much time trying to do this. Great post.