To fully explain the situation, I'll detail a bit of the setup first.
We used OpenSSL to create a private key:
openssl genrsa -out private.pem 1024
We then extracted the public key from the generated private key:
openssl rsa -in private.pem -out public.pem -pubout
You can then easily sign data with the private key like this:
echo -n "secret" | openssl rsautl -sign -inkey private.pem -out secret.txt
And then you can verify the data with the public key like so:
openssl rsautl -verify -in secret.txt -inkey public.pem -pubin
I have to pause here to point something out. Signing is not exactly encrypting with the private key. Because all you need to read the data is the public key, and after all, the public key is ... public! It's called sign/verify because all it really does is prove that the data indeed came from the person with the private key.
But this is not a tutorial about public key architecture. Wikipedia is an excellent source is you need an introduction or a refresher on public key cryptography.
So like I was saying, we could easily sign the data on our server. Now how do we verify the data in our C# app?
Our first thought was that we could use the RSACryptoServiceProvider class from .Net. It comes with a few sign/verify methods that may work. But first we need to read in the public key. Which is in PEM format. Which doesn't appear to be supported in the .Net API. And why would it? It's only the default format in OpenSSL - one of the world's most popular cryptography API's. Nevertheless - how do we read a PEM file in C#?
We found a way using some code from JavaScience: OpenSSLKey
And it worked great. Just not on every public key we had. We needed to fix a little bug first:
I was getting an exception in the PeekChar method. A little research told me all I had to do to fix the problem was initialize the BinaryReader instances with ASCII encoding.
You can verify OpenSSLKey is working by dumping the key modulus in your code, and then comparing it to what OpenSSL says the key modulus is:
openssl rsa -in public.pem -pubin -text
And since OpenSSLKey kindly gives us a RSACryptoServiceProvider object, we can attempt to verify the data from OpenSSL. And we see that it does NOT work!
After hours of mucking with it, we decided to take a different approach. We'll use OpenSSL directly from within our application.
You can download a windows installer of OpenSSL from here.
This not only allows you to use OpenSSL on the windows command line, but it also comes with the OpenSSL libraries in dll format. And once we figured out the cryptic OpenSSL API's, we were able to verify the data via the libeay32.dll. And we successfully tested this technique on many computers.
Shortly after releasing the new version, we started getting a bunch of complaints. Many users were having the same problem: .Net continually failed to load libeay32.dll. (This was version 0.9.8g of the dll on Win XP and Vista machines.) We tried just about everything to fix this problem, but came up empty handed. We had no idea why some (many) computers couldn't load this dll on their machine. So it was back to the drawing board...
After much searching on the Internet for an answer, we realized we weren't alone. It seems that many others were having similar problems, with not many people reporting success. We kept trying to get RSACryptoServiceProvider to do what we wanted, but no matter of reversing bytes or anything seemed to work.
So, in a fit of desperation, we finally emailed the developer of an RSA commercial library. We asked the developer why their RSA library didn't work with OpenSSL. And we got an excellent response:
OpenSSL: Can only sign small bits of data that fit within a single block. The data is padded and signed. The reverse is called "verify" and in that case the data is "unsigned" and then unpadded and the original data is returned.
[Windows]: Can sign any amount of data. The Sign* methods first hash the data and then the hash is padded and signed. The Verify* methods expect three inputs: the original data, a hash algorithm name, and the signature data. The original data is hashed and the result of unsigning/unpadding is compared with the hash of the original data.
In addition to this, the developer promised to add OpenSSL compatible methods to the RSA library, and delivered on that promise within 48 hours. The Chilkat RSA library can be used to interoperate with OpenSSL from within C#. We now use it, it works great, our nightmare is over, and we can finally get back to our regular development work.
We've created a project with sample code demonstrating how to accomplish this task using all 3 methods:
1. Parsing the PEM key with OpenSSLKey, and using RSACryptoServiceProvider (doesn't work)
2. Using OpenSSL directly via libeay32.dll (works, but has deployment issues)
3. Using Chilkat commercial library (works)
Download Sample Project
We hope this helps someone.