Monday, 25 November 2013

Public/Private keys, OpenSSL, RSA, PEM and DER

It's no wonder that many people stay away from security in their web apps. Trying to do something that sounds simple like "create a public/private key pair" ends up being 30 minutes of Googling. This is because there are variations and differences but also similarities between the various types of keys and certificates and then not all systems support all types. Add in things like Putty which works great but which has its own key format and it is pain on a stick. Anyway, I had openssl downloaded so to create the "Key Pair", simply run

openssl genrsa -out mycert.pem 2048

Now, the first confusion is that although you will find lots of people saying that this generates a public/private key pair you will notice that it only outputs one file and this is because a public and a private key are not really two distinct entities but rather the public key contains a subset of the information that is contained in the private key - enough to encrypt data but not enough to decrypt it or to work out the private key. For this reason, you sort of have a key pair but really it is just a single private key in a single file. Soooo. We probably want our public key to exist separately as well to give to people who want to encrypt data for us so we need to use another command to tell openssl to export the public key and to convert it, if required, into the raw digital format that some systems prefer to the base-64 encoded DER format of PEM.

openssl rsa -in mycert.pem -outform der -out mycert.pub.der -pubout
openssl rsa -in mycert.pem -outform pem -out mycert.pub -pubout

This produces another file (or two) which, in the first case above, is DER rather than PEM encoded but which only includes the public key part of the private key. The second example does the same thing but keeps the exported key in PEM format. This is enough for you to start your public-key encryption process but note that when you search for help on the topic on Google, be aware that as well as public/private keys, you might also be looking at certificate requests (like you send when you want an ssl certificate), a certificate, a certificate chain, or some combination of the above. For this reason, there are various switches to openssl so keep your eyes peeled before trying something out. On the other hand, things tend to fail quite quickly if you have done something wrong like trying to export a public key in one format to a public key in another - you apparently have to do this from the original private key instead!

Monday, 18 November 2013

Why we cannot have perfect encryption and why we can't trust anybody

Something that has only really dawned on me recently through the whole NSA spying issue and what they (and possibly others) can and can't know about me from the internet traffic I create and consume. Part of my job is writing software that is secure so people can trust it so I am very interested in what I can achieve in terms of security in my software and possibly in other products that I could produce. Let's be honest, there is much money to be made in good security products so why not? This led me to my first question that set off a whole chain of events.

1) How is my data secure from snooping?

Firstly, I use SSL for all communication links but we know SSL fails in various areas. Firstly there are a couple of known exploits Beast Attack and RC4 attacks possibly the most famous or at least the most famous of the ones that are possible even on newer implementations of TLS/SSL. This brings up another important issue, "Use TLS 1.2" they all say but we cannot. Quite simply, too many end-users are still using older browsers that don't support it and amazingly, even some new browsers do not support it out of the box: Browser support. If we are honest, many of these attacks are highly unlikely but TLS/SSL is still potentially insecure.

Secondly, what about the ciphers I use, both in TLS/SSL and also for the data itself? Well, everyone knows AES256 is amazingly hard to crack (or do we?) but what about the ciphers used in SSL. Again, a very sloppy organic growth of SSL and the need to support the widest range of web servers and browsers/clients means a frightening amount of cipher suites are possible during an SSL handshake, the web server supports a set, the browser supports a set and they negotiate which to use (my guess would be the first to match regardless of any type of ordering). The only way to reduce these is to use newer versions of TLS and browsers but again, that is not possible for public sites that want to support everyone. Even more frightening, is a classic man-in-the-middle attack that intercepts the cipher negotiation and tells both ends to use the weakest protocols it has in order to make interception and decryption much easier.

We can't really trust SSL as it stands. And back to AES256, I only trust it because that is the common consensus, I don't personally know whether it is solid, I do not know whether large governments have ways to make the cracking easier, I do not know whether specific implementations have back-doors into them put there by corporations, hackers, governments or anyone else. And Open Source? Great in some ways but how many people are really watching all the commits and spotting exploits that have been inserted by someone or other?

2) If I assume my SSL and my AES256 is secure in itself, how secure is the key?

Well, many people say that encryption should be secure if everything about it is known apart from the key. It's a bit like letting people examine the door to the vault and assuming that any weaknesses will be pointed out (they could be seen but not pointed out!) but if someone has the key, it is game over. This is part of the motivation for cryptographic hashing but this just opens other problems instead.

So how can we use our symmetric key? Well two options, we generate one using a key generator or we stretch a password to create the key dynamically. Both of these, of course, move the key problem to a storage problem.

Assuming the key generator and the password stretching algorithm are sound (again, most of us can never know this - it is easier to know something isn't true than to prove it is) the key itself or the password used to create it need to be stored somewhere. Database? Lots of ways in which that can be accessed, particularly with Cloud Computing, often you are sharing a database server which means that someone (not you) has a Super Admin account and could access ANY data in your database. If you are using a Cloud Service DO NOT store keys in a database.

We could store it on disk somewhere but again, if the storage is shared, other people potentially have access, another client perhaps (by mistake) a dodgy sys admin like Edward Snowden who (rightly or wrongly) abuses his access privileges or indeed a company who accidentally or purposely allow governments to access this data for "legitimate reasons". The problem with keys is they cannot be encrypted, otherwise you need another key which cannot be encrypted, eventually it is stored somewhere "in the clear".

The question is, what can you use to secure your keys? Servers and many higher-end modern PCs have a hardware security module which allows, theoretically, secure storage of keys in a way that cannot be obtained by reading hard disks or whatever but these are, of course, designed more to protect against stolen computers. Malware running in software is still able to obtain these keys in the same way the system obviously needs to access them. Again, can we trust the corporations not to put back doors in? Can we trust them not to make a mistake in the design which allows access to them? Do we know that governments cannot remotely login to our servers and get these keys?

In Cloud Computing systems, keys are stored in the infrastructure. Again, of course, we have no idea how these are stored and whether they are secure. If I use encryption in my Azure system and the key is the SSL certificate I uploaded to Azure (and told it the private key password) do I really know that this is secure? Of course not. The system does need access to these keys to share them amongst web instances for TLS/SSL connections but at the same time, anyone in the world could have gained access to my system and I wouldn't even know.

Conclusion
So what? Clearly, we are a long way from this Utopian idea that encryption can buy us the privacy some people so eagerly desire. I have nothing to hide in one sense but I also do not relish the idea of armies of complete strangers analyzing my digital footprint to find out about me.

Before everyone says we need to go back to the old days, remember that trust has always been an issue and I suspect always will. Chelsea Manning was trusted with US secrets and blow the whistle, the same for Edward Snowden and I'm sure many others we will never know so the idea that the communication system/storage is the only risk is, in itself, flawed. While we need to communicate with someone else, there is always the chance that the other person is the security hole.

That being said, there are some things we can do to help protect ourselves but only to a point.
  1. Know where your security risks are and the level of risk. For instance, if you are relying on TLS/SSL to protect communications, that is reasonable but record it somewhere so that if this was ever broken badly, you would immediately know where you are exposed. Sadly, I suspect most people have never done a basic risk analysis of their system.
  2. Stick with the most widely regarded patterns and modes of encryption and update regularly. For instance, why are people still using MD5 for password hashing? Also, understand why you choose these things, why is bcrypt probably better than SHA-256 for password hashing (because it is deliberately slow)
  3. Never, never, never invent your own crypto-systems, ciphers, key derivation functions etc. unless they go through peer review. Just because MD5(data) + MD5(data) seems like it will be twice as strong doesn't mean it is! The people who write these things are either stupidly clever or they take a LOT of time and effort and research to ensure they are solid.
  4. Think twice before using shared hosting servers if you are selling a security product. You can still use a data centre but if the machines are yours, you know what is on them and you know, to an extent, that they are not accessible by people who shouldn't be accessing them.
  5. Linux servers are more likely to be secure to back-door remote access than windows but I think if Windows really did have it, people would know by now with the various pen-test tools available.
  6. Rotate keys over time so that if a key was obtained, the chances of being able to decrypt older or newer data is reduced, also, the attacker would need to steal the key and the data at the same time.
  7. Be very aware of social attacks (possibly the most common attack vector) and the amount of access people have to your systems. If you are a startup, does the sales director really need access to your code just because they are on the same network? Just by doing very basic access controls, you could prevent many attacks.
  8. Don't base your entire business case on the fact you can trust any system. Even modern "browser based" encryption systems that promise the earth are mostly quite weak due to weaknesses in the design of JavaScript and browsers (for all the same reasons as before).

Wednesday, 13 November 2013

What every developer should know about password hashing before writing it!

Another leak here: Macrumors leak and I start getting all annoyed again about how often online systems are NOT using best-practice when it comes to password storage. It sounds like the system was hacked due to account privilege escalation but it doesn't really matter. If a developer does not have a proper understanding of password hashing, they should not be allowed anywhere near a password system, they should certainly not be writing their own password system.

Sadly, much data on the web on such matters is either inaccurate, too opinionated and often out-of-date but this is not always easy to notice when most forums do not expire content related to technical information, even though they probably should. Anyway, although I'm sure there are plenty of other articles out there about password hashing, some of which are written by people who know much more than I do, I want to write something by way of an introduction to password hashing and how it is used and why. Hopefully, when people understand, they will stop following bad practices and even if we see breaches in the future, users will not be so worried about it.

I guess before we even start, web apps themselves should be secure and follow best-practice guidelines. For instance, using stored procedures in a certain way would mean the web app cannot access everyone's password even when hacked. Likewise, the connection should not be made using the sa user which can do anything on the database anyway. These types of practices are beyond the scope of this article but, in my opinion, another area that every developer should be familiar with is https://www.owasp.org who look after security so other people do not need to do their own research. They have a whole wealth of information on security and you should know where it is.

Passwords

Right, we all know what passwords are. We hopefully also know that most people use the same password on most of the web apps they are members of. That means that if only one site spills the beans, the other sites are vulnerable. Your web app is not an individual, it is part of a community and you should take that responsibility seriously. The best way you can manage passwords is to use a single-sign-on service like google, twitter or PixelPin so that the password issue, including how to securely store them is in the hands of companies who specialise in it and haven't glued two pieces of paper together and written your password on it in crayons like some sites appear to!

Doing it your way

I know many of you are saying, "but I don't want to use a 3rd-party for reason xyz". In most cases, I would disagree with you but lets assume that you really need to reinvent the wheel and make a user create a totally new account on your site.

Firstly, I hope that you can easily understand why you should never, never, never, never, never store user passwords in plain text. There are so many ways in which database contents can be leaked that this would be criminally negligent. A rogue worker, someone at a data center with machine access, a hacker, another customer who has shared access, a careless developer, a vulnerability in a framework. All of these are risks and a plain text password is plain wrong! What's that? You need to store passwords in plain text so you can send them to users who forget? Please do not do that! Email is insecure in many ways, including people reading over your shoulder but also, many connections are not encrypted and the password is there for anyone reading the network connections.

Encrypting passwords (using symmetric encryption like AES or DES - encryption that can be decrypted) is another issue that has surfaced recently due to the Adobe breach. The general received wisdom is that although the encryption itself might be very solid (at least hopefully, it is not using something old and weak like DES) that an attacker might well have access to the decryption key which makes every single piece of data decryptable. In most cases, you should not use symmetric encryption for passwords, using something called hashing is preferred. For one reason, doing hashing properly and choosing a strong password makes cracking the password as good as impossible for anyone.

Encrypting Passwords

So you obviously understand that if a password is not to be stored in plain text, it obviously needs to be 'encrypted' in some way. The three broad schools of encryption are called symmetrical encryption, asymmetrical encryption and crytographic hashing. For the purposes of this discussion, the first two are the same in that encrypted data can be decrypted by a key and as discussed above, this raises the danger of the key being accessed/stolen by an attacker at which point, none of the data is safe any more. Hashing is a little different in that the password is encrypted and stored but it cannot be decrypted meaning that theoretically, even if an attacker stole the hash, they wouldn't know what the original password is, the application is also unable to decrypt it so how can that be useful?

Hashing Passwords

The trick to using hashing is a property of a hashing algorithm (there are various algorithms available, we will discuss this later) in that if you hash the same data using the same algorithm, it will ALWAYS produce the same output which is a "hash", a series of bytes, most often seen as a base-64 encoded string  of a certain length (dependent on the algorithm used) which makes it easy to read and transmit across channels like the web which are not binary friendly.

When a user creates an account, you store the hash of their password. When they login, you hash the password they type and if it matches the hash in the database, they typed in the correct password, otherwise they didn't. Theoretically, because the algorithm produces a hash, more than one password might create the same hash (a "collision") but this is so unbelievably unlikely that it is not considered a problem. In fact, if an algorithm is found to have too many collisions then it is discredited and not used any more.

As a basic example if you hash the word "password" (without the quotes) using the common algorithm known as MD5, it will produce a hash that looks like this: 5f4dcc3b5aa765d61d8327deb882cf99 Even though I told you that this hash was produced from "password" there is no known way to directly compute "password" from this hash. It is consider a one-way function. This is a bit like multiplication in maths where it is very easy to multiply two numbers to produce a result but much harder to work out what these factors are just from the result.

At its most basic level, hashing already adds some security because if someone read your database and saw that your password was stored as 5f4dcc3b5aa765d61d8327deb882cf99, they would not immediately know that your password was "password".

Basic Hashing Weaknesses

There is, however, a problem with just using a pure hash. The weakness is because the hashing algorithm will always produce the same result for the same input so if I hash a load of common passwords (including "password") and store the hashes in a big lookup table, when I come across 5f4dcc3b5aa765d61d8327deb882cf99, I can look it up in my table and see that it was produced from "password".

Ineffective Improvements

Developers are a strange breed and sometimes think they understand things that they don't. For instance, somebody thinks that rather than using a common hash algorithm, they will do something strange like invent their own hash algorithm, either from scratch or based on other algorithms. In most instances this causes something that is either no better than a basic algorithm or in some cases much weaker. The amount of time and work that has been put into attacking common algorithms proves how strong they are. If your home-made algorithm has not been reviewed in the same way (which it won't be!) then there is no way you will write anything that is any good. Please don't ever invent your own mechanisms, they are not needed since the correct way is very easy to do.

Another ineffective improvement is to add a fixed "salt" to every password before storing it. The idea being that the salt is then added to a typed in password and the hashes are compared in the same way as before. The thinking is that if I add salt of, say, "thisismysalt" to the end of "password" before I hash it, I will not get 5f4dcc3b5aa765d61d8327deb882cf99 anymore but 1d63491d7f52a91da41213205b422062 in other words, when the attacker sees it in the database, it won't match their lookup table! Win? Nope sadly not. If the attacker gets enough passwords, they can assume certain things like the most commonly occurring hash is likely to relate to one of the top 10 most common passwords like "password123", "letmein", "password" etc. in which case, it will not take long to work out the system used for hashing and what salt is used, all the attacker has to do is start hashing various combination of the top 10 passwords with data after it and perhaps before it. Some cracking systems can perform billions of such hashes per second so we might only be talking about minutes and as soon as one breaks and shows that the password was "passwordthisismysalt" it would not take an expert to realise how the passwords are constructed, at which point, the attacker simply re-hashes their password list with "thisismysalt" on the end and job-done!

Another popular method is to perform multiple hashes on the same data. Rather than running MD5 once, you run it in a loop hashing the hash for, say, 1000 times. This adds some amount of time to the process both for the attacker and the system itself but it still doesn't really help when the attacker has enough data. In the same way as attacking fixed salt, they can try various combinations of hash iterations to find what they are looking for since they can still assume that the most common hash probably relates to one of the top 10 most common passwords.

Using Salt Properly

If you are going to add a salt to a hash, a very minimum, it should be a different salt per user. Ideally, it should be random, relatively long (i.e. hard to guess) and never re-used either across users or for the same user (otherwise if historical data was eventually cracked, the information could be used to attack newly hashed data). PHP now includes very easy-to-use password hashing functions and these should be used if they are available. They are shortcuts to using bcrypt directly which makes it much easier for people who do not understand all the options. The defaults are good but they can be improved over time. .Net has security classes that perform the same functionality depending on what exactly you want.

The beauty of a "variable salt" is that you no longer have patterns of data in your database, you can no longer determine which of the passwords relates to "password" and which relates to "thisisaveryhardpasswordtocrack becAuseitislongandh@sweirdcharacter£init" this makes the work much harder even, as is the case with bcrypt, the salt is stored alongside the password hash.

Is Pepper Good

Another tactic that is often cited but should not be confused with the purpose of salt is pepper. The idea is that pepper is a deterministic way of introducing additional data to the password before hashing but it is not stored with the hash or salt in the database so is unknown to an attacker who only has the database data and not the source code. It could be something like the userid transformed in some way such as reversed, upper cased and then perhaps with a long fixed string added to the end so that it is still different per user but does not have to be cryptographically random like salt should be.

Defeating attackers with size and speed

There are, above these other techniques, two ways to defeat an attacker. The first uses the size of an algorithm to make it much less likely that an attacker can cover the required number of "guesses" before they make a match. For instance, MD5 only produces an output of 128 bits which would mean that guessing every value of MD5 would take 3^38 attempts and an average hit would take half of this time 1.7^38. Currently, this sounds like an impossible task but with some other data to hint at the answer and enough computing power (in excess of 7 billion guesses per second), these attacks are quite trivial. Compare this with something like SHA-512 which produces 512 bits of output and the number of combinations possible is now 1.3^154 which is massively more complex than MD5. This is also slower to guess (circa 200M/second) so this is a good way to defeat an attacker.

The alternative, arguable cleverer, way is to slow down the process deliberately. Imagine if your hash algorithm took, say, 300 milliseconds to compute. This would not be noticeable when one user was logging into a system but would slow down an attacker who could no longer try millions of guesses per second but just 3 per core per second! bcrypt, which uses blowfish is designed exactly for that and in many ways is a great defence for passwords. The only major problem is that the slowness incurs a memory overhead which makes it unsuitable for low-memory devices although even that is likely to become less of a problem over time.

Recommended Setup

Whatever I recommend is likely to end up being controversial but I might as well be brave! If you are using PHP, the following code is all you need to store a password and check it again afterwards:

// Note, requires PHP 5.5 Look at each function here http://us3.php.net/manual/en/ref.password.php to find equivalent code for earlier versions
function signup($username,$userpassword,etc...)
{
   // Do whatever checks are needed
   $hashedPassword = password_hash($userpassword, PASSWORD_DEFAULT);

   // Save $hashedPassword to database
}

function authenticate($username,$userpassword)
{
   // Get hashed password from database where username = username into $row
   if (password_verify($userpassword, $row->password))
   {
      // Success
   }
   else
   {
      // Failure
   }
}
One of the great things with these functions is that you can upgrade the "cost" or algorithm of the password hash as you go along. You can then test the existing database entries with password_needs_rehash() to see whether it is out-of-date. If so, check the password entered matches and if so, rehash the entered password and update the database!

.Net is a little different in that there is no built-in blowfish implementation. You can do one of two things, you can bring in another library like BouncyCastle that provides bcrypt or you can use PBKDF2 instead which is designed to do a similar thing to bcrypt but is not so cost-intensive. You can achieve this like this:
//Disclaimer: I have not used this code so it might not work exactly out of the box. I use PBKDF2 to create encryption keys using code like this
public void signup(String username, String password)
{
   // Do whatever checks are needed
   // Generate random salt using your own function or something like System.Web.Security.Membership.GeneratePassword
   var rfc2898 = new Rfc2898DeriveBytes(password, randomSalt);
   var hashedPassword = rfc2898.GetBytes(32);
   var combinedHash = randomSalt + "$" + Convert.ToBase64String(hashedPassword);    // Use dollar to make it easier to split later

   // Save the username and combinedHash in the database
}

public void authenticate(String username, String password)
{
   // Get hashed password from database where username = username
   // Split hashed password from database into dbpassword and dbsalt using the $ symbol
   var rfc2898 = new Rfc2898DeriveBytes(password, dbsalt);
   var hashedPassword = rfc2898.GetBytes(32);        // This should match what is in the database if successful

   if ( hashedPassword == dbpassword )
   {
      // Success
   }
   else
   {
      // Failure
   }
}
Edit: Thanks to Duncan Smart for pointing out that MS already do what I was attempting above here:  http://msdn.microsoft.com/en-us/library/system.web.helpers.crypto.hashpassword and http://msdn.microsoft.com/en-us/library/system.web.helpers.crypto.verifyhashedpassword

Conclusion

If you follow the suggestions above, then what an attacker gets is a randomised salted-hash, they have few if any clues (they may or may not guess what hash algorithm you are using) in which case they would have to resort to some kind of brute force. Even if they had a known-plaintext (one of the hacked hashes is for a password they know), they would have to spend some time working out what system is in use, and even if they eventually work out you are using bcrypt with 100 iterations and they know the salt from the database, they would have to construct a brute-force against each hash, one at a time and they may or may not choose a hash that is generated from an easy password, which would significantly slow down the attacker. It would not prevent them from cracking any passwords but it would certainly make them think twice about whether the effort was worth it. If you added some pepper to the system, if they did not have the code, they would probably not be able to crack any passwords at all.

AES Encryption/Decryption in Android using SpongyCastle

At the moment, Android have handicapped the BouncyCastle security library included with Android so that it doesn't support some of the more common and useful encryption algorithms. I will assume this was done for performance reasons but also possibly because older handsets might not support some of the algorithms? Anyway, I don't care and I want AES256 encryption.

I found out that the writers of BouncyCastle had another project called SpongyCastle which is the full library for Android and includes all the features I wanted. It is easy to download from here. Once you have the library installed, it works much the same way as BouncyCastle but with different namespaces. Here is an example of a module that includes encryption and decryption using AES256, a randomly generated key, which is stored in private memory on the device and the ability to use this key to encrypt or decrypt data which can be stored in other files.

Any comments most welcome on whether I have got this correct for Android. Note I can't promise that I have used the most efficient or suitable cipher types but this does work! Also note that the initialization vector, which ensures that encrypting the same data multiple times does not produce the same ciphertext, is added to the encrypted data and stored with it, it must therefore, be stripped off before decryption.

I am unsure as to the best way to read in data from file with an unknown length. The method I have used below is to make a preset-sized buffer (larger than the data I am storing) and then once it is read in, using the length to create a new buffer and copying the data over. This ensures that the buffer length is correct otherwise the decryption gets confused about padding information.

package org.MyCompany.MyApp;

import java.io.FileInputStream;
import java.io.FileNotFoundException;
import java.io.FileOutputStream;
import java.io.IOException;
import java.security.NoSuchAlgorithmException;
import java.security.SecureRandom;

import javax.crypto.KeyGenerator;
import javax.crypto.SecretKey;
import javax.crypto.spec.SecretKeySpec;

import org.spongycastle.crypto.engines.AESFastEngine;
import org.spongycastle.crypto.modes.CBCBlockCipher;
import org.spongycastle.crypto.paddings.PaddedBufferedBlockCipher;
import org.spongycastle.crypto.params.KeyParameter;
import org.spongycastle.crypto.params.ParametersWithIV;
import org.spongycastle.util.Arrays;

import android.content.Context;

/**
 * Handles the security aspects of the app such as encryption and offline storage
 * Reference takens from http://android-developers.blogspot.co.uk/2013/02/using-cryptography-to-store-credentials.html 
 * @author Luke
 *
 */
public class SecurityModule 
{
    /**
     * Store the given serialized data onto the local device
     * @param pointsSer
     * @return
     */
    public static Boolean StoreMyData(String myData)
    {
        SecretKey key = CreateOrRetrieveSecretKey();
        if ( key == null )
            return false;
        
        WriteData(encrypt(myData.getBytes(), key.getEncoded()), "mydata.ser");
        
        return true;
    }
    
    /**
     * Get the cryptographically securely stored data for this application
     * @return
     */
    public static String GetMyData()
    {
        SecretKey key = CreateOrRetrieveSecretKey();
        if ( key == null )
            return "";
        
        byte[] data;
        try 
        {
            data = ReadData("mydata.ser");
        } 
        catch (IOException e) 
        {
            e.printStackTrace();
            return "";
        }
        
        byte[] decrypted = decrypt(data, key.getEncoded());
        return new String(decrypted);
    }
    
    /**
     * Encrypt the given plaintext bytes using the given key
     * @param data The plaintext to encrypt
     * @param key The key to use for encryption
     * @return The encrypted bytes
     */
    private static byte[] encrypt(byte[] data, byte[] key) 
    {
        // 16 bytes is the IV size for AES256
        try
        {
            PaddedBufferedBlockCipher cipher = new PaddedBufferedBlockCipher(new CBCBlockCipher(new AESFastEngine()));
            // Random iv
            SecureRandom rng = new SecureRandom();
            byte[] ivBytes = new byte[16];
            rng.nextBytes(ivBytes);
            
            cipher.init(true, new ParametersWithIV(new KeyParameter(key), ivBytes));
            byte[] outBuf   = new byte[cipher.getOutputSize(data.length)];
        
            int processed = cipher.processBytes(data, 0, data.length, outBuf, 0);
            processed += cipher.doFinal(outBuf, processed);
            
            byte[] outBuf2 = new byte[processed + 16];        // Make room for iv
            System.arraycopy(ivBytes, 0, outBuf2, 0, 16);    // Add iv
            System.arraycopy(outBuf, 0, outBuf2, 16, processed);    // Then the encrypted data
            
            return outBuf2;
        }
        catch(Exception e)
        {
            e.printStackTrace();
        }
        return null;
    }
    
    /**
     * Decrypt the given data with the given key
     * @param data The data to decrypt
     * @param key The key to decrypt with
     * @return The decrypted bytes
     */
    private static byte[] decrypt(byte[] data, byte[] key) 
    {
        // 16 bytes is the IV size for AES256
        try
        {
            PaddedBufferedBlockCipher cipher = new PaddedBufferedBlockCipher(new CBCBlockCipher(new AESFastEngine()));
            byte[] ivBytes = new byte[16];
            System.arraycopy(data, 0, ivBytes, 0, ivBytes.length); // Get iv from data
            byte[] dataonly = new byte[data.length - ivBytes.length];
            System.arraycopy(data, ivBytes.length, dataonly, 0, data.length    - ivBytes.length);
    
            cipher.init(false, new ParametersWithIV(new KeyParameter(key), ivBytes));
            byte[] decrypted = new byte[cipher.getOutputSize(dataonly.length)];
            int len = cipher.processBytes(dataonly, 0, dataonly.length, decrypted,0);
            len += cipher.doFinal(decrypted, len);
    
            return decrypted;
        }
        catch(Exception e)
        {
            e.printStackTrace();
        }
        return null;
    }
    
    /**
     * Check for a currently saved key and if not present, create a new one
     * @return The newly or previously created key
     */
    private static SecretKey CreateOrRetrieveSecretKey()
    {
        try
        {
            byte[] keyBytes = ReadKey();
            SecretKey key;
            if ( keyBytes == null )
            {
                key = GenerateKey();
                WriteKey(key.getEncoded());
            }
            else
            {
                 key = new SecretKeySpec(keyBytes, 0, keyBytes.length, "AES");
            }
            return key;
        }
        catch( NoSuchAlgorithmException e )
        {
            e.printStackTrace();
        }
        return null;
    }
    
    /**
     * Generate a key suitable for AES256 encryption
     * @return The generated key
     * @throws NoSuchAlgorithmException
     */
    private static SecretKey GenerateKey() throws NoSuchAlgorithmException {
        // Generate a 256-bit key
        final int outputKeyLength = 256;

        // EDIT - do not need to create SecureRandom, this is done automatically by init() if one is not provided
        KeyGenerator keyGenerator = KeyGenerator.getInstance("AES");
        keyGenerator.init(outputKeyLength);
        SecretKey key = keyGenerator.generateKey();
        return key;
    }
    
    /**
     * Write the given data to private storage
     * @param data The data to store
     * @param filename The filename to store the data in
     */
    private static void WriteData(byte[] data, String filename)
    {
        FileOutputStream fOut = null;
        try {
            fOut = MyApp.getAppContext().openFileOutput(filename, Context.MODE_PRIVATE);
            fOut.write(data);
            fOut.flush();
            fOut.close();
        } catch (Exception e) {
            e.printStackTrace();
        }
    }

    /**
     * Write the given encryption key to private storage using the hard-coded filename
     * @param key The key to write
     */
    private static void WriteKey(byte[] key)
    {
        WriteData(key, "myappkey");
    }
    
    /**
     * Read data from private storage using the given filename
     * @param filename The filename whose contents to read
     * @return The contents of the file or null
     * @throws IOException
     */
    private static byte[] ReadData(String filename) throws IOException
    {
        byte[] key = new byte[5096];
        Arrays.fill(key, (byte)0);
        FileInputStream fOut = null;
        try 
        {
            fOut = MyApp.getAppContext().openFileInput(filename);
            int length = fOut.read(key);
            byte[] key2 = new byte[length];
            System.arraycopy(key, 0, key2, 0, length);
            fOut.close();
            return key2;
        } 
        catch(FileNotFoundException e)
        {
            return null;
        } 
    }
    
    /**
     * Read the encryption key from private storage
     * @return
     */
    private static byte[] ReadKey()
    {
        try 
        {
            return ReadData("myappkey"); // Hard-coded filename representing the encryption key
        } 
        catch (IOException e) 
        {
            e.printStackTrace();
        }
        return null;
    }
}


Tuesday, 5 November 2013

GMail email relay from PHP in Azure worker role

I have a worker role that needs to connect to a web application regularly to force it to keep it's database connection established (poor I know!) but in this case, the connection can take 10 seconds to connect and if you are the unlucky person who hits it at that point, you have to wait.

Anyway, I wanted to test how the Azure PHP worker role worked and check that it did what I thought it would do so I decided I would get it to send me an email in the worker loop to ensure it was all happy and running properly.

When you add an Azure worker role, it creates a whole load of weird files that run certain commands, set up paths etc but the only one you need to care about is index.php which is invoked by default when the worker role starts. The way it works, therefore, like .net worker roles is that you should have some PHP that sits inside a while(true) loop (assuming you want it to loop) and which then calls whatever PHP code you need it to.

In my case, I was using swiftMailer to relay email via our Google accounts and swiftMailer uses PHP's open ssl extension (which was already enabled by default).

Anyway, this was all fine but the emails weren't coming through, although the role was running. This was because I put the bit I expected to fail inside a try/catch to ensure that any errors would not make the deployment break!

I ended up enabling remote desktop for the role (see previous post!) and went straight into d:\windows\temp\php53_errors.log and saw the following error:

PHP Warning:  fsockopen(): SSL operation failed with code 1. OpenSSL Error messages:
error:140770FC:SSL routines:SSL23_GET_SERVER_HELLO:unknown protocol in E:\approot\swiftMailer\classes\Swift\Transport\StreamBuffer.php on line 233
PHP Warning:  fsockopen(): Failed to enable crypto in E:\approot\swiftMailer\classes\Swift\Transport\StreamBuffer.php on line 233
PHP Warning:  fsockopen(): unable to connect to ssl://smtp.gmail.com:587 (Unknown error) in E:\approot\swiftMailer\classes\Swift\Transport\StreamBuffer.php on line 233


Fortunately, I found a useful blog post with a similar issue on another framework and it is related to how Google works and a few shortcomings in the swiftMailer's SSL functionality. Long story short, you need to use the following config: $transport = Swift_SmtpTransport::newInstance('smtp.gmail.com', 465, 'ssl'); Note the use of port 465 (not 587) and protocol ssl.

So a complete example for a simple PHP mailer using swiftmailer is as below:

require_once('.\swiftMailer\swift_required.php');
        
$transport = Swift_SmtpTransport::newInstance('smtp.gmail.com', 465, 'ssl');
$transport->setUsername('accountUsername@gmail.com');
$transport->setPassword('yourPassword');
    
$mailer = Swift_Mailer::newInstance($transport);
    
// Create the message
$message = Swift_Message::newInstance();
$message->setSubject('This is the subject');
$message->setFrom(array('no-reply@server.com' => 'No reply'));
$message->setTo(array('someone@somewhere.com' => 'john'));
$message->setBody("This is the actual content");
$mailer->send($message);

Cannot remote desktop to Azure Worker Role

Spoiler: The certificate was missing!

Well I couldn't before, but I can now! I have a PHP Azure Cloud Service. It is not quite as integrated into Visual Studio as .Net ones (not surprisingly) so you have to do certain operations manually in the configuration files. One of these is Remote Desktop access and this is a bit messy to do by hand but then I noticed that you can enable it on the Azure portal after deployment (which is nice).

I enabled it, chose the certificate to use for the RDP connection and carried on. I RDP'd into the site but accidentally logged into one of the web roles, when I was intending to look into problems with the worker role. I tried instead to RDP into that but this time I got, "The user name or password is incorrect". After trying 5 times to make sure that I had typed the long random password in correctly, I was confused. I only set the credentials in one place and the RDP had correctly picked up the username I had chosen so what gives?

It took me a while but then I remembered choosing the certificate when setting up RDP and, in my case, the worker role did not have those certificates loaded since they were used for https on the web roles. I added the certificates to the worker role, re-deployed and it all worked fine. As with many things, it makes perfect sense once you fix it, the certificate is used for encryption and if it is not present, rather than a useful error, it encrypts the data incorrectly (or not at all) and then fails authentication at the other end.