Android - Best way to implement encrypted IM [closed]

Question

I'm creating and end-to-end encrypted IM for Android, using PHP and GCM as the middle man to deliver the messages.

I've got the IM part nailed down, but now I'm looking to implement an end-to-end encryption on all messages.

My IM is text-only and has a maximum payload of 4kb (the max GCM supports).

Whats the best way to implement this on Android? AES + RSA?

Is there any default library for implementing this in a secure and easy way?

Answer 1

It is hard to say what is best, the solution also depends on how secure you want this to be.

1. Fixed RSA keypair - you could generate an RSA keypair and use them to encrypt the traffic. This however will not protect the conversation from someone who decompiles your APK file and extracts the RSA keys - though you can try to make this hard by obfuscating your code and hiding the key by XORing it or using reflection to access it on your application. There are some relevant suggestions here and questions/answers about implementation here.
2. Dynamic encryption keys with SSL - the encryption keys to use is negotiated for each client separately, on the fly. For this the clients need to trust the server (i.e be sure that they are really talking to your server, not someone else's). This is very much like SSL, where the server proves its identity by providing a certificate signed by a third party. If using SSL is an option, there is an answer for using self-signed certificates here.
3. Dynamic encryption keys customized - you could set up your own security scheme, but then you'll have to make it bulletproof yourself.