A Hybrid Cryptosystem Using Java, AES (secret-key) and RSA (public-key) Encryption

Sorry, comments are closed.

One disadvantage of an RSA public-key cipher is that the size of the message in bytes that you can encrypt is limited by the size of the bit key (modulus) length in bytes. For this reason it’s ideal for verifying the identity of a message sender, but not for the secure transmission of larger blocks of data. This is why, just like in SSL/TLS, RSA is best suited for use in conjunction with a secret-key cipher like AES. This is a hybrid cryptosystem.

A hybrid cryptosystem is one in which both asymmetric (public-key) encryption and symmetric (secret-key) encryption are used together to securely encrypt messages. More precisely, the public-key is used to encrypt the secret-key, which is what is used to encrypt the message. The encrypted secret-key and encrypted message are sent in every transmission. Identifying which is which is implementation dependent.

An Example Scenario

Here is an example scenario:

  1. ACME Corporation is a supplier that has a public-key that can be used for encryption. The company would like all billing order transmissions from vendors to be secure, and so they require that vendors use their X.509 certificate (containing the public-key) to encrypt this message data. However, they’ve realized the billing orders are too large, and so they implement a hybrid cryptosystem.
  2. As a vendor, you are given the certificate by the ACME security officer, and you load it into your Java truststore called cacerts.
  3. You create an AES cipher that contains a secret-key, which is randomly generated for each encrypted message.
  4. You encrypt the billing order to ciphertext using the AES secret-key.
  5. You retrieve the ACME certificate from the truststore, pull out the public-key, and then encrypt the secret-key with the public-key.
  6. You transmit to ACME over TCP/IP sockets the encrypted secret-key and encrypted billing order ciphertext using a format as agreed upon.

It’s time for a word of warning. ACME Corporation’s security officer is cheap. He decided that it wasn’t worth the cost to have their certificate signed (validated) by a recognized certificate authority. You don’t really know the public-key contained within that certificate is from ACME, other than by their given word. If retrieval of the certificate “over the wire” was compromised then the certificate belonging to ACME could actually be from a hacker.

Let’s look at that scenario:

  1. ACME hosts their certificate, and allows anyone to pull it down over TCP/IP sockets for use in encryption.
  2. ACME doesn’t realize the server hosting the certificate has been compromised. The certificate has been replaced by a hacker, and a bot script is now listening for all incoming billing orders.
  3. As a vendor you retrieve the certificate, which happens to belong to the hacker, but it contains all of ACME Corporation’s identity information. Because you can’t authenticate the certificate against a third party certificate authority you think it’s valid.
  4. You perform the aforementioned encryption steps, and then send the billing order to ACME.
  5. The hacker has the private-key associated with the certificate’s public-key because he created it. He can therefore decrypt the secret-key and the ciphertext.
  6. Being the intelligent hacker, he still forwards the billing order onto ACME’s accounting system, but also copies all of the decrypted privileged account information to a remote server in Russia.

This is why SSL/TLS works as it does. During the initial handshake the communication over TCP/IP is not encrypted, and this is when the public-key is exchanged. Any number of man-in-the-middle attacks could intercept this traffic and pass through a compromised key. Only by validating the signature of a certificate authority can a client browser be relatively sure communication is with a known identity.

Working with a Java Keystore

In order to demonstrate a working example it makes sense to illustrate how the certificate would be generated by ACME, as well as how you, the vendor, would use it. You’ll need the latest secure Java JDK, and you’ll also want to download the Java Cryptography Extension (JCE) Unlimited Strength Jurisdiction Policy Files 7. The Keytool executable will be included within the bin directory.

The first command will generate a public- and private-key pair, and store it within a Java keystore at the location of your choosing. For this tutorial I store it using the TMP environment variable, but ultimately this would be a poor and insecure location on both Windows and Linux. The user under which your Java code will execute will need to have permission to read the contents of the keystore or truststore you’re using.

> keytool -genkey -keyalg RSA -keysize 2048 -keystore "%TMP%\keystore.jks" -alias widgets -storepass changeit -keypass changeit -dname "CN=Wile E. Coyote, OU=Information Technology, O=ACME Corporation, L=Los Angeles, S=California, C=US"

Now a certificate conforming to the X.509 Public Key Infrastructure Certificate standard can be exported, which will contain the public-key.

> keytool -exportcert -alias widgets -keypass changeit -keystore "%TMP%\keystore.jks" -storepass changeit -file "%TMP%\ACME_Corporation.cer"

This is the point at which ACME Corporation’s security officer should have contacted a certificate authority, and generated a certificate signing request (CSR). Keytool has the appropriate commands for this as well, but it’s outside of the scope of the tutorial. Just know that Java does have an API for validating digital signatures in certificate chains.

Working with a Java Truststore

The certificate from ACME can now be imported into your truststore. As a general rule Java’s Keytool does not make the distinction between a keystore and a truststore. Oracle may even use the term interchangeably in documentation. However, you should never store private-key and public-key relationships within a truststore. The truststore should only be for trusted certificates, and a keystore should only be used for keys.

You may have a number of keystores, but typically only one truststore. The default truststore called cacerts exists under the JDK installation directory, and then at jre/lib/security as indicated in the example. This assumes you’ve copied the certificate first to the TMP directory on the filesystem.

> keytool -import -alias ACME -file "%TMP%\ACME_Corporation.cer" -keystore "C:\Java\jdk1.7.0_79\jre\lib\security\cacerts"

This completes the use of Keytool.

The AES and RSA Ciphers

I’m not going to review the AESCipher or RSACipher classes in their entirety. If you want to know the implementation details you can read my posts Asymmetric (Public-key) Encryption Using RSA, Java and OpenSSL and Symmetric Encryption Using AES and Java. The goal is instead to demonstrate the process by which a hybrid cryptosystem can be implemented.

However, for the sake of continuity the AESCipher and RSACipher classes are shown below. Both were created to optimize the chance for secure implementations, especially in the case of the AESCipher. Whenever encryption and decryption occurs across different attack vectors the cipher should be identical, and agreed upon by all parties. This includes the mode and padding, the initialization vector (IV) length in bytes, and the method of encoding.

package com.reindel.crypto.ciphers;
import org.apache.commons.codec.binary.Base64;

import javax.crypto.*;
import javax.crypto.spec.IvParameterSpec;
import javax.crypto.spec.PBEKeySpec;
import javax.crypto.spec.SecretKeySpec;
import java.security.InvalidAlgorithmParameterException;
import java.security.InvalidKeyException;
import java.security.NoSuchAlgorithmException;
import java.security.SecureRandom;
import java.security.spec.AlgorithmParameterSpec;
import java.security.spec.InvalidKeySpecException;
import java.security.spec.KeySpec;
import java.util.Random;
public class AESCipher {
    private Cipher cipher;
    private Base64 base64;
    private int passwordLength;
    private int saltLength;
    private int initializationVectorSeedLength;
    public AESCipher()
            throws NoSuchAlgorithmException, NoSuchPaddingException {
        this(new Base64(), 16, 16, 16);
    public AESCipher(Base64 base64, int passwordLength, int saltLength, int initializationVectorSeedLength)
            throws NoSuchAlgorithmException, NoSuchPaddingException {
        try {
            this.base64 = base64;
            this.passwordLength = passwordLength;
            this.saltLength = saltLength;
            this.initializationVectorSeedLength = initializationVectorSeedLength;
            cipher = Cipher.getInstance("AES/CBC/PKCS5Padding");
        } catch (NoSuchAlgorithmException noSuchAlgorithmException) {
            throw noSuchAlgorithmException;
        } catch (NoSuchPaddingException noSuchPaddingException) {
            throw noSuchPaddingException;
    private SecretKey secretKey;
    public SecretKey getSecretKey() {
        return secretKey;
    public String getEncodedSecretKey(SecretKey secretKey) {
        return base64.encodeToString(secretKey.getEncoded());
    public SecretKey getDecodedSecretKey(String secretKey) {
        return new SecretKeySpec(base64.decode(secretKey), "AES");
    public String encrypt(String rawText, int hashIterations, KeyLength keyLength)
            throws NoSuchAlgorithmException, InvalidKeySpecException, InvalidKeyException, IllegalBlockSizeException, BadPaddingException, InvalidAlgorithmParameterException {
        SecretKeyFactory secretKeyFactory = SecretKeyFactory.getInstance("PBKDF2WithHmacSHA1");
        SecureRandom secureRandom = new SecureRandom();
        byte[] seed = secureRandom.generateSeed(initializationVectorSeedLength);
        AlgorithmParameterSpec algorithmParameterSpec = new IvParameterSpec(seed);
        KeySpec keySpec = new PBEKeySpec(getRandomPassword(), secureRandom.generateSeed(saltLength), hashIterations, keyLength.getBits());
        secretKey = new SecretKeySpec(secretKeyFactory.generateSecret(keySpec).getEncoded(), "AES");
        cipher.init(Cipher.ENCRYPT_MODE, secretKey, algorithmParameterSpec);
        byte[] encryptedMessageBytes = cipher.doFinal(rawText.getBytes());
        byte[] bytesToEncode = new byte[seed.length + encryptedMessageBytes.length];
        System.arraycopy(seed, 0, bytesToEncode, 0, seed.length);
        System.arraycopy(encryptedMessageBytes, 0, bytesToEncode, seed.length, encryptedMessageBytes.length);
        return base64.encodeToString(bytesToEncode);
    public String decrypt(String encryptedText, SecretKey secretKey)
            throws InvalidKeyException, IllegalBlockSizeException, BadPaddingException, InvalidAlgorithmParameterException {
        byte[] bytesToDecode = base64.decode(encryptedText);
        byte[] emptySeed = new byte[initializationVectorSeedLength];
        System.arraycopy(bytesToDecode, 0, emptySeed, 0, initializationVectorSeedLength);
        cipher.init(Cipher.DECRYPT_MODE, secretKey, new IvParameterSpec(emptySeed));
        int messageDecryptedBytesLength = bytesToDecode.length - initializationVectorSeedLength;
        byte[] messageDecryptedBytes = new byte[messageDecryptedBytesLength];
        System.arraycopy(bytesToDecode, initializationVectorSeedLength, messageDecryptedBytes, 0, messageDecryptedBytesLength);
        return new String(cipher.doFinal(messageDecryptedBytes));

    public enum KeyLength {


        private int bits;

        KeyLength(int bits) {
            this.bits = bits;

        public int getBits() {
            return bits;

    protected char[] getRandomPassword() {
        char[] randomPassword = new char[passwordLength];
        Random random = new Random();
        for(int i = 0; i < passwordLength; i++) {
            randomPassword[i] = (char)(random.nextInt('~' - '!' + 1) + '!');
        return randomPassword;
package com.reindel.crypto.ciphers;

import org.apache.commons.codec.binary.Base64;

import javax.crypto.Cipher;
import java.io.IOException;
import java.security.GeneralSecurityException;
import java.security.PrivateKey;
import java.security.PublicKey;

public class RSACipher {

    public String encrypt(String rawText, PublicKey publicKey, String transformation, String encoding)
            throws IOException, GeneralSecurityException {

        Cipher cipher = Cipher.getInstance(transformation);
        cipher.init(Cipher.ENCRYPT_MODE, publicKey);

        return Base64.encodeBase64String(cipher.doFinal(rawText.getBytes(encoding)));

    public String decrypt(String cipherText, PrivateKey privateKey, String transformation, String encoding)
            throws IOException, GeneralSecurityException {

        Cipher cipher = Cipher.getInstance(transformation);
        cipher.init(Cipher.DECRYPT_MODE, privateKey);

        return new String(cipher.doFinal(Base64.decodeBase64(cipherText)), encoding);

The HybridCryptosystemTest Usage

The first thing to note about the test is that if you do decide to work directly with a keystore or truststore you’ll need to store passwords somewhere. That somewhere should never be inside the application source code. Instead it should be a configuration file, and the only user with read access should be the user under which the Java code executes. No other users should have permissions.

package com.reindel.crypto.ciphers;

import org.junit.Assert;
import org.junit.Before;
import org.junit.FixMethodOrder;
import org.junit.Test;
import org.junit.runners.MethodSorters;

import java.io.File;
import java.io.FileInputStream;
import java.io.FileReader;
import java.security.KeyStore;
import java.security.cert.X509Certificate;
import java.security.interfaces.RSAPrivateCrtKey;
import java.security.interfaces.RSAPublicKey;
import java.util.Properties;

public class HybridCryptosystemTest {

    Properties properties;

    static String rawText = "John has a long mustache.";
    static String cipherText;
    static String encodedAESSecretKey;
    static String encryptedAESSecretKey;

    public void before()
            throws Exception {

        FileReader fileReader = new FileReader(new File(System.getProperty("java.io.tmpdir") + "/acme.properties"));
        properties = new Properties();

    public void testFirstVendorEncrypt()
            throws Exception {

        String truststoreProp = properties.getProperty("truststore");
        String truststorePasswordProp = properties.getProperty("truststorePassword");
        String truststoreAliasProp = properties.getProperty("truststoreAlias");

        FileInputStream fileInputStream = null;

        try {

            KeyStore keyStore = KeyStore.getInstance(KeyStore.getDefaultType());
            fileInputStream = new FileInputStream(truststoreProp);
            keyStore.load(fileInputStream, truststorePasswordProp.toCharArray());

            X509Certificate x509Certificate = (X509Certificate)keyStore.getCertificate(truststoreAliasProp);
            RSAPublicKey rsaPublicKey = (RSAPublicKey)x509Certificate.getPublicKey();

            AESCipher aesCipher = new AESCipher();
            cipherText = aesCipher.encrypt(rawText, 1000, AESCipher.KeyLength.TWO_FIFTY_SIX);

            RSACipher rsaCipher = new RSACipher();
            encodedAESSecretKey = aesCipher.getEncodedSecretKey(aesCipher.getSecretKey());
            encryptedAESSecretKey = rsaCipher.encrypt(encodedAESSecretKey, rsaPublicKey, "RSA/ECB/PKCS1Padding", "UTF-8");

        } finally {
            if (fileInputStream != null) {


    public void testSecondACMEDecrypt()
            throws Exception {

        String keystoreProp = properties.getProperty("keystore");
        String keystorePasswordProp = properties.getProperty("keystorePassword");
        String keystoreKeyPasswordProp = properties.getProperty("keystoreKeyPassword");
        String keystoreAliasProp = properties.getProperty("keystoreAlias");

        FileInputStream fileInputStream = null;
        String decryptedAESPrivateKey;
        String decryptedCipherText;

        try {

            KeyStore keyStore = KeyStore.getInstance(KeyStore.getDefaultType());
            fileInputStream = new FileInputStream(keystoreProp);
            keyStore.load(fileInputStream, keystorePasswordProp.toCharArray());

            RSAPrivateCrtKey rsaPrivateCrtKey = (RSAPrivateCrtKey)keyStore.getKey(keystoreAliasProp, keystoreKeyPasswordProp.toCharArray());
            RSACipher rsaCipher = new RSACipher();
            decryptedAESPrivateKey = rsaCipher.decrypt(encryptedAESSecretKey, rsaPrivateCrtKey, "RSA/ECB/PKCS1Padding", "UTF-8");

            AESCipher aesCipher = new AESCipher();
            decryptedCipherText = aesCipher.decrypt(cipherText, aesCipher.getDecodedSecretKey(decryptedAESPrivateKey));

        } finally {
            if (fileInputStream != null) {

        Assert.assertEquals(encodedAESSecretKey, decryptedAESPrivateKey);
        Assert.assertEquals(decryptedCipherText, rawText);

Using the example scenarios described earlier you can track the process. Assuming the certificate is already in the truststore the testFirstVendorEncrypt() method shows (1) loading a KeyStore instance, (2) grabbing the certificate by alias, (3) pulling out the public key, (4) instantiating the AESCipher with the auto-generated secret-key, (5) encrypting the raw text, (6) and then finally encrypting the secret-key with the public-key using the RSACipher.

The testSecondACMEDecrypt() method is an example of code used by ACME Corporation once the message has been received. It would contain both the ciphertext and the encrypted AES secret-key. In order for the test to be accurate the encrypted values need to be saved from the testFirstVendorEncrypt() method. Persisting this data across tests in JUnit requires using the @FixMethodOrder annotation.

The order of operations is (1) loading a KeyStore instance, (2) grabbing the certificate’s related private-key by alias, (3) instantiating the RSACipher to decrypt the secret-key, (4) and finally instantiating the AESCipher in order to decrypt the ciphertext. The equals assertions demonstrate the encryption and decryption was seamless.

Summary Notes

A hybrid cryptosystem can be a powerful tool if used correctly, and with careful consideration for security across all possible attack vectors. This entire process is very similar to how SSL/TLS works in practice, but without the certificate authority. This Java implementation extends beyond HTTP, and should suit your needs as a protocol independent encryption solution.

Revision History

Revision 1.0, 09/21/2015 – Publication.

Code examples assume JDK 1.7+, and tests assume JUnit 4+.