GZIP and Blowfish!

Hi everybody!

I am facing a weird problem with an application that generates an encrypted URL with an URL encoded string as input.

These are are the steps followed in generating the encrypted URL:

//

// Steps for creating an encrypted URL:

//

//1 - Calculate the checksum

//2 - Concatenate the checksum and payload

//3 - URL Encode

//4 - Compress the string using zipIt (GZIP)

//5 - Encrypt using Blowfish and SunJCE

//6 - Base 64 encode

//7 - URL encode

//8 - Return the complete string

The decryption program follows these steps in the reverse order to generate the original payload.

This application has been in production for a few years. But, recently, a problem has been found in calling the encryption program from a loop to generate multiple URLs. Some of the encrypted URLs turn out to be invalid and this is sporadic.

The input payload is a combination of several parameters which are URL encoded and passed to the encryption program. The length is constant but the string itself could vary due to different parameter values. One thing that I noticed was that one of the parameters at the end, an error URL seems to work if it is a short string but if these are longer strings, that's when the problem is noticed.

I am pasting relevant pieces of the code here.

The parameter, "authKey" to encryptPayload is a pre-defined 56 byte fixed key. The payload string length is 253 characters which is fixed though the string value could be different due to the different parameter values which make up the payload.

publicclass MainClass{

public MainClass(){

}

publicstatic String encryptPayload (String payload, String authKey){

try{

BlowFishUtil bfUtil =new BlowFishUtil();

String sCheckSum= getMD5Checksum(payload);

String encodeBuffer = sCheckSum + payload;

encodeBuffer= URLEncoder.encode(encodeBuffer);

byte[] aryEncodeBuf = zipIt(encodeBuffer);

aryEncodeBuf= bfUtil.encryptSecretKey(authKey.getBytes(),aryEncodeBuf);

encodeBuffer= Base64.encodeBytes(aryEncodeBuf);

encodeBuffer= URLEncoder.encode(encodeBuffer);

return encodeBuffer;

}

catch (Exception e){

returnnull;

}

}

publicstatic String decryptPayload (String payload, String authKey){

try{

String decodeBuffer ="";

BlowFishUtil bfUtil =new BlowFishUtil();

decodeBuffer = URLDecoder.decode(payload);

decodeBuffer = Base64.decodeToString(decodeBuffer);

byte[] aryDecodeBuf = bfUtil.decryptSecretKey(authKey.getBytes(), decodeBuffer.getBytes());

aryDecodeBuf = unzipIt(aryDecodeBuf);

String sBuf =new String(aryDecodeBuf);

sBuf = URLDecoder.decode(sBuf);

int iPos = sBuf.indexOf("gh=");

if (iPos>=0){

String sURLCheckSum = sBuf.substring(0,iPos);

sBuf= sBuf.substring(iPos,sBuf.length());

String sCheckSum= getMD5Checksum(sBuf);

if (!sCheckSum.equals(sURLCheckSum)){

}

}

returnnew String(sBuf);

}

catch (Exception e){

returnnull;

}

}

privatestatic String getMD5Checksum( String mess ){

String hexHashCode ="";

String hashCode ="";

try{

MessageDigest md = MessageDigest.getInstance("MD5");

byte[] message = mess.getBytes("UTF-8");

md.update( message );

byte[] hash = md.digest();

for (int i=0; i < hash.length; i++ ){

int x = hash[i] & 0xFF;

if (x < 0x10){

hexHashCode +="0";

}

hexHashCode += (Integer.toHexString(x));

hashCode += hash[i]+" _ ";

}

}

catch (Exception e){

e.printStackTrace();

return"error";

}

return hexHashCode;

}

privatestaticbyte[] zipIt ( String parameterString )

{

byte[] zipped;

try{

ByteArrayOutputStream catcher =new ByteArrayOutputStream();

GZIPOutputStream gzipOut =new GZIPOutputStream( catcher );

byte[] bytesToZip = parameterString.getBytes();

gzipOut.write( bytesToZip, 0, bytesToZip.length );

gzipOut.close();

return catcher.toByteArray();

}

catch ( Exception ioe ){

ioe.printStackTrace();

return"error".getBytes();

}

}

privatestaticbyte[] unzipIt (byte[] buffer ){

ByteArrayOutputStream outBuffer =new ByteArrayOutputStream();

ByteArrayInputStream inBuffer =new ByteArrayInputStream(buffer);

try{

GZIPInputStream gzip =new GZIPInputStream(inBuffer);

byte[] tmpBuffer =newbyte[256];

int n;

while ((n = gzip.read(tmpBuffer)) >= 0){

outBuffer.write(tmpBuffer, 0, n);

}

return outBuffer.toByteArray();

}

catch (Exception e){

returnnull;

}

}

}

Following is the encryption/decryption module

// encryptSecretKey - uses javax.crypto.SecretKey to encrypt a java.util.String. It returns

//this string as a byte array.

//

publicstaticbyte[] encryptSecretKey(byte[] keyBytes,byte[] unencrypted)

{

try{

Provider sunJce =new com.sun.crypto.provider.SunJCE();

Security.addProvider(sunJce);

SecretKeySpec skeySpec =new SecretKeySpec(keyBytes,"Blowfish");

Cipher cipher = Cipher.getInstance("Blowfish");

cipher.init(Cipher.ENCRYPT_MODE, skeySpec);

byte[] myEncrypt = cipher.doFinal(unencrypted);

return myEncrypt;

}

catch (Exception e){

e.printStackTrace();

returnnull;

}

}

//

// decryptSecretKey - uses a java.crypto.SecretKey to decrypt a byte array of encrypted characters

//

publicstaticbyte[] decryptSecretKey(byte[] keyBytes,byte[] encrypted)

{

try{

Provider sunJCE =new com.sun.crypto.provider.SunJCE();

Security.addProvider(sunJCE);

SecretKeySpec skeySpec =new SecretKeySpec(keyBytes,"Blowfish");

Cipher cipher = javax.crypto.Cipher.getInstance("Blowfish");

cipher.init(javax.crypto.Cipher.DECRYPT_MODE, skeySpec);

byte[] decrypted = cipher.doFinal(encrypted);

return decrypted;

}

catch (Exception e){

returnnull;

}

}

Thanks in advance!

coderiyer

[11407 byte] By [coderiyera] at [2007-10-3 2:40:19]
# 1
As I said in the other place I'm amazed it works at all. What's the 'hashCode' variable for in the MD5 calculation? and why the += '0'?
ejpa at 2007-7-14 19:38:41 > top of Java-index,Security,Cryptography...
# 2

Hi everybody!

Is there anybody who could help me with this?

One thing I noticed was that this works a little better with JDK 5.0 than with JDK 1.3.1 (which is the version used by the caller of this program).

I would appreciate if somebody could take a look at this and provide some tips on some potential root causes.

Thanks in advance!

coderiyer

coderiyera at 2007-7-14 19:38:41 > top of Java-index,Security,Cryptography...
# 3

Is there a reason why it would not work sporadically? Moreover, this does not work only when it is called from a loop. And it seems to work better with JDK 5.0 than with JDK 1.3.1.

To answer your other question on why SSL is not being used, this program was implemented in ASP/Visual Basic and then ported to Java. I was not involved in the original design. And, unless warranted, I will not be able to change the design at this point.

My basic question still remains as to what could be causing this?

Could this be a JDK related issue? An issue with JCE? An issue with converting from string to a byte array and vice versa? Or, an encoding problem?

Thanks,

coderiyer

coderiyera at 2007-7-14 19:38:41 > top of Java-index,Security,Cryptography...
# 4
I think the += 0 encodes a leading zero where toHexString() leaves it off, e.g. he wants to encode 0x55 as "55" and 0xb as "0b", not "b".I don't see what hashCode is used for either.
ghstarka at 2007-7-14 19:38:41 > top of Java-index,Security,Cryptography...
# 5

> Is there a reason why it would not work sporadically?

I would expect it to come up with both valid and invalid URLs, but in what proportion is anybody's guess. Apparently it works most of the time, which I find pretty amazing considering what comes out of GZIP and encryption.

I understand your situation with an installed application and legacy code etc but I think you need to do the investigation for yourself, run it in a loop with random data and establish what proportion of URLs it generated are invalid, then either find a strategy for dealing with that or take the statistics to management as a case for changing the system.

In the code itself I'd be very concerned about the following:

decodeBuffer = Base64.decodeToString(decodeBuffer);

byte[] aryDecodeBuf = bfUtil.decryptSecretKey(authKey.getBytes(), decodeBuffer.getBytes());

In addition to the decoding, this is turning binary into a String and back again, and this isn't a lossless operation. I would use a base64 operation that returned a byte[] array directly and pass that directly to decryptSecretKey.

I'm also curious about why your step (3) above is considered necessary.

ejpa at 2007-7-14 19:38:41 > top of Java-index,Security,Cryptography...
# 6

Later .. you definitely need to do the fix above. You also need quite a few more "UTF-8" specifications and there is something asymmetric about your handling of "gh=" - where does this come from? Step 3 is unnecessary. I would also hope that returning null isn't the real way you handle exceptions: all these exceptions should be thrown by the methods concerned, not caught.

I would omit the URLDecoder step altogether - the result of that isn't necessarily a valid URL. And if you do that you should omit step (3) as well, it is redundant.

ejpa at 2007-7-14 19:38:41 > top of Java-index,Security,Cryptography...
# 7

Thanks for your tips. I am in the process of incorporating these into the program. By the way, I tested this already several times. It would work like a charm and I would send it over to the client only for them to notice the problem and then even I would run into a few invalid URLs. Even as we speak I ran the program in a loop iterating 500 times and it did not generate a single invalid URL (using JDK 1.3.1). I increased the counter to 5000 and I got a bunch of invalid URLs. One thing I noticed was that for an encrypted string length of a particular value, all URLs are either valid or invalid (for e.g. an encrypted length of 200 would always generate a valid URL and a length of 192 would always generate an invalid URL). Moreover, just looking at the final encoded string itself, one can make out if the URL is valid or not. The invalid ones seem to be truncated without one or two "=" at the end (%3D after the URL encoding at the end).

The value "gh" is one of the query string parameters in the payload after which comes the MD5 checksum. Hence it is being handled in the checksum method.

I hope it gives you further insight into the problem. I will incorporate your suggested changes and let you know. Any other ideas that you can share will be appreciated.

Thanks again!

coderiyer

coderiyera at 2007-7-14 19:38:41 > top of Java-index,Security,Cryptography...
# 8

I've fixed the bad conversion of binary to String and back I mentioned, and eliminated step (3) and the URLDecode. I'm running a test at the moment with random data. I've just got up to 228,000 successful cycles with no failures. The test will keep running to a million but it looks good so far. The only other concern is that I'm using a fixed-length initial URL as I got the impression from your OP that this was the case.

ejpa at 2007-7-14 19:38:41 > top of Java-index,Security,Cryptography...
# 9

That's great! Where did you fix the conversions between String to byte array and back (since there are several)? What did you use to fix this? Hex conversion? Base64? And did you remove both the URLEncoding/decoding that's being done in encryptPayload and decryptPayload?

And yes, the URL length is fixed though the individual parameter values may vary as I already mentioned in my earlier post.

I really really appreciate your help. Please let me know your results and also exactly how and what fixes you made. I feel good already. :-)

-coderiyer

coderiyera at 2007-7-14 19:38:41 > top of Java-index,Security,Cryptography...
# 10

Well just to prove I do post code occasionally here it is. Note that I've deleted the addition of the Sun provider, which doesn't seem to be necessary, and that I'm using DES instead of Blowfish, and probably a different Base64 class than you have, but the principles are clear. The String/binary thing I was worried about was in the decrypt method. Note that I added some stuff about "gh=" including a +3 that you will probably want to delete again. I've also fixed all the exception handling according to my lights.

/*

* GZipURLTest.java

*

* Created on 16 August 2006, 12:27

*/

import java.io.*;

import java.net.*;

import java.security.*;

import java.security.*;

import java.util.*;

import java.util.zip.*;

import javax.crypto.*;

import javax.crypto.spec.*;

import sun.misc.*;

public class GZipURLTest

{

static class BlowFishUtil

{

static

{

//Provider sunJCE = new com.sun.crypto.provider.SunJCE();

//Security.addProvider(sunJCE);

}

// encryptSecretKey - uses javax.crypto.SecretKey to encrypt a java.util.String. It returns

//this string as a byte array.

//

public static byte[] encryptSecretKey(byte[] keyBytes, byte[] unencrypted) throws GeneralSecurityException

{

SecretKeySpec skeySpec = new SecretKeySpec(keyBytes, "DES");

Cipher cipher = Cipher.getInstance("DES");

cipher.init(Cipher.ENCRYPT_MODE, skeySpec);

byte[] myEncrypt = cipher.doFinal(unencrypted);

//System.out.println("encrypt length: "+myEncrypt.length);

return myEncrypt;

}

//

// decryptSecretKey - uses a java.crypto.SecretKey to decrypt a byte array of encrypted characters

//

public static byte[] decryptSecretKey(byte[] keyBytes, byte[] encrypted) throws GeneralSecurityException

{

SecretKeySpec skeySpec = new SecretKeySpec(keyBytes, "DES");

Cipher cipher = javax.crypto.Cipher.getInstance("DES");

cipher.init(javax.crypto.Cipher.DECRYPT_MODE, skeySpec);

byte[] decrypted = cipher.doFinal(encrypted);

return decrypted;

}

}

static class Base64

{

static BASE64Encoder enc = new BASE64Encoder();

static BASE64Decoder dec = new BASE64Decoder();

static String encodeBytes(byte[] bytes)

{

return enc.encode(bytes);

}

static byte[]decodeToBytes(String string) throws IOException

{

byte[]decoded = dec.decodeBuffer(string);

return decoded;

}

static StringdecodeToString(String string) throws IOException

{

byte[]decoded = dec.decodeBuffer(string);

return new String(decoded,0,decoded.length,"UTF-8");// dubious

}

}

static BlowFishUtil bfUtil = new BlowFishUtil();

public static String encryptPayload(String payload, String authKey) throws Exception

{

String sCheckSum= getMD5Checksum(payload);

String encodeBuffer = sCheckSum + "gh="+payload;

//encodeBuffer= URLEncoder.encode(encodeBuffer);// why?

byte[] aryEncodeBuf = zipIt(encodeBuffer);

aryEncodeBuf= bfUtil.encryptSecretKey(authKey.getBytes("UTF-8"),aryEncodeBuf);

encodeBuffer= Base64.encodeBytes(aryEncodeBuf);

encodeBuffer= URLEncoder.encode(encodeBuffer);

return encodeBuffer;

}

public static String decryptPayload(String payload, String authKey) throws Exception

{

String decodeBuffer = "";

BlowFishUtil bfUtil = new BlowFishUtil();

decodeBuffer = URLDecoder.decode(payload);

//decodeBuffer = Base64.decodeToString(decodeBuffer);

//byte[] aryDecodeBuf = bfUtil.decryptSecretKey(authKey.getBytes("UTF-8"), decodeBuffer.getBytes("UTF-8"));

byte[] decodedBytes = Base64.decodeToBytes(decodeBuffer);

byte[] aryDecodeBuf = bfUtil.decryptSecretKey(authKey.getBytes("UTF-8"), decodedBytes);

aryDecodeBuf = unzipIt(aryDecodeBuf);

String sBuf = new String(aryDecodeBuf,0,aryDecodeBuf.length,"UTF-8");

//sBuf = URLDecoder.decode(sBuf);

int iPos = sBuf.indexOf("gh=");

if (iPos>=0)

{

String sURLCheckSum = sBuf.substring(0,iPos);

sBuf= sBuf.substring(iPos+3,sBuf.length());

String sCheckSum= getMD5Checksum(sBuf);

if (!sCheckSum.equals(sURLCheckSum))

{

System.out.println("checksum failure on '"+sBuf+"': "+sCheckSum+"; "+sURLCheckSum);

}

}

return new String(sBuf);

}

private static String getMD5Checksum(String mess)

throwsGeneralSecurityException,

UnsupportedEncodingException

{

String hexHashCode = "";

//String hashCode = "";

MessageDigest md = MessageDigest.getInstance("MD5");

byte[] message = mess.getBytes("UTF-8");

md.update( message );

byte[] hash = md.digest();

for ( int i=0; i < hash.length; i++ )

{

int x = hash[i] & 0xFF;

if (x < 0x10)

{

hexHashCode += "0";

}

hexHashCode += (Integer.toHexString(x));

//hashCode += hash[i]+" _ ";

}

return hexHashCode;

}

private static byte[] zipIt(String parameterString) throws IOException

{

byte[] zipped;

ByteArrayOutputStream catcher = new ByteArrayOutputStream();

GZIPOutputStream gzipOut = new GZIPOutputStream(catcher);

byte[] bytesToZip = parameterString.getBytes("UTF-8");

gzipOut.write( bytesToZip, 0, bytesToZip.length );

gzipOut.close();

return catcher.toByteArray();

}

private static byte[] unzipIt(byte[] buffer) throws IOException

{

ByteArrayOutputStream outBuffer = new ByteArrayOutputStream();

ByteArrayInputStream inBuffer = new ByteArrayInputStream(buffer);

GZIPInputStream gzip = new GZIPInputStream(inBuffer);

byte[] tmpBuffer = new byte[256];

int n;

while ((n = gzip.read(tmpBuffer)) >= 0)

{

outBuffer.write(tmpBuffer, 0, n);

}

return outBuffer.toByteArray();

}

public static voidmain(String[] args) throws Exception

{

Randomrandom = new Random();

intfailures = 0;

byte[]data = new byte[253];

Thread.currentThread().setPriority(Thread.MIN_PRIORITY);

for (int i = 0; i < 1000000; i++)

{

if ((i % 1000) == 0)

System.out.println(i);

try

{

StringurlString = "http://www.telekinesis.com.au/People/Esmond%20Pitt/";

random.nextBytes(data);

urlString += Base64.encodeBytes(data);

urlString += "#1961&arg1=1&arg2=2";

URLurl = new URL(urlString);

Stringkey = "password";

Stringencrypted = encryptPayload(urlString,key);

Stringdecrypted = decryptPayload(encrypted,key);

//System.out.println(urlString+": decrypted to: "+decrypted);

//System.out.println(new URL(decrypted));

Thread.sleep(2);

}

catch (Exception exc)

{

exc.printStackTrace();

failures++;

}

}

System.out.println("1000000 trials "+failures+" failures");

}

}

Some hours later: it's up to 800,000 iterations and no failures.

Message was edited by:

ejp

Later still: 1,000,000 trials, zero failures.

Message was edited by:

ejp

ejpa at 2007-7-14 19:38:41 > top of Java-index,Security,Cryptography...
# 11

Thanks a lot for posting the code and for taking the trouble to go through a million iterations. I incorporated all the changes suggested by you and I am generating six URLs at a time. When I keep running the program again and again, at some point I encounter some invalid URLs. And, just by looking at the encrypted URL, I can figure out if it is a valid one or not. I am stumped by this right now. Is there an email address that I can contact you at to post some of my results? By the way, does the key size make a difference? It is a 56 byte fixed key.

The length of the output that comes from the zipIt method seems to have an impact on the encyrpted URL. If the original length of the cleartext being fed to the zipIt program is 193, sometimes the output from the zipIt method seems to have a length of 167 which becomes 168 after encryption (all URLs generated from zipped values of this length fail). If the zipped length is 169 (which becomes 176 after encryption), all generated URLs seem to be valid.

Thanks for all your help.

- coderiyer

coderiyera at 2007-7-14 19:38:41 > top of Java-index,Security,Cryptography...
# 12
I've put enough time into this already, but personally I would get rid of the GZIP step altogether. It takes time; adds no security; from what you've just said it's not even saving anything worthwhile in space; and it is causing a problem.
ejpa at 2007-7-14 19:38:41 > top of Java-index,Security,Cryptography...
# 13
Later: I tried with length of 193 but can't reproduce your problem over 100,000 iterations - I can't even get the zipped length of 167.I can't use 56-byte keys for some reason, DES will only let me use 8.
ejpa at 2007-7-14 19:38:41 > top of Java-index,Security,Cryptography...
# 14
> I can't use 56-byte keys for some reason, DES will> only let me use 8.A DES key is 56 bits normally packaged with 7 bits in each of 8 bytes with the lsb of each byte a parity bit. SunJCE ignores the parity bit.
sabre150a at 2007-7-14 19:38:41 > top of Java-index,Security,Cryptography...
# 15
Thanks.
ejpa at 2007-7-21 9:56:19 > top of Java-index,Security,Cryptography...
# 16

Hi ejp!

I really appreciate your help.

I removed the zip/unzip from the encryptPayload/decryptPayload methods. I am not able to access the page using the encrypted URL after removing the zipping method. Could you briefly outline the steps needed to generate a valid URL both on the encryption/decryption programs?

Thanks again for all your help.

coderiyer

coderiyera at 2007-7-21 9:56:19 > top of Java-index,Security,Cryptography...