Hi,
I'm working with very large datasets and would like to be able to process arrays with more than 2^31-1 elements. This limit seems to come about because Java requires a signed integer to be passed as a parameter to an array declaration, e.g.:
Object bytes =newbyte[2147483647];// works
Object bytes =newbyte[2147483648];// fails
Is there *any* way to get around this?
Thanks very much,
- Davi
Only thing you can do is re-think the data structures. There's no way around the fact that arrays are indexed by integers, and that integers have a MAX_VALUE. Is there any reason you need a single array that size, or could it be broken into several arrays? That might make more sense anyway, regardless of this limitiation
Arrays are indexed with an int, meaning 2^31-1 is the highest size possible. You wouldn't want to keep that much data in memory anyway, most computers don't have 4 gigs or ram.
As a side note, if you want a byte array, then declare it as a byte array:
byte[] bytes = new byte[2147483647]; // works
http://java.sun.com/docs/books/tutorial/java/nutsandbolts/arrays.html
Thanks very much for replies so far.
By way of some context, these data are single multigigabyte images. I'm processing them on a machine running a 64-bit JVM, which has plenty of physical RAM (16GB+). I'm not worried about deploying the code I write to smaller machines. I just need something that will let me work with and process these images conveniently.
I appreciate any further suggestions people might have.
- Davi
You're not trying to store all 2 billion images in ram at once right? Cause 2 billion 2 billion-byte images is... (carry the one...) a whole lot more than 16 gigs.
Edit:
Oh, wait, you mean the entire array is a single image? Nevermind then.
Message was edited by:
hunter9000
In fact, you might find better performance by using multidimensional arrays.
new byte [100][1000000]
will give you 100 arrays of a million bytes each. If you can restrict manipulation to one array at a time, it should be faster than fewer large arrays.
Remember to make the last dimension the largest to minimize the number of arrays created.
> > Is there *any* way to get around this?
>
> What about creating a class that aggregates several
> arrays and implements the List API?
>
> Kaj
Unfortunately, many of the methods in the List interface depend on int indexes also. You would probably want these methods to use a long index instead.
> > > Is there *any* way to get around this?
> >
> > What about creating a class that aggregates
> several
> > arrays and implements the List API?
> >
> > Kaj
>
> Unfortunately, many of the methods in the List
> interface depend on int indexes also. You would
> probably want these methods to use a long index
> instead.
True. I should have said that he should mimic the behaviour of the List interface, but not implement it.
Kaj
