How to make this faster? read millions of record from txt file
Hi there,
I got an issue. There is a txt file contains 2 million records, I also got another file contains over 10000 numbers. Now, I need to compare this 10000 numbers with that 2 million records if any records contains a number which belongs to 10000 number set, i retrieve this record and keep it. later on, when i finish the comparison i'll write all the result records into a txt file.
What kind of data structure shall i use to keep the records and numbers? how to make the comparison quicker? Any idea will do!
Thanks!
[552 byte] By [
JbloodHa] at [2007-11-26 22:29:26]
# 2
i am using arraylist to restore the 10000 numbers.
the problem is that , it takes too long time to make the comparision.
obviously i cannt restore the 2million records into some data structure, so what i am doing is to read each record use bufferedReader and compare it with 10000 numbers use a for loop. something like this:
// @param numberArrayList = 10000 numbers
int j=10000; // i just start with reading 10000 records from 2 million
while (j-- > 0)
{
if ((line = file.readLine()) == null)
break;
else
{
for (int i = 0; i < nullArrayList.size(); i++)
{
if ((line.contains((String) numberArrayList.get(i)))&&(line.substring(10,15).equals("SOMETHING"))) {
masterInputMap.put(nullArrayList.get(i),line);break;
}
}
}
}
which takes 20 sec to finish the comparison to this 10000 records. since there are over 2 million records... it will be a nightmare....help~
