I want to be able to read just few files at a time in a given directory. File Class provides .list() but it gets all the files or if I use FileNameFilter then based on file name - still it will try to read all the file names. But what I want is to read few files. We have 3 million files in a directory, how can I read 100 or say 1000 files at a time. File.list() is going to try to load all the file names at once.
I suspect that you're out of luck. Java uses native file methods to get directory informtation, and I'm not aware of any native directory list methods that will do what you want.
couldn't you try getting a list of files that match some search criteria? like a find method maybe? then you can get all the files that start with A, then B or something like that, i dunno, probably a bad idea anyway
and i concur, 3 million files in 1 directory is not the best idea
> couldn't you try getting a list of files that match some search criteria? like a find method maybe?
How to do that without reading the entire directory? :)
Exactly, how to do it without having to read entire directory ?
I do agree these files need to go to subdirectories, but currently that's how it is ..I am thinking of writing a script that will drop few files at a time to a directory and then java program will read that directory. That's most efficient way I can think of.
> How to do that without reading the entire directory?
JVM doesn't have to load the entire list to do this. It could be implemented using native OS calls. I don't know if they exist or not or if JVM has bindings to them, etc. But I was hoping there was something there. I guess not.
> > How to do that without reading the entire
> directory?
>
> JVM doesn't have to load the entire list to do this.
> It could be implemented using native OS calls. I
> don't know if they exist or not or if JVM has
> bindings to them, etc. But I was hoping there was
> something there. I guess not.
"...getting a list of files that match some search criteria?" (as you said) requires something (Java or an underlying native OS) to acquire the entire directory listing and then select from the listing, in some manner. And this is what the OP didn't want to do.
> Exactly, how to do it without having to read entire
> directory ?
>
> I do agree these files need to go to subdirectories,
> but currently that's how it is ..I am thinking of
> writing a script that will drop few files at a time
> to a directory and then java program will read that
> directory. That's most efficient way I can think of.
The best way is to separate them, it will make things much easier.
Scrapped - See above post.
> "...getting a list of files that match some search
> criteria?" (as you said) requires something (Java or
> an underlying native OS) to acquire the entire
> directory listing and then select from the listing,
> in some manner. And this is what the OP didn't want
> to do.
exactly, as i said, it's something the OS could do but i wasn't sure if it did or not. All I'm saying is that until you're sure the OS (whichever OP is using) doesn't support this, then we cannot be sure there isn't a solution. I never said the JVM would/should read the entire list.
My experiments show that on my machine, the process takes
about 20 seconds and the String[] takes typically 200-300M, depending of course on how long the file names are.
Is this too much? All depends on your application:
interactive or not, is the process repeated often or not etc.
