Home.... | log in.... | Pub my question! | Sign-up....

Loading a lot of small files

I have a program that needs to load a lot of small files quickly. It needs to load about 5000 files from a set of nearly a million files that are each about 5kB as fast as it can. Right now I am using the simplest method and just loading each file one after another using the FileReader class. I would like to know what the best method is for reading many small files quickly. Should I use threads to load files in parallel? Are there any 3rd party libraries that will work faster?

[488 byte] By [tbpckisaa] at [2007-11-26 14:51:42]

«« executing JAR file ignores classpath
»» Fetching international characters from Clob

# 1

Usually very large directories suffer from bad access times. If you have a single directory with nearly a million files, you will discover that the time for opening the file is much greater than the time for reading it.

If you have a directory tree containing the files (say that you have a directory, with 100 subdirectories, each one containing 100 subdirectories, each one containing 100 files) you can open the files much faster.

edsonwa at 2007-7-8 8:39:55 >

# 2

I do have a setup with subdirectories setup very similar to the example you gave.

tbpckisaa at 2007-7-8 8:39:55 >

# 3

You might find that BufferedFilReader is of value. See my reply #4 here: http://forum.java.sun.com/thread.jspa?threadID=528654&messageID=2540312

ChuckBinga at 2007-7-8 8:39:55 >

Core

Core APIs

Java programming resources: FAQs, tutorials, compiler and browser download sites, documentation, books lists, IDEs, etc.

Loading a lot of small files

Core New Topic

Core Hot Topic

Core

All Classified