bad performance of File.renameTo()
i use Fil.renameTo()methode to move 100k files from a folder to another.Both the folder are in the same mount.
Os:solaris 10 JDK:1.4.2
When have 20 concurreny users , the performance is bad.
this is the time used to move:(unit ms)
07/24 14:02:01 596===
07/07/24 14:02:01 3423===
07/07/24 14:02:02 3567===
07/07/24 14:02:02 3851===
07/07/24 14:02:02 3256===
07/07/24 14:02:02 3646===
07/07/24 14:02:02 3048===
07/07/24 14:02:02 5198===
07/07/24 14:02:03 5374===
07/07/24 14:02:03 5574===
07/07/24 14:02:04 6786===
07/07/24 14:02:04 4651===
07/07/24 14:02:04 4201===
07/07/24 14:02:04 3845===
07/07/24 14:02:04 478===
The rename action just changes the path of the file , not move the data actually, but why so slow.
Thx for your attention
Are you saying that it takes about 3-7 seconds to rename one file? Are you sure that they are on the same physical disk and filesystem/partition?
Disks can be really slow if many users/processes are trying to access them at the same time.
kajbja at 2007-7-29 16:33:43 >
100,000 files is too many in a single Unix folder, by a couple of orders of magnitude. Rethink your problem.
ejpa at 2007-7-29 16:33:43 >
yes , iam sure the folders are in the same disk
sorry for the ambiguity.
100k is the file size.
ths for your reply
hi kajbj.
I think the 20 users maybe not many.
> hi kajbj.
> I think the 20 users maybe not many.
What happens if you only have one user? What happens if you use larger files? (files that are about 10-20 MB)
Sounds like the program actually copies/moves the files instead of just renames them.
kajbja at 2007-7-29 16:33:43 >
here is the rename code in JDk.
public boolean rename(File f1, File f2) {
// Keep canonicalization caches in sync after file deletion
// and renaming operations. Could be more clever than this
// (i.e., only remove/update affected entries) but probably
// not worth it since these entries expire after 30 seconds
// anyway.
cache.clear();
javaHomePrefixCache.clear();
return rename0(f1, f2);
}
does the cache clear() use much time?
Impossible to say for sure without knowing the datatypes, or what rename0() actually does, but clearing most JDK collections is generally instantaneous.
ejpa at 2007-7-29 16:33:43 >
if 1user, it will be :
07/07/24 15:11:53 166===
07/07/24 15:11:54 155===
07/07/24 15:11:54 99===
07/07/24 15:11:55 169===
07/07/24 15:11:55 119===
07/07/24 15:11:56 172===
07/07/24 15:11:56 85===
07/07/24 15:11:57 128===
07/07/24 15:11:57 107===
07/07/24 15:11:58 199===
07/07/24 15:11:58 166===
07/07/24 15:11:59 127===
07/07/24 15:11:59 86===
07/07/24 15:12:00 155===
07/07/24 15:12:00 141===
07/07/24 15:12:01 236===
> es the cache clear() use much time?
Probably not. Try with larger files just as I said. The files are copied/moved instead of renamed if the execution times increases.
Kaj
kajbja at 2007-7-29 16:33:43 >
this is the code in JDK calss .UnixFileSystem
here is the cache/ Caches for canonicalization results to improve startup performance.
// The first cache handles repeated canonicalizations of the same path
// name. The prefix cache handles repeated canonicalizations within the
// same directory, and must not create results differing from the true
// canonicalization algorithm in canonicalize_md.c. For this reason the
// prefix cache is conservative and is not used for complex path names.
private ExpiringCache cache = new ExpiringCache();
// On Unix symlinks can jump anywhere in the file system, so we only
// treat prefixes in java.home as trusted and cacheable in the
// canonicalization algorithm
private ExpiringCache javaHomePrefixCache = new ExpiringCache();
>>Probably not. Try with larger files just as I said. The files are copied/moved instead of renamed if the execution times increases.
try 12M file,time:
07/07/24 15:20:07 230===
07/07/24 15:20:12 141===
07/07/24 15:20:17 91===
07/07/24 15:20:21 94===
07/07/24 15:20:26 186===
> >>Probably not. Try with larger files just as I said.
> The files are copied/moved instead of renamed if the
> execution times increases.
>
> try 12M file,time:
> 07/07/24 15:20:07 230===
> 07/07/24 15:20:12 141===
> 07/07/24 15:20:17 91===
> 07/07/24 15:20:21 94===
> 07/07/24 15:20:26 186===
With 20 users?
kajbja at 2007-7-29 16:33:43 >
Here is my code:
long start=System.currentTimeMillis();
if (!file.renameTo(tempFile))
throw new DefinedException(CommonConstant.MSG_EXCEPTION,CommonConstant.DES_EXCEPTION,"Error when rename file from /in to /temp.");
long end=System.currentTimeMillis();
System.out.println(end-start+"===");
Execute it with 20 users and 12MB files.
kajbja at 2007-7-29 16:33:51 >
> 1user
Yes, but execute it with 20 users and 12 MB files, and post the result.
kajbja at 2007-7-29 16:33:51 >
>>>.Execute it with 20 users and 12MB files
here is
07/07/24 15:44:00 12942===
07/07/24 15:44:00 11220===
07/07/24 15:44:00 11249===
07/07/24 15:44:00 10415===
07/07/24 15:44:00 12956===
07/07/24 15:44:00 12960===
07/07/24 15:44:00 11326===
07/07/24 15:44:00 73===
07/07/24 15:44:00 260===
07/07/24 15:44:00 445===
07/07/24 15:44:00 521===
07/07/24 15:44:01 570===
07/07/24 15:44:01 713===
07/07/24 15:44:02 2053===
07/07/24 15:44:02 2058===
07/07/24 15:44:02 2059===
07/07/24 15:44:02 2053===
07/07/24 15:44:02 2061===
07/07/24 15:44:02 2075===
07/07/24 15:44:02 180===
So it gets worse? That means that the files are copied.
kajbja at 2007-7-29 16:33:51 >
really?
But i use the renameTo() methode in JDK.
> really?
> But i use the renameTo() methode in JDK.
I still think that you actually are trying to rename between to different file systems or partitions. Everything points in that direction.
kajbja at 2007-7-29 16:33:51 >
> > really?
> > But i use the renameTo() methode in JDK.
>
> I still think that you actually are trying to rename
> between to different file systems or partitions.
> Everything points in that direction.
In which case, the OP could implement a fire-and-forget renaming process that did the renaming/moving work and then the OP's code would just add the rename task to the fire-and-forget queue.
>>In which case, the OP could implement a fire-and-forget renaming process that did the renaming/moving work and then the OP's code would just add the rename task to the fire-and-forget queue.
could u give a more detail explanation of this?
thx
>>In which case, the OP could implement a fire-and-forget renaming process that did the renaming/moving work and then the OP's code would just add the rename task to the fire-and-forget queue.
could u tell me what is the OP means?
You. OP means Original Post or Original Poster (dependent on context).
