Unix 'find' and Solaris 10
I am trying to redirect output from a 'find' invocation in Solaris 10. This find command:
find . -type d -name "02" -print
Works great when run without redirection. When I attempt to do a simple redirect, like this:
find . -type d -name "02" -print > /tmp/stuff.out
Nothing ends up in the "/tmp/stuff.out" file. When I run this under CentOS v4.x, it works exactly as expected (the find output ends up in the /tmp/stuff.out file).
What am I doing wrong?
# 1
Hmm, just pasted your commands into a Solaris 10 system and the redirect worked:
bash-3.00$ cd /tmp
bash-3.00$ mkdir test
bash-3.00$ mkdir test/02
bash-3.00$ find . -type d -name "02" -print
./test/02
bash-3.00$ find . -type d -name "02" -print > /tmp/stuff.out
bash-3.00$ cat /tmp/stuff.out
./test/02
bash-3.00$ uname -a
SunOS blade2000 5.10 Generic_118833-33 sun4u sparc SUNW,Sun-Blade-1000
Can you post the output with and without the redirect?
# 2
Thanks for your quick reply. I neglected to mention in my original post that I am running this as root, just to be sure there are no permission problems anywhere.
When I run the command without the redirection, this is the output:
# cd /backup/current
# find . -type d -name "02" -print
./angelfish/200702/02
^C#
When I run it with redirection, nothing ends up in the file, even after 75 minutes of waiting so there isn't really anything to see. (When I run it without redirection, I get output within 60 seconds.) But here is a run with redirection:
# find . -type d -name "02" -print > /tmp/stuff.out
^C# more /tmp/stuff.out
#
FYI the directory levels go several levels below the "02" level. For example:
# find . -print
.
./angelfish
./angelfish/200702
./angelfish/200702/02
./angelfish/200702/02/etc
./angelfish/200702/02/etc/.pwd.lock
./angelfish/200702/02/etc/CORBA
./angelfish/200702/02/etc/CORBA/servers
Thanks for your help!
# 3
If it doesn't return, then the command is hanging..where?Use 'pwdx <PID>' on the find command.You might also want to truss it and see what it's trying to do.Is anything wrong with /tmp?-- Darren
# 4
Thanks for your response.
Thanks for the pointer to "pwdx". When I run "pwdx" with the find process, it presents a different directory with each invocation, so the "find" process seems to be going through the directories just fine and not hanging.
Running a truss shows the same result as pwdx. It doesn't seem to be caught in a loop, and it appears to be stepping over each file in the directory just like I'd expect.
Here are the perms on /tmp:
drwxrwxrwt7 rootsys 1044 Jul 6 03:30 .
Looks OK to me.
# 5
Oh.. I thought you were saying that the first one *finished* and the second didn't. But you were just saying that you see the intermediate results on the first run and not the second.
What you are seeing is very likely I/O buffering. Many programs (find apparently one of them) detect whether their output is going to a terminal or something else.
When the output goes to a terminal, it's flushed after each line so that you can see the data quickly. When no terminal is detected (output to a file), then it is buffered for efficiency. So 'find' has almost certainly already printed the data, but the I/O buffer has not emptied. When the buffer fills or the output is closed, the buffer will flush.
'grep' does the same thing. Compare the output of these two lines.
# perl -le '$|=1; print "Hello" ; sleep 10' | grep H
Hello
# perl -le '$|=1; print "Hello" ; sleep 10' | grep H | cat -
Hello
The first one prints immedately then exits 10 seconds later. The second one doesn't print until the 10 seconds are up. This is due to 'grep' setting the output to autoflush when the output is detected to be a terminal, but not the second when the output is a pipe.
--
Darren
# 6
how close is this to your situation?
bash-3.00$ ls -lR angelfish
angelfish:
total 8
drwxr-xr-x3 alanother104 Jul 6 11:01 200702
angelfish/200702:
total 8
drwxr-xr-x3 alanother105 Jul 6 11:01 02
angelfish/200702/02:
total 8
drwxr-xr-x3 alanother149 Jul 6 11:01 etc
angelfish/200702/02/etc:
total 8
drwxr-xr-x2 alanother109 Jul 6 11:01 CORBA
angelfish/200702/02/etc/CORBA:
total 0
-rw-r--r--1 alanother 0 Jul 6 11:01 servers
find . -type d -name "02" -print
./angelfish/200702/02
find . -type d -name "02" -print > /tmp/stuff.out
bash-3.00$ cat /tmp/stuff.out
./angelfish/200702/02
which find
/usr/bin/find
bash-3.00$ ls -l /usr/bin/find
-r-xr-xr-x1 rootbin19428 Nov 9 2006 /usr/bin/find
alan
# 7
This looks pretty close to my setup. The only thing I'd say is that there are thousands of files/directories under the main directory that the find is running in, in my case.
I'm glad this works for you, but unfortunately, it doesn't help me in getting find to work for me.
> how close is this to your situation?
>
> bash-3.00$ ls -lR angelfish
> angelfish:
> total 8
> drwxr-xr-x3 alanother104 Jul 6 11:01
> 200702
>
> angelfish/200702:
> total 8
> drwxr-xr-x3 alanother105 Jul 6 11:01
> 02
>
> angelfish/200702/02:
> total 8
> drwxr-xr-x3 alanother149 Jul 6 11:01
> etc
>
> angelfish/200702/02/etc:
> total 8
> drwxr-xr-x2 alanother109 Jul 6 11:01
> CORBA
>
> angelfish/200702/02/etc/CORBA:
> total 0
> -rw-r--r--1 alanother 0 Jul 6 11:01
> servers
>
>
> find . -type d -name "02" -print
> /angelfish/200702/02
>
>
> find . -type d -name "02" -print > /tmp/stuff.out
> bash-3.00$ cat /tmp/stuff.out
> ./angelfish/200702/02
>
>
> which find
>
> /usr/bin/find
>
> bash-3.00$ ls -l /usr/bin/find
> -r-xr-xr-x1 rootbin19428 Nov 9 2006
> /usr/bin/find
>
> alan
# 8
I'd thought of the buffering point you bring up, but wouldn't the <control-c> flush the buffers and close all open files?
Does anyone have any suggestions on how to flush the buffers to force the output to the file? I don't care that it takes hours to run. I guess I can set up a test find to run over the weekend and see what the output is next week.
> Oh.. I thought you were saying that the first one
> *finished* and the second didn't. But you were just
> saying that you see the intermediate results on the
> first run and not the second.
>
> What you are seeing is very likely I/O buffering.
> Many programs (find apparently one of them) detect
> whether their output is going to a terminal or
> something else.
>
> When the output goes to a terminal, it's flushed
> after each line so that you can see the data quickly.
> When no terminal is detected (output to a file),
> then it is buffered for efficiency. So 'find' has
> almost certainly already printed the data, but the
> I/O buffer has not emptied. When the buffer fills
> or the output is closed, the buffer will flush.
>
> 'grep' does the same thing. Compare the output of
> these two lines.
> # perl -le '$|=1; print "Hello" ; sleep 10' | grep H
> Hello
> # perl -le '$|=1; print "Hello" ; sleep 10' | grep H
> | cat -
> Hello
>
> The first one prints immedately then exits 10 seconds
> later. The second one doesn't print until the 10
> seconds are up. This is due to 'grep' setting the
> output to autoflush when the output is detected to be
> a terminal, but not the second when the output is a
> pipe.
>
> --
> Darren
# 9
> > which find> > > > /usr/bin/find> > > > bash-3.00$ ls -l /usr/bin/find> > -r-xr-xr-x1 rootbin19428 Nov 9> 2006> /usr/bin/findWhat are your values for find posted above?alan
# 10
Sorry I neglected to answer your question. I have tried two versions of find, both with the same result:
# sum /usr/bin/find
53974 55 /usr/bin/find
# ls -al /usr/bin/find
-r-xr-xr-x1 rootbin28136 Apr 28 2006 /usr/bin/find
And I also installed the sunfreeware.com version of GNU find:
# sum /usr/local/bin/find
64850 928 /usr/local/bin/find
# ls -al /usr/local/bin/find
-rwxr-xr-x1 binbin474964 Mar 4 05:40 /usr/local/bin/find
In addition, I tried both /bin/bash as well as /bin/sh. All have the same result.
All of this leads me to believe that the issue is something deep in the Solaris 10.
> > > which find
> > >
> > > /usr/bin/find
> > >
> > > bash-3.00$ ls -l /usr/bin/find
> > > -r-xr-xr-x1 rootbin19428 Nov 9
> > 2006
> > /usr/bin/find
>
>
> What are your values for find posted above?
>
> alan
# 11
> # sum /usr/bin/find> 53974 55 /usr/bin/findmy sum is different.Try the md5 output on the following site which includes md5 binaries: http://sunsolve.sun.com/pub-cgi/fileFingerprints.plalan
# 12
> I'd thought of the buffering point you bring up, but
> wouldn't the <control-c> flush the buffers and close
> all open files?
You can try that example..
# perl -le '$|=1; print "Hello" ; sleep 10' | grep H | cat -
[ wait 10 seconds....]
Hello
# perl -le '$|=1; print "Hello" ; sleep 10' | grep H | cat -
^C
#
Nope. No output.
> Does anyone have any suggestions on how to flush the
> buffers to force the output to the file? I don't care
> that it takes hours to run. I guess I can set up a
> test find to run over the weekend and see what the
> output is next week.
That's up to the program. You don't have any real way to control it externally.
If you can't wait for it to complete, one alternative is to lie to it. The 'expect' tool pretends to be a terminal to programs to allow automation of interactive tools. It would be pretty easy to connect the output of 'find' into 'expect' so that find did not enable its buffering. Then 'expect' could just spit the data out for you.
Another alternative is to expand the things that it finds so that it actually fills the buffer (causing a flush) in a reasonable amount of time. Or use -prune and search smaller areas so that it finishes more quickly.
--
Darren
# 13
What happens if you try this:
bash-3.00$ cd /tmp
bash-3.00$ touch a
bash-3.00$ echo a >> /tmp/a
bash-3.00$ cat /tmp/a
a
or > from any other command?
From your posts it finally dawned on me that find isn't the problem, its as you stated originally, the redirection is what isn't working.
alan
# 14
> I'd thought of the buffering point you bring up, but
> wouldn't the <control-c> flush the buffers and close
> all open files?
Regardless of the suggestion, I am unable to replicate this with 'find' on Solaris 10 (or older versions). While 'grep' is subject to this buffering, '/usr/bin/find' does not appear to be.
I've got no problems doing small outputs and seeing them immediately.
One easy way to verify this is the following:
find . -ls -exec sleep 1 \;
(should see one line a second).
find . -ls -exec sleep 1 \; > /tmp/file &
tail -f /tmp/file
(should see one line a second)
Can you replicate this? Because this prints out every file, you should if it's working or not immediately.
--
Darren
# 15
OK, so when I run either the non-redirect find and redirect find code snippet you suggested, they both output one line per second. *sigh* This is done in the same top level directory I am trying to run my "production" find in, so the # of files is same for both of my runs (your redirect test and my production).
Now, I'm totally confused.
> Regardless of the suggestion, I am unable to
> replicate this with 'find' on Solaris 10 (or older
> versions). While 'grep' is subject to this
> buffering, '/usr/bin/find' does not appear to be.
>
> I've got no problems doing small outputs and seeing
> them immediately.
>
> One easy way to verify this is the following:
>
> find . -ls -exec sleep 1 \;
> (should see one line a second).
> find . -ls -exec sleep 1 \; > /tmp/file &
> tail -f /tmp/file
> (should see one line a second)
>
> Can you replicate this? Because this prints out
> every file, you should if it's working or not
> immediately.
>
> --
> Darren
# 16
How about if you truss it and ask what it's writing?truss -t write -wall -find . -type d -name "02" -print > /tmp/stuff.outYou should get output on screen with all the lines written. Are you getting 'truss' lines but nothing in /tmp/stuff.out?--
# 17
So, I figured out what I was doing wrong. It seems that the entire find must complete before *any* output is sent to the redirect file. I did a test run, and exactly when the command finished (about 3 hours after starting it), the output appeared in the file. I was impatient before, stopping the command after about 1 hour in previous run attempts.
> How about if you truss it and ask what it's writing?
>
> truss -t write -wall -find . -type d -name "02"
> -print > /tmp/stuff.out
>
> You should get output on screen with all the lines
> written.
>
> Are you getting 'truss' lines but nothing in
> /tmp/stuff.out?
>
> --
> Darren
# 18
That seems odd. While I expect programs to do that, I'm unable to replicate that with the solaris 'find' program, and the one with the one-second delay seemed to print out for you immediately.
I have no answer for why this particular search doesn't print while the others do.
--
Darren
