Sun Studio 12 doesn't do better than gcc
First, version info:
luoyi@4Email:~/qp$ gcc --version
gcc (GCC) 4.1.2
Copyright (C) 2006 Free Software Foundation, Inc.
This is free software; see the source for copying conditions. There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
luoyi@4Email:~/qp$ suncc -V
cc: Sun C 5.9 Linux_i386 2007/05/03
usage: cc [ options] files. Use 'cc -flags' for details
and I compile the program with flag:
suncc: -fast -native -xipo
gcc: -O2 -march=i686
the cpu:
luoyi@4Email:~/qp$ cat /proc/cpuinfo
processor: 0
vendor_id: GenuineIntel
cpu family: 15
model: 1
model name: Intel(R) Celeron(R) CPU 1.70GHz
stepping: 3
cpu MHz : 1704.001
cache size: 128 KB
fdiv_bug: no
hlt_bug : no
f00f_bug: no
coma_bug: no
fpu : yes
fpu_exception: yes
cpuid level: 2
wp : yes
flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm up
bogomips: 3411.04
the result:
this is suncc's result
luoyi@4Email:~/qp$ time lib/bmwdict -i data/alitao.config -o data/alitao.idx
data/taobao.dict902
data/alibaba.dict1002
Token = 450382Word = 400251OR_TOKEN = 0AP_TOKEN = 0
real0m37.274s
user0m36.082s
sys0m1.196s
and this is the gcc result
luoyi@4Email:~/qp$ time gcclib/bmwdict -i data/alitao.config -o data/alitao.idx
data/taobao.dict902
data/alibaba.dict1002
Token = 450382Word = 400251OR_TOKEN = 0AP_TOKEN = 0
real0m35.012s
user0m34.190s
sys0m0.824s
and we all know gcc 4.1.x is not as fast as gcc 3.4.x !
[1771 byte] By [
luoyi82a] at [2007-11-27 7:52:54]
# 1
Hi
What is this test supposed to show? It's just one app, on one architecture. If this is the only app that you are developing, then it looks like you'll be better sticking with GCC. If you want to do a proper compiler benchmark, then you need to test a variety of different application types, and also more than one architecture (in particular, x86 and amd64, since it doesn't look like you're using Solaris in any form).
Paul
# 2
sorry, I can't agree with you that . we can look to http://blogs.sun.com/tatkar/, vijay said "SunStudio 12 Compiler establishes World Record on Woodcrest chip", if you are right, then I can also say SPEC is just also a single app and Woodcrest is just a single arch.
I don't have done any popular benchmark with compilers, because we all know compiler vendors often do some special tuning to their products so they can get better result.
I've compare PGI, SunCC, GCC, Intel C++, PathCC with more than 10 app I myself write on 3 machines(C1.7, P2.8, AMD x86_64 3600+). and all results is just the same, I don't think this is coincidence.
PGI < SunCC < GCC < Intel C++ <= PathCC
and finally, this fourm's topic is " Sun Studio for Linux" , I don't know why
you say "since it doesn't look like you're using Solaris in any form".
# 3
> sorry, I can't agree with you that . we can look to
> http://blogs.sun.com/tatkar/, vijay said "SunStudio
> 12 Compiler establishes World Record on Woodcrest
> chip", if you are right, then I can also say SPEC is
> just also a single app and Woodcrest is just a
> single arch.
SPEC is a suite of tests; no idea how many make up SPECint2006. I agree with you that Woodcrest is just a single arch; more architectures would be more representative.
From the blog that you quote:
"New Sun Systems with SunStudio compilers and Solaris 10 on Intel chips are leading the way with performance."
There is no mention of Linux there.
> I don't have done any popular benchmark with
> compilers, because we all know compiler vendors often
> do some special tuning to their products so they can
> get better result.
I'll let someone from Sun reply to that.
> I've compare PGI, SunCC, GCC, Intel C++, PathCC with
> more than 10 app I myself write on 3 machines(C1.7,
> P2.8, AMD x86_64 3600+). and all results is just the
> same, I don't think this is coincidence.
>
> PGI < SunCC < GCC < Intel C++ <= PathCC
And what sort of code do you write? The claim in the blog that you mention refers to SPECin2006. If your code does not perform intensive integer operations, then that might explain why, with your apps, Sun Studio does not perform so well.
> and finally, this fourm's topic is " Sun Studio for
> Linux" , I don't know why
> you say "since it doesn't look like you're using
> Solaris in any form".
Because it is also available for Solaris/x86 and Solaris/SPARC (as is GCC).
Paul
# 4
> sorry, I can't agree with you that . we can look to
> http://blogs.sun.com/tatkar/, vijay said "SunStudio
> 12 Compiler establishes World Record on Woodcrest
> chip", if you are right, then I can also say SPEC is
> just also a single app and Woodcrest is just a
> single arch.
The benchmark you mention is a benchmark of a whole system, it does not say that replacing suncc with gcc would not improve the performance.
> I've compare PGI, SunCC, GCC, Intel C++, PathCC with
> more than 10 app I myself write on 3 machines(C1.7,
> P2.8, AMD x86_64 3600+). and all results is just the
> same, I don't think this is coincidence.
>
> PGI < SunCC < GCC < Intel C++ <= PathCC
Other people have made similar remarks, and I expect the Studio developers are aware of it. What I know for sure is that saying there exists an application where gcc is better does not help. Providing the source code would already be much better. Making the testcase as small as possible would help even more. But then I don't know if the developers are looking for testcases that are not well optimized or if they already have enough for now.
# 5
> "New Sun Systems with SunStudio compilers and Solaris 10 on Intel chips are leading the way with performance."
> There is no mention of Linux there.
Thats intentional, because we can not really claim that our compilers master Linux in terms of performance yet. Linux is so much tied to gcc, and our compiler (unlike, say, intel's or pathscale's) does not claim itself being gcc. It is really hard for us to ditch last drop of performance from the Linux libc/libm etc... Solaris is much more honest to the compilers (e.g. with system math libraries being optimized, not compiler-specific parts of math headers like on Linux).
We will definitely improve there in future releases, but for this one compatibility was much more the goal for us than uber-performance.
> I don't have done any popular benchmark with compilers, because we all know compiler vendors often
> do some special tuning to their products so they can get better result.
Thats true for *every* single compiler vendor out there. But SPEC rules are especially hard on "cheating" and all the submitted results from Sun, Intel and other vendors are seriously checked against those rules.
Surely still every compiler is tuned against performance on those benchmarks, but if you take a look at these benchmarks you will see that these are all real applications with a great diversity of algorithms and programming aproaches there. Unless you are doing something completely different you really can benefit from paying attention to these spec numbers. After all, even *measuring* performance might be nontrivial.
Btw, there is one small trick out there - for the ordinary usage you might want to pay attention to *base* SPEC numbers (and base compiler options), not *peak* ones, because its with *base* where compiler has absolutelly minimal chance to "cheat" you. And compiler/system vendors usually brag with *peak* numbers only.
Hope it was helpful to you.
regards,
__Fedor.
SFVa at 2007-7-12 19:34:08 >
# 6
Luoyi,
Since you quote me in your posting, I hope you wont mind hearing from me directly on this thread.
Note, first of all, if you've my blogs often enough that I have called out whenever our compilers have helped Sun Systems to establish new World Records of Performance. From my previous counting of this, we've done it about 35+times in the past two years.
I have reported on such things as:
World Record SPECfp performance, particularly with Opteron chips. At one stage, we even claimed to beat Power5, the highest performing SPECfp chip of the past few years. That didnt last long (esp. with 4.7GHz Power6 on the market now), but it was a fun record to hold while we could!
World Record on SPEC OMP, especially, the medium-size problem set (since our x86/x64 machines typically only went upto 4-8 CPUs) but now with the large-size problem set as well. As far back as I remember, Sun has excelled in SPEC OMP, even on SPARC. This speaks volume for how well we've tuned for scalability and parallelism support, over the years.
Beating GCC, Intel and others on STREAM by a huge margin (both GCC and Intel compilers by 2x or more) on the x64 boxes.
Beating GCC, on most counts on nbench
There are several others I can point out where this is more than true and GCC isnt even close. But thats not the point.
The point I'm trying to make is that: SunStudio on x86/x64 had a bad rap once upon a time. We worked hard to fix that. We now do very well.
However,
this shouldnt diminish the performance work that other compilers are doing.Intel compilers are VERY good in their own way (my post about beating Intel compilers on their very own Woodcrest chip is simply a message that we do pretty well there as well; dont just assume that because its an Intel chip, Intel Compilers are the only ones that can perform well there). So is GCC. Each compiler has its own "target market" where it does better than other compilers. Even PGI, which trails everyone else in your list, is actually better than others for some kinds of applications, particularly in HPC space.
We should be happy that we have so many competent compilers in this (x86/x64) space. I am not into bashing the competition; its more important to see if our strengths and focus match what you, as a developer, are looking for in the future. Performance is certainly one important criterion and we are no slouches there, but there are several other reasons to use our compilers. Among them:
Integrated IDE, with advanced debugger, profilers with GUIs
Advanced MT support to help move applications to exploit future multicores
Productivity tools like Thread Analyzer, Memory debugger, etc
Advanced Parallelism features like -xautopar, OMP API support, etc
Common platform support across SPARC, x86/x64, Solaris and Linux
But coming back from "the marketing message" to this topic, we'll be happy to look at where we are slower than others and go fix those areas of weakness. We very much appreciate your help in pointing out these areas.
# 7
very thx to all your replys. and today when I try to do the benchmark on a x86_64 box, I encount an *very seriously* bug in SunStudio12. My compile process end up with a error msg says:
sunCC -L./lib-o lib/bmwdict lib/bmwdict.o lib/bmw.o lib/dbnew.o -lcommon
/usr/lib64/libm.so: file not recognized: File format not recognized
make: *** [lib/bmwdict] Error 1
after Google for some while, I found this entry for it:
http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=6533627
and I can just reproduced this bug on my box. I don't know how you can get SPEC test result on the x86_64 Linux box.
the bug is submit on March, and it doesn't get fixed untill now. I want to know does this mean SunStudio currently have some *more seriously* bug than this so your engineers doesn't have time to look at it ?
# 8
Well according to this thread http://forum.java.sun.com/thread.jspa?forumID=855&threadID=5147264They need to release a new updated ld or get binutils to except thier changes.Can we get an updated ld with next patch, so all OpenSUSE 10.2 users will be
# 9
my system is slackware-current, and thx for your tips, I can use -Qpath /usr/bin to get things work now
# 10
> I don't know how you can get SPEC test result on the x86_64 Linux box.
Install on a supported platform? :) That might help...
> I want to know does this mean SunStudio currently have some *more seriously* bug than this so your engineers doesn't have time to look at it ?
On *supported* platforms - no.
Look, you do not have to fight us, really. We try to play honest and tell you which platforms do we develop and test our compilers on.
We seriously plan to extend the list of supported platforms in upcoming Express releases, but our first and foremost goal in Studio12 release was to get as much functionality as possible on those *supported platforms*.
We would just distract ourselves trying to kill 10 more birds with that single stone.
Would you tell that getting gcc-style inline assembler fully working on supported platforms is less important than trying to forwardport our linker fix to every available linker out there?
regards,
__Fedor.
SFVa at 2007-7-12 19:34:08 >
# 11
"gcc-style inline assembler fully working" ? I don't think so.
my code:
static INLINE void
cnqp_add(int i, int *v)
{
__asm__ __volatile__(
"lock addl %1,%0"
:"+m" (*v)
:"ir" (i));
}
static INLINE void
cnqp_inc(int *v)
{
__asm__ __volatile__(
"lock incl %0"
:"+m" (*v));
}
has to be changed to
static INLINE void
cnqp_add(int i, int *v)
{
__asm__ (
"addl %1,%0"
:"+m" (*v)
:"ir" (i));
}
static INLINE void
cnqp_inc(int *v)
{
__asm__ (
"incl %0"
:"+m" (*v));
}
I can understand you don't like "__volatile__", because it's gcc special, but I can't understand why you don't support "lock". without "lock" ,how can we " autopar" ? :-)
and after some time played with -Qpath /usr/bin, I found it isn't the right way to resolve the problem:
luoyi@test:~/qp$ sunCC -L./lib -o lib/cnqp_mod_bmw.so lib/cnqp_mod_bmw.do lib/bmw.do lib/dbnew.do -lcommon_d -Qpath /usr/bin
/usr/bin/c++filt: invalid option -- f
Usage: /usr/bin/c++filt [options] [mangled names]
Options are:
[-_|--strip-underscore]Ignore first leading underscore
[-n|--no-strip-underscore] Do not ignore a leading underscore (default)
[-p|--no-params]Do not display function arguments
[-i|--no-verbose]Do not show implementation details (if any)
[-t|--types]Also attempt to demangle type encodings
[-s|--format {none,auto,gnu,lucid,arm,hp,edg,gnu-v3,java,gnat}]
[@<file>]Read extra options from <file>
[-h|--help]Display this information
[-v|--version] Show the version information
Demangled names are displayed to stdout.
If a name cannot be demangled it is just echoed to stdout.
If no names are provided on the command line, stdin is read.
# 12
> "gcc-style inline assembler fully working" ? I don't think so.
He did not say it was fully working now, he said they were giving this a higher priority than supporting random distributions of linux (which they are still trying to do). The first patch to studio 12 (some time this summer) is supposed to improve the gcc-style inline assembler significantly (though I don't know if it will work with your particular exemple, the support gets added step by step).
OTOH I agree that a link somewhere in the documentation to the gnu linker patch would be very nice.
# 13
> "gcc-style inline assembler fully working" ? I don't think so.
> my code ... has to be changed ...
Can you, please, show me the command line and compilation error (please, add -V there to show the compiler version)?
We surely implemented __volatile__.
As for the "lock" stuff - thats a know problem which has a simple workaround - specify any optimization level (-Ox) and you'll be fine.
As Marc has already pointed out, priorities is exactly what I meant telling that we cant go conquer whole world until we got important functionality limping.
> and after some time played with -Qpath /usr/bin, I found it isn't the right way to resolve the problem:
The right way to resolve the problem with nonfunctional "ld" is just to remove "ld".
It will not do any good for you on that platform, and as you already figured out -Qpath approach has its own bad sides.
SFVa at 2007-7-12 19:34:08 >
# 14
sorry, SFV, it's my fault that say suncc doesn't support __volatile__ without do some more expriments. I just used suncc to compile my code and it throwd out some error msg on the asm line, then I made changes as I post, then all things OK. sorry again for my careless.
I've tested bmwdict on my x86_64 box, and it shows up 1s slower than gcc 4.1.2(14s vs 13s) I'll clean my code and put it to public access. hope that'll help.
# 15
We expect a patch for Studio 12 compilers on Linux in late July. (That's not a promise, but a guess.) The C++ compiler accepts the sample asm code, but the C compiler has not yet been updated to accept the syntax.
