crash on dlopen a dldumped library
Hi folks,
I am trying to find a way to achieve faster startup and better runtime performance for a library. I built a non PIC version of a library and I found the runtime performance is much better than the PIC version. However, it comes with a cost for startup, the huge relocation overhead. Someone told me to run "crle" with "-f RTLD_REL_RELATIVE" to the non PIC version of the library and I found it significantly reduced the startup time. It is cool.
In the meantime, I found that the crle (d) version of that library requires itself to be always mapped to a fixed address. If I purposely use that address to map something else, then dlopen to that library will fail. I am just wondering whether someone can shed light on this? I thought the crle(d) version should relocate itself to somewhere else when something like that occurs. I can tolerate the relocation overhead in that case which hopefully would be rare.
Thanks,
-Xiaobin
[968 byte] By [
jxlua] at [2007-11-26 13:57:03]
# 1
Let's review a few things about object files, relocations, and PIC.
When linked into an executable, the relative locations in an object file need to be relocated to the virtual addresses where they will be found, along with the references to such symbols. The relocation is generally performed at link time, or for dynamic libraries, at library load time. The third option suggested to you was to do the relocations with crle so they don't have to be done at library load time. Since crle does the one-time relocation, you have to put the library in a block of unused addresses. If that block is not free for some application, the crle-d library can't used with that application.
Position-independent code (PIC) is used in shared libraries so that only global symbols and references to them need to be relocated. All other references are independent of the location of the code in memory. This feature allows the same code to be shared by multiple processes at the same time. Sharing a library means that only one copy of it resides in computer memory, and only one copy needs to occupy disk space.
When you put non-PIC code in a dynamic library, all non-PIC symbols and references to them must be relocated -- either at library load time, or by first running crle. The library code is not actually shared. Every program using the library gets its own copy. You wind up with most of the disadvantages of static code and of shared libraries, with few of the advantages of either.
So let's review some options.
1. You don't say what platform you are using -- sparc, x86, x64. On sparc, you have two options for PIC code: -KPIC (-xcode=pic32) and -Kpic (-xcode=pic13). the -xcode=pic13 option generates more efficient code, but limits the number of external symbols in the library. Did you try --xcode=pic13? If you don't hit the symbol limit and the performance is adequate, problem solved. (If you exceed the symbol limit, you find out when you build the library.)
2. If instead of a dynamic (.so) library you build a static (.a) library, you are done. The code will be as efficient as possible, and you don't have any load-time relocations. The disadvantages of a static library are
- each program using the library has its own copy (but only the parts that it uses).
- you must relink each program if you update the library.
- you cannot dynamically load the library -- it is permanently linked in the application.
If --xcode=pic13 still does not meet your needs and you must have a dynamic library, then you need someone like a sysadmin to reserve a block of addresses for the crle-d library. Ensure that all programs using the library keep that block available. Make the reserved block big enough to allow for library expansion.
You might get other suggestions in the Open Solaris linker forum:
http://www.opensolaris.org/jive/forum.jspa?forumID=63
# 2
Thanks for your reply.
Couple related questions.
I am wondering if I mix non-PIC and PIC code together in a library. Will the library be partially shared by multiple processes?
Secondly, how can I make crle to relocate the library to an used address? Is there such an option where I can teach crle to pick up that address?
I don't think linking it into .a file is a viable option for us. But I am going to try -xcode=pic13 to see if there is any differences. We are actually using both sparc and x86 platforms.
jxlua at 2007-7-8 1:36:54 >
# 3
> I am wondering if I mix non-PIC and PIC code together
> in a library. Will the library be partially shared by
> multiple processes?
Sharing is per-page. If a page contains code with no relocations that need to be modified, it will be shared. In principle, you could figure out which pages, if any, meet the criteria for sharing, or you could monitor the process to see which pages are modified. In practice, this would be a lot of work for little payoff.
> Secondly, how can I make crle to relocate the library
> to an used address? Is there such an option where I
> can teach crle to pick up that address?
Check the crle man page, and the Linker and Libraries Guide at docs.sun.com
> I don't think linking it into .a file is a viable
> option for us. But I am going to try -xcode=pic13 to
> see if there is any differences. We are actually
> using both sparc and x86 platforms.
On x86, there is only one form of PIC. That is, the -KPIC and -Kpic options are the same, and the -xcode option is not available. If you have already determined that PIC on x86 does not have acceptable performance, you'll have to explore the other alternatives.
I have another suggestion that was implied by what I posted before. Find the computational hot spots in the library, and put them in a static library. The rest of the code can be compiled as PIC and put in a shared library. Then put the static library on the link command line after the shared library.
