In Solaris, many of the routines called from libc are “direct bound” so that references from inside libc will always find the function implementations that are inside libc. This approach prevents outside libraries from interposing (substituting) different implmentations of common functions. The largest exception to this is the malloc family of routines. The malloc routines (when called by libc, for example from strdup) MUST call an externally supplied malloc routine, if one is supplied via LD_PRELOAD or library link ordering.

There is a huge gotcha related to malloc interposing. If you get a pointer from malloc, you have to free it using the free routine in the same library that allocated it. But how do you guarantee that? If every program has libc, then every program will have at least one allocator in it. Any program that uses libumem will have at least two (one from libumu and one from libc). If the user wants to LD_PRELOAD their own memory checker library, it just gets worse.

It gets even worse because malloc libraries implement many additional routines to allocate memory. Let’s say my app calls valloc in libc. Let’s say I want to interpose libmymalloc because it has a spiffy new memory checker that I want to use. Now let’s say libmymalloc doesn’t include a definition for valloc. My app will crash, because valloc gets memory from the libc pool, and free will free it to the libmymalloc pool.

At this point there are people who will say: “Easy, just make sure they all implement the same set of functions.” Well yes, that would solve the problem, if there were a way to do this. But there is no standard for what this list of routines is. Memory allocation libraries are useful because they offer additional functionality beyond the plain malloc and free in libc. So they will always be adding functions that are not in anyone else’s implementation. If I write my app to use libmalloc_alpha.so, and someone interposes libmalloc_beta.so, then all the custom functions in libmalloc_alpha.so that I was calling will still go to libmalloc_alpha.so, but all the customary ones will go to libmalloc_beta.so. The result is undefined.

Unfortunately the idea that you can replace a memory allocator library by just interposing a different one is a widely known “fact”. You can read about how this bit the Solaris libraries in bugid 4846556. The problem came up in comp.unix.solaris recently as well.