Archive for October, 2005

The story of lazy stabs

Thursday, October 27th, 2005

There’s a dbx feature called “lazy stabs” that is clever, but a little confusing sometimes. I figured I’d talk about it a little to give an overview of what’s happening. There are really two parts to the idea of “lazy stabs”, one part is something we do all the time (demand loading most information when you first visit a source file), and the other part depends on how you compile your code (most debug info can be aggregated into the a.out or it can be left in the .o files).

You can read more about stabs and dwarf here:

stabs versus dwarf

Index debug info (stabs and dwarf)

Dbx will always demand-load line number information and other information about local symbols. So until you visit a source file for some reason, dbx won’t bother loading the majority of the debug information for that file. This makes dbx start up much faster. When you first load the binary, only the global symbols and other index information are loaded.

For example, if you want to stop at “foo.h:12” then dbx needs to load the detailed source information for all files that include code from foo.h. Then dbx figures out which object files have code from line 12, and sets the breakpoint(s) you need.

In stabs, the index information is stored in the .stab.index section. In dwarf, the index information is stored in multiple sections with names like: .debug_pubnames, .debug_varnames etc.

reading debug info from .o files (stabs only)

Stabs were carefully designed not to depend on relocation records (which need to be resolved by the linker).

For most functions and variables, stabs uses the linker name of the function or variable to represent that object. At runtime dbx will access the global symbol table in the a.out and look up the symbol by name. For C++, this process uses the linker name of the symbol, and the character strings recorded as part of stabs can get very very huge. A few releases ago, we started using a compressed form of mangled names, which helped somewhat.

Because of that design, dbx can read most stabs from a .o file, and make sense of them. (Of course, the index stabs still come from the a.out.) This allows the a.out to have a smaller size on disk. Note that having stabs in a program never affects the run-time size of a program, or it’s performance, because stabs are not loaded at runtime.)

This has bitten some people working on mozilla in the past:

Dwarf information encodes the absolute addresses of functions and variables, and so it needs to be relocated by the linker in order to make sense. That means we can’t support this aspect of “lazy stabs” (really it’s better called “dispersed stabs” or something like that). The a.out has to include all the dwarf information for the program. In exchange for this, dwarf takes up significantly less space in C++ programs that use long mangled names.

Larger a.outs (dwarf, or stabs with -xs)

When using stabs, you can compile with the -xs flag which will tell the compiler to collect all the stabs into the a.out. Dbx will still demand-load them, but it works better if you want to archive the binary with debug information, or if you want to clean up your build area, but keep a debuggable binary. When dbx is loading stabs from the .o files, if you move the directory that has the .o files in it (or move the .o files themselves), then you have to use the pathmap command in dbx to tell dbx where they went to.

(Aside: You might think -xs would logically be used at link time, but you need to use -xs at compile time so the compiler can tag the stabs sections with a flag that means “accumulate into a.out”. This flag causes the linker to aggregate the stabs at link time. )

The increase in a.out size with dwarf will probably be a surprise to people who are used to smaller a.out’s when using stabs, but I’m personally looking forward to it. I’ve had to deal with many many users over the years who wanted to send me a binary with debug information in order to reproduce a bug, but the a.out is normally missing the majority of the debug information with stabs. I had to tell them to rebuild their program with the -xs option, or else tar up the entire build tree and send it to me. With dwarf, that problem won’t come up again. Everything will be in the a.out.

One important thing to remember is that the stabs and dwarf information isn’t ever loaded into your program when it’s run. So it won’t affect the runtime performance or take up any memory when your program runs. The information only takes up disk space. And it’s disk space that is also taken up by the .o files. So if you previously were saving your object files so that you could debug your program, you can stop doing that if you are using a compiler that emits dwarf. It also makes it easier to keep the non-stripped version of a binary that you strip and ship as part of a product.


Tuesday, October 4th, 2005

So your program has Obj * p; in it, and you stop in dbx and say print *p Dbx comes back and says the memory is illegal or unmapped. But then you continue your program, and your program reads from *p just fine. What’s going on?

This gotcha showed up again on a Sun alias. It’s been known to happen with a large database application whose initials are O. It’s not really a bug in dbx or a bug in the program, it’s just a slightly confusing “feature” in the Solaris.

Solaris allows you to call mmap to allocate memory in the address space, and it supports an option called MAP_NORESERVE. When you supply this option, the kernel will create a range of virtual memory that can be accessed by the program, but this memory will be filled with zeros on demand. Swap space is not reserved for the memory until it’s used, which makes it great for very large arrays that are only sparsely filled in.

Unfortunately, this creates memory which is perfectly valid from the program’s pont of view, but which Solaris (and the /proc interface) doesn’t admit is really there, when dbx asks about it.

In theory this bug could be “fixed” by modifying the /proc interface in Solaris so that it allows reading from all NORESERVE addresses and returns a value of 0. I suppose the kernel should prevent /proc from writing to any of these locations unless the user made some explicit statement that they wanted to modify the memory map of the target process.

It’s usually not a big deal to work around, you just have to be aware that dbx will tell you something is unmapped, even though the program might correctly be able to read/write to that location.