Debugging GlastRelease with gdb

  bash$ source setup.sh

You can run gdb

• Directly from the shell, like this:

  $ gdb test_CalibSvc.exe
  Copyright 2001 Free Software Foundation, Inc.
  GDB is free software, covered by the GNU General Public License, and...
  (gdb)
  

• Using the graphical interface ddd:

  $ ddd test_CalibSvc.exe
  

  From the ddd man pages:

      "Classical" UNIX debuggers such as the GNU debugger  (GDB)
      provide  a  command-line interface and a multitude of com-
      mands for these and other debugging purposes.   DDD  is  a
      wrapper  around  an  underlying  GDB  or DBX debugger.  In
      addition to the GDB or  DBX  command-line  interface,  DDD
      provides  a  graphical  user  interface  to support common
      debugging tasks.  As a special  feature,  DDD  provides  a
      graphical data display for interactive exploration of data
      structures.
  

  See the ddd man pages or info pages for more information.
• From within an emacs session. This is the option I find the most convenient. See detailed instructions in the following section.

M-x gdb

Run gdb (like this): gdb $CALIBSVCROOT/rh72_gcc2953/test_CalibSvc.exe

(gdb) break main
(gdb) run

Typical things you'll want to do in gdb, as in any debugger, include

• Using break points
  • break (set a breakpoint)
  • info break (view breakpoints)
  • disable
  • enable
  • ignore N COUNT (ignore breakpoint N for COUNT iterations; then break)
  • commands (see full description below)
  • delete
• Executing code
  • set args (set one or more arguments to the program before running or re-running it)
  • run (start running from the beginning)
  • continue (Continue execution until the next break point is reached or the program exits.)
  • next (Execute one source line, stepping over function calls.)
  • step (Execute one source line, stepping into function calls.)
  • finish (Continue execution until the end of the current function call.)
  • until (Execute until specified target is reached; like a one-time only break. Target defaults to next line.)
• Examining and changing data
  • print (Print an expression.)
  • x (Examine a memory location.)
  • set variable (Evaluate expression and assign to variable.)
• Debugging a newly built version of a program.
  • quit (Ends current gdb session.)
  • gdb or, within emacs, M-x gdb (Restart gdb.) Within emacs, if you have quit an earlier session, you will be prompted with the same program name you used before. If that's what you want to debug, as is usually the case, all you have to do is hit Enter

For more information, such as description of arguments, on these or any other commands, simply ask for it in the debugger buffer, e.g.

(gdb) help break
Set breakpoint at specified line or function.
Argument may be line number, function name, or "*" and an address.
If line number is specified, break at start of code for that line.
If function is specified, break at start of code for that function.
If an address is specified, break at that exact address.
With no arg, uses current execution address of selected stack frame.
This is useful for breaking on return to a stack frame.

Multiple breakpoints at one place are permitted, and useful if conditional.

Do "help breakpoints" for info on other commands dealing with breakpoints.
(gdb) 

gdb won't let you set breakpoints in code which hasn't been loaded yet. One way to handle this is to step your way past the Gaudi configure step. At that point component libraries have been loaded and you can set breaks in the usual way in all code. If you rerun the program without restarting the debugger you will not have to reset breakpoints in component libraries. gdb will complain that it can't set the breakpoints, but then when the execution passes the point at which the component libraries have been loaded it will stop complaining and the breakpoints will be in effect.

Depending on the precise version of the main test program and Gaudi version, you'll need something like the following sequence of commands to get you past the configure step:

(gdb) break main
(gdb) run                                                emacs frame
(gdb) break 89 break on line 89 of current routine (main)
(gdb) cont                                               emacs frame
(gdb) s 5      Same as step command issued 5 times       emacs frame
(gdb) n 3      Same as next command issued 3 times       emacs frame

However, don't take this too literally. The initial break in main should be set at the line invoking appMgr->run(), which might not be line 89. Line 89 is correct test programs using recent (as of May, 2006) versions of the GlastPolicy package, which contains TestGlastMain.cxx. Line 91 is the correct line when running Gleam (as oppose to test_Gleam). Depending on Gaudi version, the numbers 5 and 3 in the example might not be correct.

In order to see what's going on after stepping into the line appMgr->run() (allowing you to determine how many step and how many next commands you need to issue) gdb has to know where the Gaudi source is. For the SLAC central Linux installation of Gaudi v18r1, you can copy these lines to a file and issue them from gdb. Suppose your file is called gaudi_v18r1.gdb. Then from gdb type

If you're debugging on a remote Linux machine, adjust the directories in the file to point to your installation of Gaudi.

On occasion (like now: Oct. 2003) I've seen weird behavior when I try to step into a routine, but setting a break on the routine (or an executable line within it) is OK. If you're seeing symptoms like this, an alternate way to get past the configure step is

(gdb) break main
(gdb) run 
(gdb) break 89 break on line 89 of current routine (main)
(gdb) cont 
(gdb) break ApplicationMgr::run Can't set this break until the
                                ApplicationMgr instance has been created
(gdb) cont 

After the final cont you can issue n 3 to get past the configure step.

[Note added May 2006: I haven't had this problem for a long time. I doubt this gambit is ever necessary any more.]

A few gdb commands which I've never had occasion to use before can come in very handy when debugging a large complicated application with dynamically-loaded libraries like Gleam. Detailed description in grey indicates the command is not likely to be needed in a typical debugging session.

print var-name.method(..)

If name is a std::string you may see extra characters if you just ask gdb to print it, but you can invoke the c_str() method, just as you would in code:

(gdb) print name.c_str()
$2 = 0x83b4fc8 "EventCnvSvc"

commands break-number

Specify one or more commands to be executed each time the breakpoint (specified by number) is reached. The commands to be executed are typed in one to a line, followed by a line containing just the word "end".

control-C control-C

This sequence issues SIGINT. Since our applications don't handle SIGINT, the default action is for the debugger handle it by stopping execution and bringing up the (gdb) prompt. You can then examine variables, set breakpoints, etc. This can be useful if you want to break in to set a breakpoint while the application is waiting for interactive input (e.g., request for next event) or if application execution seems to be going wild.

info sharedlibrary

shows you which shared libraries have been loaded. You can't set a breakpoint in a shared library until it's been loaded. (But gdb does keep track if you rerun the program. It will complain when you first restart the program that breakpoints in dynamically-loaded shareables can't be enabled, but will automatically reactivate them when the shareable is again loaded.)

directory path-specification

This command adds a new directory to the list which will be searched for source code. By default this list includes only the current directory and the path of the source when it was compiled. Usually this is good enough, but not if object files or the source they were compiled from has moved. In particular, the pre-compiled version of Gaudi9 available to public Linux machines at SLAC fell into this category; it seems that Gaudi11 (v11r4p0) does also. In order to help gdb find the source for GaudiKernel, for example, you can issue the following (as of October, 2003, for $GLAST_EXT/gaudi/v11r4p0)

(gdb) directory /afs/slac/g/glast/ground/releases/GaudiSys_v9r0p6/GaudiKernel/v11r0p2/src/Lib

For some unknown reason in this context gdb would not translate an environment variable; I could only make this work by typing out the full absolute path.

To see what the current search path is, use the command show directories.

The commands necessary to get to the source for Gaudi9 GaudiKernel and GaudiSvc packages can be found in ~jrb/util/gaudi9src.gdb. For Gaudi11 (v11r4p0) see ~jrb/util/gaudi11r4p0src.gdb. To run these commands from within gdb (on SLAC Linux), type

(gdb) source ~jrb/util/gaudi11r4p0src.gdb

gdb can be used as a post-mortem tool as well as for debugging a "live" executable. Suppose there is a dump from a crash of Gleam.exe named core.1234. Then, assuming for simplicity that Gleam.exe and core.1234 are in your current directory (if not, include their paths), from the command line you can type

   $ gdb  Gleam.exe core.1234

This will show all the loaded libraries; if asked whether to quit or continue, type <return>.

Usually the first thing you'll want to know is where the program crashed. Type

   gdb> backtrace

to get the complete call stack or

   gdb> backtrace N

where N is a positive integer. In the latter case gdb will show you the innermost n active stack frames. You can then examine variables in the current (innermost) frame in the usual way, or move from one frame to another by means of the commands

  gdb> up N
  gdb> down N
  gdb> frame N

where up N moves N frames towards the outermost frame, down moves inwards, and frame N selects frame N where the innermost frame is frame 0. Here is a nice explanation of relevant concepts, extracted from the gdb online help segment Examining the Stack:

When your program has stopped, the first thing you need to know is where it stopped and how it got there.

Each time your program performs a function call, information about the call is generated. That information includes the location of the call in your program, the arguments of the call, and the local variables of the function being called. The information is saved in a block of data called a "stack frame". The stack frames are allocated in a region of memory called the "call stack".

When your program stops, the GDB commands for examining the stack allow you to see all of this information.

One of the stack frames is "selected" by GDB and many GDB commands refer implicitly to the selected frame. In particular, whenever you ask GDB for the value of a variable in your program, the value is found in the selected frame. There are special GDB commands to select whichever frame you are interested in.

From within gdb, just type help and you will get this:

(gdb) help
List of classes of commands:

aliases -- Aliases of other commands
breakpoints -- Making program stop at certain points
data -- Examining data
files -- Specifying and examining files
internals -- Maintenance commands
obscure -- Obscure features
running -- Running the program
stack -- Examining the stack
status -- Status inquiries
support -- Support facilities
tracepoints -- Tracing of program execution without stopping the program
user-defined -- User-defined commands

Type "help" followed by a class name for a list of commands in that class.
Type "help" followed by command name for full documentation.
Command name abbreviations are allowed if unambiguous.

Here is typical output from a help command where the argument is one of the class names. I didn't bother to type the whole name since it's unambiguous.

(gdb) help dat
Examining data.

List of commands:

call -- Call a function in the program
delete display -- Cancel some expressions to be displayed when program stops
delete mem -- Delete memory region
disable display -- Disable some expressions to be displayed when program stops
disable mem -- Disable memory region
disassemble -- Disassemble a specified section of memory
display -- Print value of expression EXP each time the program stops
enable display -- Enable some expressions to be displayed when program stops
enable mem -- Enable memory region
inspect -- Same as "print" command
mem -- Define attributes for memory region
output -- Like "print" but don't put in value history and don't print newline
print -- Print value of expression EXP
printf -- Printf "printf format string"
ptype -- Print definition of type TYPE
set -- Evaluate expression EXP and assign result to variable VAR
set variable -- Evaluate expression EXP and assign result to variable VAR
undisplay -- Cancel some expressions to be displayed when program stops
whatis -- Print data type of expression EXP
x -- Examine memory: x/FMT ADDRESS

Type "help" followed by command name for full documentation.
Command name abbreviations are allowed if unambiguous.

Now suppose I want to know more about x

(gdb) help x
Examine memory: x/FMT ADDRESS.
ADDRESS is an expression for the memory address to examine.
FMT is a repeat count followed by a format letter and a size letter.
Format letters are o(octal), x(hex), d(decimal), u(unsigned decimal),
  t(binary), f(float), a(address), i(instruction), c(char) and s(string).
Size letters are b(byte), h(halfword), w(word), g(giant, 8 bytes).
The specified number of objects of the specified size are printed
according to the format.

Defaults for format and size letters are those previously used.
Default count is 1.  Default address is following last thing printed
with this command or "print".

Generally speaking, the help information is clearly written and well organized, as in these examples. If your gdb session is taking place inside emacs the output from gdb is just another buffer that you can scroll through, search, save to a file, etc., just as you would any other emacs buffer.

Using Info

You don't have to be running gdb to get help on its commands. From a typical Linux machine, including the public SLAC nodes, you can type

$ info gdb

or from within emacs type C-h i and go to the gdb node. info is a sort of pre-html hypertext documentation system, quite efficient to use once you know how. There are info pages for many commands besides gdb, including ddd. If you're a regular Linux or emacs user and don't already know about info type

$ info info

or C-h i from inside emacs, then type h (for "help") and spend a few minutes going through the tutorial. It's time well-spent.

See also a guide to using glastpack.

Created: 18 February 2003  J. Bogart
Last modified: