Notes on GDB

GDB user manual

GDB internals documentation

A useful GDB initilization script .gdbinit (a local copy here) Detailed explanation here

Hacking GDB

To see how a function in GDB is implemented, seek calls to the following functions in GDB source tree: 
add_com
add_com_alias
add_info
add_info_alias
where add_info add a hook to the implementation of commands beginning with info.

For example, info functions command is added in function _initialize_symtab gdb/symtab.c as 

  add_info ("functions", functions_info,
        _("All function names, or those matching REGEXP."));
so the actual implementation is functions_info

Invoking GDB

gdb program
gdb program -c core Debug a program with core file
gdb -p pid Debug a running process whose ID is pid
gdb --args program arg1 arg2 Debug program and pass arg1 arg2 to program

Running debuggee ("inferior" in GDB parlance) and set up its environments

r > outputfile Run the debuggee and redirect all console output to outputfile
start Run the debuggee and stop at main
kill Kill the debuggee. Use "r" again to restart the debuggee.
set args arg1 arg2 Set the command-line arguments arg1 arg2 for debuggee
show args
set env varname=value Set the environment varname=value for debuggee
unset env varname Remove the environment variable varname
set exec-wrapper env 'LD_PRELOAD=libfoo.so' Run the debuggee with wrapper program env and use 'LD_PRELOAD=libfoo.so' as env's command-line argument
show env
attach pid Attach to the running process pid
info threads List threads
thread threadID Switch to thread threadID
set non-stop on When one thread stops, GDB will not stop other threads.
set scheduler-locking on Only the current thread may run when the debuggee is resumed.
set scheduler-locking step In the single-step mode, only the current thread may run.
info inferiors List processes
shell command string Invoke a standard shell to execute command string
set follow-fork-mode child
set follow-fork-mode parent
show follow-fork-mode		 Follow the child process on fork. The default is to follow the parent.
set follow-exec-mode new
set follow-exec-mode same
show follow-exec-mode		 On exec, creates a new inferior and rebinds the process to this new inferior. The default is the same.
set detach-on-fork off
show detach-on-fork		 All forked processes will be held under the control of debugger. The default is on.
set logging on
set logging file foo
set logging off
set logging overwrite off
show logging		 Enable logging and set the output to foo. The default logging file is gdb.txt

Break points, watch points, single-step execution

One should set the environmental variable LD_BIND_NOW to 1 if one wants to set a break point in a function in a dynamic link library.

set breakpoint pending on When setting a break point on a not-yet-loaded dynamic library, this will automatically create a pending break point WITHOUT asking.
b function Set a break point at first line of function
tbreak function Set one-time break point at first line of function
b num Set a break point at line number num
b *0x12345 Set a break point at address 0x12345
b *function Set a break point at actual starting address of function
b *0x12345 if foo==5 Stop at address 0x12345 if foo==5

See here for more details on conditional break points.

watch foo Stop if the variable foo changed.

The number of watch points is limited (depending on hardware). On x86, only 4 hardware watch points can be set simultaneously.
set can-use-hw-watchpoint 0 Do not use hardware watch points.
watch *0x12345 Stop if the value at address 0x12345 changed
rwatch foo Stop if the variable foo is read.

This feature is hardware dependent.

awatch foo Stop if the variable foo is read or written.

This feature is hardware dependent.

catch syscall Stop when any system call occurs.
catch syscall [name| number]
catch exec
catch fork
catch vfork		 Stop when a specific system call (e.g. exec, fork, etc) occurs.
catch throw Stop when a C++ exception is thrown.
handle signal stop
handle signal nostop
handle signal print		 Signal handling. 
b function if foo==5
commands
  silent
  printf "x is %d\n",x
  c
end
Execute the commands "silent", "print...", and "c" after reaching the break point at function
info break List all break/watch points
clear Remove all break/watch points
clear function Remove the break point at function
disable num Disable the break point num (use "info break" first to get a list of break/watch points)
c Continue execution after reaching a break point
rc Reverse execution until reaching a break point
finish Continue execution until the current function returns
return 1 Return from current function immediately with value 1
s Execute one line of source code
n Same as above, but don't jump into function calls
si Execute one machine instruction
ni Same as above, but don't jump into function calls
reverse-step Reverse execute one line of source code.

See here for a tutorial.

reverse-next Same as above, but don't jump into function calls
reverse-stepi Reverse execute one machine instruction
reverse-nexti Same as above, but don't jump into function calls
checkpoint Save a snapshot of debuggee's state
restart checkpoint_id Restore the debuggee's state to checkpoint_id
delete checkpoint checkpoint_id Delete checkpoint_id
info checkpoints Show all checkpoints
(Record and replay the execution) See here for details and here for a tutorial.

Examining the stack

bt Print a backtrace (list of stack frames)
bt full Print a backtrace as well as local variables
f num Select frame number num
down num Move down num frames (most recent ones)
info f Print info about the frame
info args Print arguments passed to the frame
info local Print local variables of the frame

Examining source files and machine code

list function Show source code of function
list filename:function Show source code of function in file filename
list *0x12345 Show source code at address 0x12345
info line function Show the starting and ending address of the machine code for function
disas function Disassemble function
set disassembly-flavor intel
set disassembly-flavor att		 Use specified syntax for disassmbly
disas/m function Disassemble function and show the source code side-by-side (the debuggee must be compiled with -g option)
disas/r function Disassemble function and show the raw instructions in hexadecimals
set disassemble-next-line on Disassemble the next instruction whenever GDB stops
set print asm-demangle on Show demangled symbol names in disassembly 
define hook-stop
x/i $pc
end
Disassemble the next instruction whenever GDB stops.

This is a special example of user-defined hooks.
x/4i 0x12345 Show the machine code for the first 4 instructions starting at address 0x12345
x/4i $pc-6 Show the machine code of 4 instructions starting at current program counter minus 6

Examining data, memory, registers

p var Print the value of var.

var can be a register, e.g. $r1

whatis var Print the data type of var.
ptype foo Print the detailed data type of symbol foo. This is especially useful if foo is a C struct.
p/t var Print the value of var in binary
p/c var Print the value of var in format c:
x=hexadecimal, c=char, t=binary, d=signed, u=unsigned, o=octal, a=address, f=floating
p function::var Print the value of var in function
p *array@len Print the values of array (allocated by malloc) with length len
call getpid() Print the PID of current process
info inferiors List processes
x &var Print the address of var.
info address foo Print the address of symbol foo.
info symbol 0x12345 Print the name of the symbol which is stored at 0x12345
info macro foo Print the definition of macro foo.

To use this feature, first, the debuggee must be compiled with -ggdb3 flags, and second, one must start the debugee (e.g. via start command) or execute the list command.

To see the definitions of all macros (e.g. from header files), compile the program with -ggdb3 and use the dwarfdump utility:

dwarfdump a.out | grep DW_MACINFO_define
macro expand foo Expand the definition of macro foo. The prerequisite of this command is the same as info macro command.
x/1s 0x12345 Print 1 null-terminated string starting at 0x12345
x/5xb 0x12345 Print 5 bytes of memory in hexadecimal starting at 0x12345
x/nfu 0x12345 n is repeat count
f is format (i=instruction, s=string, x=hexadecimal, c=char, t=binary, d=signed, u=unsigned, o=octal, a=address, f=floating)
u is unit (b=byte, h=2 bytes, w=4 bytes, g=8 bytes)
info registers
info float
info vector
info all-registers
 List regular/floating-point/vector/all registers
p/x $pc Print the program counter
display var Automatically display var each time GDB stops. 
define hook-stop
print var
end
Display var each time GDB stops.

This is a special example of user-defined hooks.
display/nfu 0x12345 Display address 0x12345 (with format nfu) each time GDB stops.
info display List all automatic displays
undisplay num Remove display num
find start_addr, +len, val1 [, val2, ...] Search memory
heap Show heap usage and classify heap objects

This is a Fedora extension for GDB. See here and here for details.

Changing data, memory, registers, execution

set var foo+=5 Set the value of variable foo to increment by 5
set {int}0x12345=6 Set the value of at address 0x12345 to be 6 (treat as an integer)
set $rax=$rbx+5 Set the value of x86_64 register RAX to be the sum of 5 and register RBX
set $ps=$ps|1

set $ps=$ps&~1

 Set/Clean the Carry flag (x86)
set $ps=$ps|0x4 Set the Parity flag (x86)
set $ps=$ps|0x40 Set the Zero flag (x86)
set $ps=$ps|0x80 Set the Sign flag (x86)
set $ps=$ps|0x800 Set the Overflow flag (x86)
set write on Make text region writeable
signal SIGSEGV Send the signal SIGSEGV to debuggee
generate-core-file Create a core dump file

Examining the symbol table

info scope foo Print all variables in the scope of function foo
info shared Print info about the shared libraries
info source Print info about the source file
info functions Print info about all functions
info variables Print info about all variables

Dump memory to files

dump memory file 0x12345 0x34567 Dump the memory region 0x12345 to 0x34567 to file
restore file 0 0x12345 Load file content to memory starting at 0x12345

Memory region attributes

mem 0x12345 0x34567 ro Set the memory region from 0x12345 to 0x34567 to be read-only (this restriction applies to GDB only)
info mem List all memory regions
delete mem num Remove memory region num
disable mem num Disable memory region num

Tracing facility

trace foo.c:121 Set a tracepoint at line 121 of foo.c
action Set a series of actions for tracepoint
tstart Start the trace experiment
tdump Dump the trace
tstatus Print info about current trace data collection

Troubleshooting GDB

What is
[Thread debugging using libthread_db enabled]
Cannot find new threads: generic error
error ?

According to this link, it seems GDB has issue when the debuggee "suddenly" loads libpthread.so. The fix is to start GDB with 
 $ LD_PRELOAD=/lib/libpthread.so.0 gdb --args ./myapps
or at GDB prompt (untested!), 
 (gdb) set exec-wrapper env 'LD_PRELOAD=/lib/libpthread.so.0'

Target does not support this type of hardware watchpoint error ?

When setting a read-access watch point (rwatch), one could encounter this error. The fix is to launch the debuggee first (e.g. set a break point somewhere), then use rwatch: 
 (gdb) start
 (gdb) rwatch foo