Notes on Calpont InfiniDB

Calpont InfiniDB Enterprise

Calpont InfiniDB Community Edition

(Below I assume Calpont InfiniDB Enterprise version 3.6.1, whose MySQL frontend is version 5.1.39)

Configuration

Suppose it is to be run by non-root user foobar and it is already installed under /opt/Calpont
  1. Create an entry in /etc/sudoer by root:
    foobar ALL = NOPASSWD: ALL
    
  2. Set up environment variables
    export INFINIDB_INSTALL_DIR=/opt/Calpont
    export LD_LIBRARY_PATH=$INFINIDB_INSTALL_DIR/lib:$INFINIDB_INSTALL_DIR/mysql/lib/mysql:$LD_LIBRARY_PATH
    source /opt/Calpont/bin/calpontAlias
    
  3. Under /opt/Calpont/lib, make sure symbolic links such as libbatchloader.so (linked to libbatchloader.so.1.0.0) are created.

    To check, execute ldd /opt/Calpont/bin/PrimProc

  4. Edit /opt/Calpont/mysql/my.cnf file.

    Set up socket file location, port number, datadir, max_allowed_packet, etc.

  5. Execute /opt/Calpont/bin/postConfigure

Start/Shutdown Server

  1. Set up environment variables
    export INFINIDB_INSTALL_DIR=/opt/Calpont
    export LD_LIBRARY_PATH=$INFINIDB_INSTALL_DIR/lib:$INFINIDB_INSTALL_DIR/mysql/lib/mysql:$LD_LIBRARY_PATH
    
  2. Start server: /opt/Calpont/bin/calpontConsole startSystem
  3. Stop server: /opt/Calpont/bin/calpontConsole shutdownSystem y

Command-line tools, log files

  • (MySQL) client

    /opt/Calpont/mysql/bin/mysql --defaults-file=/opt/Calpont/mysql/my.cnf -u root

  • Console

    /opt/Calpont/bin/calpontConsole

  • Debug logs

    tail -f /var/log/Calpont/debug.log

  • Info logs

    tail -f /var/log/Calpont/info.log

Process tree

Once Calpont InfiniDB is running (single-node mode), one should see the following process tree:
/bin/bash /opt/Calpont/bin/run.sh /opt/Calpont/bin/ProcMon
 \_ /opt/Calpont/bin/ProcMon
     \_ [ProcMgr]
     \_ /opt/Calpont/bin/controllernode fg
     \_ /opt/Calpont/bin/ServerMonitor
     \_ /opt/Calpont/bin/workernode DBRM_Worker1 fg
     \_ [PrimProc]
     \_ [ExeMgr]
     \_ [WriteEngineServ]
     \_ [DMLProc]
     \_ [DDLProc]
/bin/sh /opt/Calpont/mysql//bin/mysqld_safe --defaults-file=/opt/Calpont/mysql/my.cnf --datadir=....
 \_ /opt/Calpont/mysql/libexec/mysqld --defaults-file=/opt/Calpont/mysql//my.cnf --basedir=/opt/Calpont ...

Confluence Wiki Markup Langauge cheat sheet

Source: Confluence Wiki Notation Guide

Escape character \
Bold text *Bold text*
Italic text _Italic text_
Citation ??Citation??
Monospaced text {{Monospaced text}}
Single-line quote
bq. Single-line quote
Multiple-line
quote
{quote}
Multiple-line
quote
{quote}
    
Colored text {color:green}Colored text{color}
Underlined (inserted) text +Underlined (inserted) text+
Deleted text -Deleted text-
Superscript ^Superscript^
Subscript ~Subscript~

Heading size 1

.h1 Heading size 1
Heading size 5
.h5 Heading size 5
  • Bulleted list item 1
  • Bulleted list item 2
* Bulleted list item 1
* Bulleted list item 2
    
Note the space between * and the text following

* can be replaced by -

  1. Numbered list item 1
  2. Numbered list item 2
# Numbered list item 1
# Numbered list item 2
    
Note the space between # and the text following
  • List item 1
    1. Nested list item 1
    2. Nested list item 2
  • List item 2
* List item 1
*# Nested list item 1
*# Nested list item 2
* List item 2
    
Horizontal
rule
Horizontal
----
rule
    
(Long dash symbol) — ---
(Short dash symbol) – --
External link [External link|http://www.microsoft.com]
http://www.microsoft.com [http://www.microsoft.com]
Jump to my anchor
[#myAnchor]

[Jump to my anchor|#myAnchor]
    
External image !https://www.microsoft.com/logo.png!
Attached image "violin.png" !violin.jpg!
Attached image "violin.png" from "gallery macro" page !gallery macro^violin.jpg!
Image attributes !https://www.microsoft.com/logo.png|align=right, width=500!
Emoticons See here for a complete list.
Code block with syntax highlighting For XML:
{code:xml}
<test>
<another tag="attribute"/>
</test>
{code}
    

For source code:
{code:language=java|title=Bar.java|borderStyle=solid}
public String getFoo()
{ return foo; }
{code}
    
xml can be replaced by Java (the default), JavaScript, ActionScript, HTML, SQL (see here for a complete list)
Note/Info/Tip/Warning block
{note}Be careful{note}

{tip}Ask a question!{tip}

{warning:title=Don't Panic}Be happy.{warning}
	
{info:title=Don't Panic|icon=false}Be happy.{info}
	
Noformat block
{noformat}I do not want this text formatted!{noformat}
	
Charting See here for details.

"features" in ARM Linux's /proc/cpuinfo

Ever wonder what these "features" (hardware capabilities) in ARM Linux's /proc/cpuinfo mean ? Here is the answer (from Linux kernel source file arch/arm/include/asm/hwcap.h):

26bit Program Counter is 26-bit long
crunch MaverickCrunch instructoin extension. Now obsolete.
edsp DSP Enhancement instructions
fastmult Fast multiplication
fpa Floating Point Accelerator instructions. Now obsolete.
half ?
idiv Integer divide instructions available in both ARM and Thumb
idiva Integer divide instructions available in ARM only
idivt Integer divide instructions available in Thumb only
iwmmxt Intel Wireless MMX Technology, an implementation of MMX on embedded processors.
java Jazelle technology
neon NEON SIMD instructions
swp SWP (SWaP) instruction, which is used to implement a binary semaphore (mutex)
thumb Thumb instructions
thumbee Thumb Execution Environment
tls Thread local storage
vfp Vector Floating Point instruction extension
vfpv3 Vector Floating Point instruction extension version 3, with 32 double-precision registers
vfpv3d16 Vector Floating Point instruction extension version 3, with 16 double-precision registers
vfpv4 Vector Floating Point instruction extension version 4
Also, check this link.

Notes on ARM Cortex-A processors

Useful references

ARM Cortex-A series

Brawny cores still beat wimpy cores, most of the time by Urs Holzle of Google

ARM versus x86 performance comparison

12-core ARM cluster benchmarked against Intel Atom, Ivy Bridge, AMD Fusion

ARM (Nvidia Tegra 2/3, Samsung Exynos 5) vs. & Intel Core i7 performance & power consumption comparison

ARM Cortex-A family architecture by Hiroshige Goto

ARM Cortex-A series

Architecture Instruction set Extra features Clock rate (GHz) Cores Implementations
Cortex-A5 ARM v7, Jazelle, Thumb-2 DSP, VFPv3 floating point (optional), NEON SIMD (optional)
In-order, 9-stage single-decode pipeline
0.53-1 1-4
Cortex-A7 ARM v7, Jazelle, Thumb-2 DSP, VFPv4 floating point, NEON SIMD, virtualization
In-order, 8-stage limited dual-decode pipeline
> 1 1-4
Cortex-A8 ARM v7, Thumb-2 VFPv3 floating point, NEON SIMD 0.53-1 1 Apple A4, Freescale I.MX5?, Samsung Hummingbird, TI OMAP3
In-order, 13-stage dual-decode pipeline
Cortex-A9 ARM v7, Jazelle, Thumb-2 DSP, VFPv3 floating point (optional), NEON SIMD (optional) 0.8-2 1-4 Apple A5/A6, nVidia Tegra 2, Samsung Exynos 4210, ST-Ericsson NovaThor U8500, TI OMAP4
Out-of-order, 9-12 stage dual-decode pipeline
Cortex-A15 ARM v7, Jazelle, Thumb-2 TrustZone, VFPv4 floating point, 40-bit Large Physical Address Extensions, Virtualization, L2 cache
Out-of-order, 15+ stages triple-decode pipeline
1-2.5 1-32 ST-Ericsson Nova A9600, TI OMAP5, Samsung Exynos5

ASCII codes

American Standard Code for Information Interchange (ASCII) codes

Dec Hex  Oct  Meaning

  0 0x00 \000  NUL Null character, Ctrl-@
  1 0x01 \001  SOH Start of Header, Ctrl-A
  2 0x02 \002  STX Start of Text, Ctrl-B
  3 0x03 \003  ETX End of Text, Ctrl-C
  4 0x04 \004  EOT End of Transmission, End-of-file, Ctrl-D
  5 0x05 \005  ENQ Enquiry, Ctrl-E
  6 0x06 \006  ACK Acknowledgment, Ctrl-F
  7 0x07 \007  BEL Bell, Ctrl-G, '\a'
  8 0x08 \010   BS Backspace, Ctrl-H, '\b'
  9 0x09 \011   HT Horizontal Tab, Ctrl-I, '\t'
 10 0x0A \012   LF Line Feed, Ctrl-J, '\n'
 11 0x0B \013   VT Vertical Tab, Ctrl-K, '\v'
 12 0x0C \014   FF Form Feed, Ctrl-L, '\f'
 13 0x0D \015   CR Carriage Return, Ctrl-M, '\r'
 14 0x0E \016   SO Shift Out, Ctrl-N
 15 0x0F \017   SI Shift In, Ctrl-O
 16 0x10 \020  DLE Data Link Escape, Ctrl-P
 17 0x11 \021  DC1 XON/Device Control 1, Ctrl-Q
 18 0x12 \022  DC2 Device Control 2
 19 0x13 \023  DC3 XOFF/Device Control 3, Ctrl-S
 20 0x14 \024  DC4 Device Control 4
 21 0x15 \025  NAK Negative Acknowledgement
 22 0x16 \026  SYN Synchronous Idle
 23 0x17 \027  ETB End of Transmission Block
 24 0x18 \030  CAN Cancel, Ctrl-X
 25 0x19 \031   EM End of Medium
 26 0x1A \032  SUB Substitute
 27 0x1B \033  ESC Escape, Ctrl-[, '\e'
 28 0x1C \034   FS File Separator, Ctrl+\
 29 0x1D \035   GS Group Separator
 30 0x1E \036   RS Record Separator
 31 0x1F \037   US Unit Separator
 32 0x20 \040   SP Space
 33 0x21 \041    ! Exclamation mark
 34 0x22 \042    " Double quote
 35 0x23 \043    # Number sign, pound, hash
 36 0x24 \044    $ Dollar sign
 37 0x25 \045    % Percent
 38 0x26 \046    & Ampersand
 39 0x27 \047    ' Single quote, apostrophe, prime
 40 0x28 \050    ( Left/opening parenthesis
 41 0x29 \051    ) Right/closing parenthesis
 42 0x2A \052    * Asterisk
 43 0x2B \053    + Plus
 44 0x2C \054    , Comma
 45 0x2D \055    - Minus, dash
 46 0x2E \056    . Dot, period, full stop
 47 0x2F \057    / Forward slash
 48 0x30 \060    0
 49 0x31 \061    1
 50 0x32 \062    2
 51 0x33 \063    3
 52 0x34 \064    4
 53 0x35 \065    5
 54 0x36 \066    6
 55 0x37 \067    7
 56 0x38 \070    8
 57 0x39 \071    9
 58 0x3A \072    : Colon
 59 0x3B \073    ; Semi-colon
 60 0x3C \074    < Less than
 61 0x3D \075    = Equal sign
 62 0x3E \076    > Greater than
 63 0x3F \077    ? Question mark
 64 0x40 \100    @ At sign
 65 0x41 \101    A
 66 0x42 \102    B
 67 0x43 \103    C
 68 0x44 \104    D
 69 0x45 \105    E
 70 0x46 \106    F
 71 0x47 \107    G
 72 0x48 \110    H
 73 0x49 \111    I
 74 0x4A \112    J
 75 0x4B \113    K
 76 0x4C \114    L
 77 0x4D \115    M
 78 0x4E \116    N
 79 0x4F \117    O
 80 0x50 \120    P
 81 0x51 \121    Q
 82 0x52 \122    R
 83 0x53 \123    S
 84 0x54 \124    T
 85 0x55 \125    U
 86 0x56 \126    V
 87 0x57 \127    W
 88 0x58 \130    X
 89 0x59 \131    Y
 90 0x5A \132    Z
 91 0x5B \133    [ Left/opening bracket, square brackets
 92 0x5C \134    \ Back slash
 93 0x5D \135    ] Right/closing bracket, square brackets
 94 0x5E \136    ^ Caret, circumflex
 95 0x5F \137    _ Underscore
 96 0x60 \140    ` Grave accent
 97 0x61 \141    a
 98 0x62 \142    b
 99 0x63 \143    c
100 0x64 \144    d
101 0x65 \145    e
102 0x66 \146    f
103 0x67 \147    g
104 0x68 \150    h
105 0x69 \151    i
106 0x6A \152    j
107 0x6B \153    k
108 0x6C \154    l
109 0x6D \155    m
110 0x6E \156    n
111 0x6F \157    o
112 0x70 \160    p
113 0x71 \161    q
114 0x72 \162    r
115 0x73 \163    s
116 0x74 \164    t
117 0x75 \165    u
118 0x76 \166    v
119 0x77 \167    w
120 0x78 \170    x
121 0x79 \171    y
122 0x7A \172    z
123 0x7B \173    { Left/opening brace, curly brackets
124 0x7C \174    | Vertical bar, pipe
125 0x7D \175    } Right/closing brace, curly brackets
126 0x7E \176    ~ Tilde
127 0x7F \177  DEL Delete

(The following are valid in HTML 4 ISO 8859-1 encoding)

160 0xA0 \240    Non-breaking space (&nbsp;)
161 0xA1 \241  ¡ Inverted exclamation mark
162 0xA2 \242  ¢ Cent sign
163 0xA3 \243  £ Pound sign
164 0xA4 \244  ¤ Currency sign
165 0xA5 \245  ¥ Yen sign
166 0xA6 \246  ¦ Broken vertical bar
167 0xA7 \247  § Section sign (&sect;)
168 0xA8 \250  ¨ Spacing diaeresis, umlaut
169 0xA9 \251  © Copyright sign (&copy;)
170 0xAA \252  ª Feminine ordinal indicator
171 0xAB \253  « Left double angle quotes (&laquo;)
172 0xAC \254  ¬ Logical NOT sign (&not;)
173 0xAD \255  ­  Soft hyphen
174 0xAE \256  ® Registered trade mark sign (&reg;)
175 0xAF \257  ¯ Spacing macron, overline
176 0xB0 \260  ° Degree sign (&deg;)
177 0xB1 \261  ± Plus-or-minus sign (&plusmn;)
178 0xB2 \262  ² Superscript two, squared (&sup2;)
179 0xB3 \263  ³ Superscript three, cubed (&sup3;)
180 0xB4 \264  ´ Acute accent, spacing acute
181 0xB5 \265  µ Micro sign (&micro;)
182 0xB6 \266  ¶ Pilcrow sign, paragraph sign
183 0xB7 \267  · Middle dot, Georgian comma
184 0xB8 \270  ¸ Spacing cedilla
185 0xB9 \271  ¹ Superscript one (&sup1;)
186 0xBA \272  º Masculine ordinal indicator
187 0xBB \273  » Right double angle quotes (&raquo;)
188 0xBC \274  ¼ Fraction one quarter (&frac14;)
189 0xBD \275  ½ Fraction one half (&frac12;)
190 0xBE \276  ¾ Fraction three quarters (&frac34;)
191 0xBF \277  ¿ Inverted question mark

Lock/mutex primitives in multi-threading libraries

Locking/Mutex primitives in Pthreads

Spin locks

pthread_spin_{init|lock|trylock|unlock|destroy}

On x86_64, pthread_spin_lock is implemented by lock decl followed by pause. pthread_spin_trylock is implemented by lock cmpxchg followed by cmove (move if equal)

Mutexes

pthread_mutex_{init|lock|timedlock|trylock|unlock|destroy}
pthread_mutex_{get|set}prioceiling

The implementation of Mutex in Glibc changes from version to version. The latest version uses low-level locks (in Glibc source tree, nptl/sysdeps/unix/sysv/linux/x86_64/lowlevelock.h) in conjunction with the Fast Userspace Locking system call futex.

POSIX requires that a mutex to have an attribute called robustness (see here). If thread owning a robust mutex terminates while holding the mutex, the next thread that acquires the mutex shall be notified about the termination by the erroneous return value EOWNERDEAD. What the notified thread can do is either

  • Call pthread_mutex_consistent to make the mutex usable again, and then call pthread_mutex_unlock
  • Call pthread_mutex_unlock directly, then the mutex would end up in a permanently unusable state and all attempts to lock it again shall fail with the error ENOTRECOVERABLE. The unusbale mutex can only be removed by pthread_mutex_destroy call

POSIX also requires that a mutex to have an attribute called protocol (see here). By default, the protocol is None. If it is not None, it can be:

  • Priority Inheritance (PI): The owner of a PI mutex inherits the priority of the highest priority thread waiting for it until it releases it. Use the feature test macro _POSIX_THREAD_PRIO_INHERIT in unistd.h to see if this is supported.
  • Priority Protection (PP): A PP mutex is the one with a priority ceiling (pthread_mutex_{get|set}prioceiling). The owner of a PP mutex will have the highest priority of the priority ceilings of all the PP mutexes it currently owns, regardless of any other thread waiting for the mutex. Use the feature test macro _POSIX_THREAD_PRIO_PROTECT in unistd.h to see if this is supported.

Read/write locks

pthread_rwlock_{init|destroy}
pthread_rwlock_{rdlock|timedrdlock|tryrdlock}
pthread_rwlock_{wrlock|timedwrlock|trywrlock}

Conditional variables

pthread_cond_{init|wait|signal|broadcast|destroy}

Non-standard Pthread call

pthread_yield

Locking/Mutex primitives in GNU OpenMP

GCC supports OpenMP since version 4.x.0, and it uses GNU OpenMP. Details of OpenMP and automatic parallelization in GCC can be found in Diego Novillo's paper and slides

Mutexes

gomp_mutex_{lock|unlock} and gomp_ptrlock_{get|set}, which are inline functions.

gomp_mutex_* are for integers, while gomp_ptrlock_* work for pointers.

Depending on how libgomp is built (when running ./configure script, whether --enable-linux-futex, which is default, or --disable-linux-futex), gomp_mutex_{lock|unlock} will either use its own implementation (as in libgomp/config/linux/mutex.h), or use pthread_mutex_{lock|unlock} (as in libgomp/config/linux/posix.h)

In the former case, gomp_mutex_{lock|unlock} will do a quick test using the GCC built-in __sync_lock_test_and_set function, and if it fails, it falls through to gomp_mutex_{lock|unlock}_slow.

gomp_mutex_lock_slow will spin a couple million or billion times (can be controlled by the environmental variable OMP_WAIT_POLICY and GOMP_SPINCOUNT, and this spin count also depends on whether there is any oversubscription, i.e. more threads than the number of CPUs) before using the Fast Userspace Locking system call futex.

gomp_mutex_unlock_slow will simply call futex with opcode being FUTEX_WAKE.

gomp_ptrlock_{get|set}_slow are similar.

Semaphores

gomp_sem_{wait|post}, which are inline functions.

As in previous case, there are two implementations. The POSIX version uses pthread_mutex_{lock|unlock} and pthread_cond_{wait|signal}

The Linux version is very much like mutexes: gomp_sem_{wait|post} will do a quick test using the built-in __sync_bool_compare_and_swap function, and if it fails, it falls through to gomp_sem_{wait|post}_slow.

Semaphores here are used to implement Barriers.

Locking/Mutex primitives in Intel OpenMP

There are certain functions which seem related to locks (the intiail "k" refers to KAI, Kuck & Associates, Inc, which was a high-performance compiler vendor and was acquired by Intel in 2000.)
__kmp_acquire_lock
__kmp_init_lock
__kmp_wait_yield_4
__kmp_x86_pause
__kmp_acquire_ticket_lock
__kmp_query_cpuid
and some related to profiling:
kmp_gvs_event
kmp_send_omp_collector_event
Also note that the global variable __kmp_lock_method will be set to 1 if HyperThreading is available, and 2 otherwise.

Binary File Descriptor library (BFD)

Useful references

BFD manual

objdump man page

Executable and Linkable Format (ELF)

Microsoft Portable Executable and Common Object File Format Specification

Common Object File Format (COFF)

An In-Depth Look into the Win32 Portable Executable File Format

GNU Binutils and BFD

Binary File Descriptor library supports manipulation of object files in a variety of formats ("targets").

What "targets" and "architectures" do my installed GNU Binutils support ?

Use
objdump -H
and one should see something like:
  ....

objdump: supported targets: elf64-x86-64 elf32-i386 a.out-i386-linux pei-i386 pei-x86-64 elf64-l1om elf64-little elf64-big elf32-little elf32-big srec symbolsrec verilog tekhex binary ihex
objdump: supported architectures: i386 i386:x86-64 i8086 i386:intel i386:x86-64:intel l1om l1om:intel
  ....
"Targets" are really just file formats and can be used in conjuction with objdump's -b option, and "architectures" are for disassembly and can be used in conjuction with objdump's -m option.

One can also use

objdump -i
to see more information about supported "targets".

How to make GNU Binutils support COFF/PEi ?

By default, if GNU Binutils is compiled on x86_64, then i386 & x86_64 PEi (PEi is Microsoft's extension to the PE format, as mentioned in the comments in
bfd/peicode.h) are supported automatically (at least for GNU Binutils version 2.21). However, COFF is not supported by default.

i386 COFF can be enabled by running GNU Binutils' configure script with --enable-targets=i386-foobar-coff

For x86_64 COFF, as of GNU Binutils version 2.21, there is no switch to enable it, unless one modifies bfd/config.bfd: At the bottom of this file, add the following line:

targ_selvecs="${targ_selvecs} i386coff_vec x86_64coff_vec"

To see all available targets which can be used in bfd/config.bfd, do

    grep -o '[a-zA-Z0-9_-]*_vec' bfd/configure.in | grep '^[a-zA-Z0-9]'
or
    grep 'extern const bfd_target' bfd/targets.c | grep -o '[a-zA-Z0-9_-]*_vec'

How to make GNU Binutils support different architectures ?

For example, to make GNU Binutils support Itanium, at the bottom of bfd/config.bfd, add the following line:
targ_archs="${targ_archs} bfd_ia64_arch"

To see all available architectures which can be used in bfd/config.bfd, do

    grep -o 'bfd_[a-zA-Z0-9-]*_arch' opcodes/configure.in
Putting all together, for example, to make GNU Binutils support i386/x86_64, Itanium, and PowerPC, add the following code to the bottom of bfd/config.bfd:
\grep -o '[a-zA-Z0-9_-]*_vec' configure.in | grep '^[a-zA-Z0-9]' | sort > $$.t1
\grep 'extern const bfd_target' targets.c | grep -o '[a-zA-Z0-9_-]*_vec' | sort > $$.t2

for t in `comm -1 -2 $$.t1 $$.t2`
do
    case "$t" in
        *386*|*86*64*|*ia64*|*power*|*rs6000*|mach*)  
            targ_selvecs="${targ_selvecs} $t"
            ;;
    esac
done

\rm -f $$.t?

targ_archs="${targ_archs} bfd_i386_arch"
targ_archs="${targ_archs} bfd_ia64_arch"
targ_archs="${targ_archs} bfd_rs6000_arch bfd_powerpc_arch"
and run the configure script with --enable-gold=no --enable-ld=no options

Executable and Linkable Format (ELF)

Acronyms relevant to Executable and Linkable Format (ELF)

ABIApplication binary interface
a.outAssembler output file format
BSSBlock started by symbol. The uninitialized data segment containing statically-allocated variables.
COFFCommon object file format
DTVDynamic thread vector (for TLS)
DWARFA standardized debugging data format
GDGlobal Dynamic (dynamic TLS) One of the Thread-Local Storage access models.
GOTGlobal offset table
IEInitial Executable (static TLS with assigned offsets) One of the Thread-Local Storage access models.
LDLocal Dynamic (dynamic TLS of local symbols) One of the Thread-Local Storage access models.
LELocal Executable (static TLS) One of the Thread-Local Storage access models.
Mach-OMach object file format
PCProgram counter. On x86, this is the same as IP (Instruction Pointer) register.
PEPortable executable
PHTProgram header table
PICPosition independent code
PIEPosition independent executable
PLTProcedure linkage table
REL
RELA
Relocation
RVARelative virtual address
SHFSection header flag
SHTSection header table
SOShared object (another name for dynamic link library)
VMAVirtual memory area/address

Useful books and references

ELF man page

System V Application Binary Interface

AMD64 System V Application Binary Interface

The gen on function calling conventions

Section II of Linux Standard Base 4.0 Core Specification

Self-Service Linux: Mastering the Art of Problem Determination by Mark Wilding and Dan Behman

Solaris Linker and Libraries Guide

Linkers and Loaders by John Levine

Understanding Linux ELF RTLD internals by mayhem (this article gives you an idea how the runtime linker ld.so works)

ld.so man page

Prelink by Jakub Jelinek (and prelink man page)

Executable and Linkable Format

An ELF executable binary contains at least two kinds of headers: ELF file header (see struct Elf32_Ehdr/struct Elf64_Ehdr in /usr/include/elf.h) and one or more Program Headers (see struct Elf32_Phdr/struct Elf64_Phdr in /usr/include/elf.h)

Usually there is another kind of header called Section Header, which describe attributes of an ELF section (e.g. .text, .data, .bss, etc) The Section Header is described by struct Elf32_Shdr/struct Elf64_Shdr in /usr/include/elf.h

The Program Headers are used during execution (ELF's "execution view"); it tells the kernel or the runtime linker ld.so what to load into memory and how to find dynamic linking information.

The Section Headers are used during compile-time linking (ELF's "linking view"); it tells the link editor ld how to resolve symbols, and how to group similar byte streams from different ELF binary objects.

Conceptually, the two ELF's "views" are as follows (borrowed from Shaun Clowes's Fixing/Making Holes in Binaries slides):

              +-----------------+
         +----| ELF File Header |----+
         |    +-----------------+    |
         v                           v
 +-----------------+      +-----------------+
 | Program Headers |      | Section Headers |
 +-----------------+      +-----------------+
      ||                               ||
      ||                               ||
      ||                               ||
      ||   +------------------------+  ||
      +--> | Contents (Byte Stream) |<--+
           +------------------------+

In reality, the layout of a typical ELF executable binary on a disk file is like this:

    +-------------------------------+
    | ELF File Header               |
    +-------------------------------+
    | Program Header for segment #1 |
    +-------------------------------+
    | Program Header for segment #2 |
    +-------------------------------+
    | ...                           |
    +-------------------------------+
    | Contents (Byte Stream)        |
    | ...                           |
    +-------------------------------+
    | Section Header for section #1 |
    +-------------------------------+
    | Section Header for section #2 |
    +-------------------------------+
    | ...                           |
    +-------------------------------+
    | ".shstrtab" section           |
    +-------------------------------+
    | ".symtab"   section           |
    +-------------------------------+
    | ".strtab"   section           |
    +-------------------------------+
The ELF File Header contains the file offsets of the first Program Header, the first Section Header, and .shstrtab section which contains the section names (a series of NULL-terminated strings)

The ELF File Header also contains the number of Program Headers and the number of Section Headers.

Each Program Header describes a "segment": It contains the permissions (Readable, Writeable, or Executable) , offset of the "segment" (which is just a byte stream) into the file, and the size of the "segment". The following table shows the purposes of special segments. Some information can be found in GNU Binutil's source file include/elf/common.h:

ELF Segment Purpose
DYNAMIC For dynamic binaries, this segment hold dynamic linking information and is usually the same as .dynamic section in ELF's linking view. See paragraph below.
GNU_EH_FRAME Frame unwind information (EH = Exception Handling). This segment is usually the same as .eh_frame_hdr section in ELF's linking view.
GNU_RELRO This segment indicates the memory region which should be made Read-Only after relocation is done. This segment usually appears in a dynamic link library and it contains .ctors, .dtors, .dynamic, .got sections. See paragraph below.
GNU_STACK The permission flag of this segment indicates whether the stack is executable or not. This segment does not have any content; it is just an indicator.
INTERP For dynamic binaries, this holds the full pathname of runtime linker ld.so

This segement is the same as .interp section in ELF's linking view.

LOAD Loadable program segment. Only segments of this type are loaded into memory during execution.
NOTE Auxiliary information.

For core dumps, this segment contains the status of the process (when the core dump is created), such as the signal (the process received and caused it to dump core), pending & held signals, process ID, parent process ID, user ID, nice value, cumulative user & system time, values of registers (including the program counter!)

For more info, see struct elf_prstatus and struct elf_prpsinfo in Linux kernel source file include/linux/elfcore.h and struct user_regs_struct in arch/x86/include/asm/user_64.h

TLS Thread-Local Storage

Likewise, each Section Header contains the file offset of its corresponding "content" and the size of the "content". The following table shows the purposes of some special sections. Most information here comes from LSB specification. Some information can be found in GNU Binutil's source file bfd/elf.c (look for bfd_elf_special_section) and bfd/elflink.c (look for double-quoted section names such as ".got.plt")

ELF Section Purpose
.bss Uninitialized global data ("Block Started by Symbol").

Depending on the compilers, uninitialized global variables could be stored in a nameness section called COMMON (named after Fortran 77's "common blocks".) To wit, consider the following code:

    int globalVar;
    static int globalStaticVar;
    void dummy() {
       static int localStaticVar;
    }
    
Compile with gcc -c, then on x86_64, the resulting object file has the following structure:
    $ objdump -t foo.o

    SYMBOL TABLE:
     ....
    0000000000000000 l     O .bss   0000000000000004 globalStaticVar
    0000000000000004 l     O .bss   0000000000000004 localStaticVar.1619
     ....
    0000000000000004       O *COM*  0000000000000004 globalVar
    
so only the file-scope and local-scope global variables are in the .bss section.

If one wants globalVar to reside in the .bss section, use the -fno-common compiler command-line option. Using -fno-common is encouraged, as the following example shows:

    $ cat foo.c
    int globalVar;
    $ cat bar.c
    double globalVar;
    int main(){}
    $ gcc foo.c bar.c
    
Not only there is no error message about redefinition of the same symbol in both source files (notice we did not use the extern keyword here), there is no complaint about their different data types and sizes either. However, if one uses -fno-common, the compiler will complain:
    /tmp/ccM71JR7.o:(.bss+0x0): multiple definition of `globalVar'
    /tmp/ccIbS5MO.o:(.bss+0x0): first defined here
    ld: Warning: size of symbol `globalVar' changed from 8 in /tmp/ccIbS5MO.o to 4 in /tmp/ccM71JR7.o
    
.comment A series of NULL-terminated strings containing compiler information.
.ctors Pointers to functions which are marked as __attribute__ ((constructor)) as well as static C++ objects' constructors. They will be used by __libc_global_ctors function.

See paragraphs below.

.data Initialized data.
.data.rel.ro Similar to .data section, but this section should be made Read-Only after relocation is done.
.debug_XXX Debugging information (for the programs which are compiled with -g option) which is in the DWARF 2.0 format.

See here for DWARF debugging format.

.dtors Pointers to functions which are marked as __attribute__ ((destructor)) as well as static C++ objects' destructors.

See paragraphs below.

.dynamic For dynamic binaries, this section holds dynamic linking information used by ld.so. See paragraphs below.
.dynstr NULL-terminated strings of names of symbols in .dynsym section.

One can use commands such as readelf -p .dynstr a.out to see these strings.

.dynsym Runtime/Dynamic symbol table. For dynamic binaries, this section is the symbol table of globally visible symbols. For example, if a dynamic link library wants to export its symbols, these symbols will be stored here. On the other hand, if a dynamic executable binary uses symbols from a dynamic link library, then these symbols are stored here too.

The symbol names (as NULL-terminated strings) are stored in .dynstr section.

.eh_frame
.eh_frame_hdr
Frame unwind information (EH = Exception Handling).

See here for details.

To see the content of .eh_frame section, use

readelf --debug-dump=frames-interp a.out
.fini Code which will be executed when program exits normally. See paragraphs below.
.fini_array Pointers to functions which will be executed when program exits normally. See paragraphs below.
.GCC.command.line A series of NULL-terminated strings containing GCC command-line (that is used to compile the code) options.

This feature is supported since GCC 4.5 and the program must be compiled with -frecord-gcc-switches option.

.gnu.hash GNU's extension to hash table for symbols.

See here for its structure and the hash algorithm.

The link editor ld calls bfd_elf_gnu_hash in in GNU Binutil's source file bfd/elf.c to compute the hash value.

The runtime linker ld.so calls do_lookup_x in elf/dl-lookup.c to do the symbol look-up. The hash computing function here is dl_new_hash.

.gnu.linkonceXXX GNU's extension. It means only a single copy of the section will be used in linking. This is used to by g++. g++ will emit each template expansion in its own section. The symbols will be defined as weak, so that multiple definitions are permitted.
.gnu.version Versions of symbols.

See here, here, here, and here for details of symbol versioning.

.gnu.version_d Version definitions of symbols.
.gnu.version_r Version references (version needs) of symbols.
.got For dynamic binaries, this Global Offset Table holds the addresses of variables which are relocated upon loading. See paragraphs below.
.got.plt For dynamic binaries, this Global Offset Table holds the addresses of functions in dynamic libraries. They are used by trampoline code in .plt section. If .got.plt section is present, it contains at least three entries, which have special meanings. See paragraphs below.
.hash Hash table for symbols.

See here for its structure and the hash algorithm.

The link editor ld calls bfd_elf_hash in in GNU Binutil's source file bfd/elf.c to compute the hash value.

The runtime linker ld.so calls do_lookup_x in elf/dl-lookup.c to do the symbol look-up. The hash computing function here is _dl_elf_hash.

.init Code which will be executed when program initializes. See paragraphs below.
.init_array Pointers to functions which will be executed when program starts. See paragraphs below.
.interp For dynamic binaries, this holds the full pathname of runtime linker ld.so
.jcr Java class registration information.

Like .ctors section, it contains a list of addresses which will be used by _Jv_RegisterClasses function in CRT (C Runtime) startup files (see gcc/crtstuff.c in GCC's source tree)

.note.ABI-tag This Linux-specific section is structured as a note section in ELF specification. Its content is mandated here.
.note.gnu.build-id A unique build ID. See here and here
.note.GNU-stack See here
.nvFatBinSegment This segment contains information of nVidia's CUDA fat binary container. Its format is described by struct __cudaFatCudaBinaryRec in __cudaFatFormat.h
.plt For dynamic binaries, this Procedure Linkage Table holds the trampoline/linkage code. See paragraphs below.
.preinit_array Similar to .init_array section. See paragraphs below.
.rela.dyn Runtime/Dynamic relocation table.

For dynamic binaries, this relocation table holds information of variables which must be relocated upon loading. Each entry in this table is a struct Elf64_Rela (see /usr/include/elf.h) which has only three members:

  • offset (the variable's [usually position-independent] virtual memory address which holds the "patched" value during the relocation process)
  • info (Index into .dynsym section and Relocation Type)
  • addend
See paragraphs below for details about runtime relocation.
.rela.plt Runtime/Dynamic relocation table.

This relocation table is similar to the one in .rela.dyn section; the difference is this one is for functions, not variables.

The relocation type of entries in this table is R_386_JMP_SLOT or R_X86_64_JUMP_SLOT and the "offset" refers to memory addresses which are inside .got.plt section.

Simply put, this table holds information to relocate entries in .got.plt section.

.rel.text
.rela.text
Compile-time/Static relocation table.

For programs compiled with -c option, this section provides information to the link editor ld where and how to "patch" executable code in .text section.

The difference between .rel.text and .rela.text is entries in the former does not have addend member. (Compare struct Elf64_Rel with struct Elf64_Rela in /usr/include/elf.h) Instead, the addend is taken from the memory location described by offset member.

Whether to use .rel or .rela is platform-dependent. For x86_32, it is .rel and for x86_64, .rela

.rel.XXX
.rela.XXX
Compile-time/Static relocation table for other sections. For example, .rela.init_array is the relocation table for .init_array section.
.rodata Read-only data.
.shstrtab NULL-terminated strings of section names.

One can use commands such as readelf -p .shstrtab a.out to see these strings.

.strtab NULL-terminated strings of names of symbols in .symtab section.

One can use commands such as readelf -p .strtab a.out to see these strings.

.symtab Compile-time/Static symbol table.

This is the main symbol table used in compile-time linking or runtime debugging.

The symbol names (as NULL-terminated strings) are stored in .strtab section.

Both .symtab and .symtab can be stripped away by the strip command.

.tbss Similar to .bss section, but for Thread-Local data. See paragraphs below.
.tdata Similar to .data section, but for Thread-Local data. See paragraphs below.
.text User's executable code

How is an executable binary in Linux being executed ?

First, the operating system must recognize executable binaries. For example, zcat /proc/config.gz | grep CONFIG_BINFMT_ELF can show whether the Linux kernel is compiled to support ELF executable binary format (if /proc/config.gz does not exist, try /lib/modules/`uname -r`/build/.config)

When the shell makes an execvc system call to run an executable binary, the Linux kernel responds as follows (see here and here for more details) in sequence:

  1. sys_execve function (in arch/x86/kernel/process.c) handles the execvc system call from user space. It calls do_execve function.
  2. do_execve function (in fs/exec.c) opens the executable binary file and does some preparation. It calls search_binary_handler function.
  3. search_binary_handler function (in fs/exec.c) finds out the type of executable binary and calls the corresponding handler, which in our case, is load_elf_binary function.
  4. load_elf_binary (in fs/binfmt_elf.c) loads the user's executable binary file into memory. It allocates memory segments and zeros out the BSS section by calling the padzero function.

    load_elf_binary also examines whether the user's executable binary contains an INTERP segment or not.

  5. If the executable binary is dynamically linked, then the compiler will usually creates an INTERP segment (which is usually the same as .interp section in ELF's "linking view"), which contains the full pathname of an "interpreter", usually is the Glibc runtime linker ld.so.

    To see this, use command readelf -p .interp a.out

    According to AMD64 System V Application Binary Interface, the only valid interpreter for programs conforming to AMD64 ABI is /lib/ld64.so.1 and on Linux, GCC usually uses /lib64/ld-linux-x86-64.so.2 or /lib/ld-linux-x86-64.so.2 instead:

    $ gcc -dumpspecs
    ....
    
    *link:
    ...
      %{!m32:%{!dynamic-linker:-dynamic-linker %{muclibc:%{mglibc:%e-mglibc and -muclibc used
    together}/lib/ld64-uClibc.so.0;:/lib/ld-linux-x86-64.so.2}}}}
    ...
    

    To change the runtime linker, compile the program using something like

    gcc foo.c -Wl,-I/my/own/ld.so

    The System V Application Binary Interface specifies, the operating system, instead of running the user's executable binary, should run this "interpreter". This interpreter should complete the binding of user's executable binary to its dependencies.

  6. Thus, if the ELF executable binary file contains an INTERP segment, load_elf_binary will call load_elf_interp function to load the image of this interpreter as well.
  7. Finally, load_elf_binary calls start_thread (in arch/x86/kernel/process_64.c) and passes control to either the interpreter or the user program.

What about ld.so ?

ld.so is the runtime linker/loader (the compile-time linker ld is formally called "link editor") for dynamic executables. It provides the following services:
  • Analyzes the user's executable binary's DYNAMIC segment and determines what dependencies are required. (See below)
  • Locates and loads these dependencies, analyzes their DYNAMIC segments to determine if more dependencies are required.
  • Performs any necessary relocations to bind these objects.
  • Calls any initialization functions (see below) provided by these dependencies.
  • Passes control to user's executable binary.

Compile your own ld.so

The internal working of ld.so is complex, so you might want to compile and experiment your own ld.so. The source code of ld.so can be found in Glibc. The main files are elf/rtld.c, elf/dl-reloc.c, and sysdeps/x86_64/dl-machine.h.

This link provides general tips for building Glibc. Glibc's own INSTALL and FAQ documents are useful too.

To compile Glibc (ld.so cannot be compiled independently) download and unpack Glibc source tarball.

  • Make sure the version of Glibc you downloaded is the same as the system's current one.
  • Make sure the environmental variable LD_RUN_PATH is not set.
  • Read the INSTALL and make sure all necessary tool chains (Make, Binutils, etc) are up-to-date.
  • Make sure the file system you are doing the compilation is case sensitive, or you will see weird errors like
    /scratch/elf/librtld.os: In function `process_envvars':
    /tmp/glibc-2.x.y/elf/rtld.c:2718: undefined reference to `__open'
    ...
    
  • ld.so should be compiled with the optimization flag on (-O2 is the default). Failing to do so will end up with weird errors (see Question 1.23 in FAQ)
  • Suppose Glibc is unpacked at
    /tmp/glibc-2.x.y/
    Then edit /tmp/glibc-2.x.y/Makefile.in: Un-comment the line
    # PARALLELMFLAGS = -j 4
    and change 4 to an appropriate number.

  • Since we are only interested in ld.so and not the whole Glibc, we only want to build the essential source files needed by ld.so. To do so, edit /tmp/glibc-2.x.y/Makeconfig: Find the line started with
    all-subdirs = csu assert ctype locale intl catgets math setjmp signal \
      ...
    
    and change it to
    all-subdirs = csu elf gmon io misc posix setjmp signal stdlib string time
    
  • Find a scratch directory, say /scratch. Then
    $ cd /scratch
    $ /tmp/glibc-2.x.y/configure --prefix=/scratch --disable-profile
    $ gmake
    
  • Since we are not building the entire Glibc, when the gmake stops (probably with some errors), check if /scratch/elf/ld.so exists or not.
  • ld.so is a static binary, which means it has its own implementation of standard C routines (e.g. memcpy, strcmp, etc) It has its own printf-like routine called _dl_debug_printf.

    _dl_debug_printf is not the full-blown printf and has very limited capabilities. For example, to print the address, one would need to use

    _dl_debug_printf("0x%0*lx\n", (int)sizeof (void*)*2, &foo);
    

How does ld.so work ?

ld.so, by its nature, cannot be a dynamic executable itself. The entry point of ld.so is _start defined in the macro RTLD_START (in sysdeps/x86_64/dl-machine.h). _start is placed at the beginning of .text section, and the default ld script specifies "Entry point address" (in ELF header, use readelf -h ld.so|grep Entry command to see) to be the address of _start (use ld -verbose | grep ENTRY command to see). One can set the entry point to a different address at compile time by -e option) so ld.so is executed from here. The very first thing it does is to call _dl_start of elf/rtld.c. To see this, run gdb on some ELF executable binary, and do
(gdb) break _dl_start
Function "_dl_start" not defined.
Make breakpoint pending on future shared library load? (y or [n]) y
Breakpoint 1 (_dl_start) pending.
(gdb) run
Starting program: a.out

Breakpoint 1, 0x0000003433e00fa0 in _dl_start () from /lib64/ld-linux-x86-64.so.2
(gdb) bt
#0  0x0000003433e00fa0 in _dl_start () from /lib64/ld-linux-x86-64.so.2
#1  0x0000003433e00a78 in _start () from /lib64/ld-linux-x86-64.so.2
#2  0x0000000000000001 in ?? ()
#3  0x00007fffffffe4f2 in ?? ()
#4  0x0000000000000000 in ?? ()
...
(gdb) x/10i $pc
   0x3433e00a70 <_start>:       mov    %rsp,%rdi
   0x3433e00a73 <_start+3>:     callq  0x3433e00fa0 <_dl_start>
   0x3433e00a78 <_dl_start_user>:       mov    %rax,%r12
   0x3433e00a7b <_dl_start_user+3>:     mov    0x21b30b(%rip),%eax        # 0x343401bd8c <_dl_skip_args>
...
At this breakpoint, we can use pmap to see the memory map of a.out, which would look like this:
0000000000400000      8K r-x--  a.out
0000000000601000      4K rw---  a.out
0000003433e00000    112K r-x--  /lib64/ld-2.5.so
000000343401b000      8K rw---  /lib64/ld-2.5.so
00007ffffffea000     84K rw---    [ stack ]
ffffffffff600000   8192K -----    [ anon ]
 total             8408K
The memory segment of /lib64/ld-2.5.so indeed starts at 3433e00000 (page aligned) and this can be verified by running readelf -t /lib64/ld-2.5.so.

If we put another breakpoint at main and continue, then when it stops, the memory map would change to this:

0000000000400000      8K r-x--  a.out
0000000000601000      4K rw---  a.out
0000003433e00000    112K r-x--  /lib64/ld-2.5.so
000000343401b000      4K r----  /lib64/ld-2.5.so
000000343401c000      4K rw---  /lib64/ld-2.5.so
0000003434200000   1336K r-x--  /lib64/libc-2.5.so     <-- The first "LOAD" segment, which contains .text and .rodata sections
000000343434e000   2044K -----  /lib64/libc-2.5.so     <-- "Hole"
000000343454d000     16K r----  /lib64/libc-2.5.so     <-- Relocation (GNU_RELRO) info  -+---- The second "LOAD" segment
0000003434551000      4K rw---  /lib64/libc-2.5.so     <-- .got.plt .data sections      -+
0000003434552000     20K rw---    [ anon ]             <-- The remaining zero-filled sections (e.g. .bss)
0000003434e00000     88K r-x--  /lib64/libpthread-2.5.so     <-- The first "LOAD" segment, which contains .text and .rodata sections
0000003434e16000   2044K -----  /lib64/libpthread-2.5.so     <-- "Hole"
0000003435015000      4K r----  /lib64/libpthread-2.5.so     <-- Relocation (GNU_RELRO) info  -+---- The second "LOAD" segment
0000003435016000      4K rw---  /lib64/libpthread-2.5.so     <-- .got.plt .data sections      -+
0000003435017000     16K rw---    [ anon ]                   <-- The remaining zero-filled sections (e.g. .bss)
00002aaaaaaab000      4K rw---    [ anon ]
00002aaaaaac6000     12K rw---    [ anon ]
00007ffffffea000     84K rw---    [ stack ]
ffffffffff600000   8192K -----    [ anon ]
 total            14000K
Indeed, ld.so has brought in all the required dynamic libraries.

Note that there are two memory regions of 2044KB with null permissions. As mentioned earlier, the ELF's 'execution view' is concerned with how to load an executable binary into memory. When ld.so brings in the dynamic libraries, it looks at the segments labelled as LOAD (look at "Program Headers" and "Section to Segment mapping" from readelf -a xxx.so command.) Usually there are two LOAD segments, and there is a "hole" between the two segments (look at the VirtAddr and MemSiz of these two segments), so ld.so will make this hole inaccessible deliberately: Look for the PROT_NONE symbol in _dl_map_object_from_fd in elf/dl-load.c

Also note that each of libc-2.5.so and libpthread-2.5.so has a read-only memory region (at 0x343454d000 and 0x3435015000, respectively). This is a for security reasons. These are relocation info (run readelf -l xxx.so |grep GNU_RELRO command to see) which should be made Read-Only after the relocation is done and this achieved by _dl_protect_relro function in elf/dl-reloc.c. The GNU_RELRO segment is contained in the the second LOAD segment, which contains the following sections (look at "Program Headers" and "Section to Segment mapping" from readelf -l xxx.so command): .tdata, .fini_array, .ctors, .dtors, __libc_subfreeres, __libc_atexit, __libc_thread_subfreeres, .data.rel.ro, .dynamic, .got, .got.plt, .data, and .bss. Except for .got.plt, .data, and .bss, all sections in the the second LOAD segment are also in the GNU_RELRO segment, and they are thus made read-only.

The two [anon] memory segments at 0x3434552000 and 0x3435017000 are for sections which do not take space in the ELF binary files. For example, readelf -t xxx.so will show that .bss section has NOBITS flag, which means that section takes no disk space. When segments containing NOBITS sections are mapped into memory, ld.so allocates extra memory pages to accomodate these NOBITS sections. A LOAD segment is usually structured as a series of contiguous sections, and if a segment contains NOBITS sections, these NOBITS sections will be grouped together and placed at the tail of the segment.

So what does _dl_start do ?

  • Allocate the initial TLS block and initialize the Thread Pointer if needed (these are for ld.so's own, not for the user program)
  • Call _dl_sysdep_start, which will call dl_main
  • dl_main does the majority of the hard work, for example:

    It calls process_envvars to handle these LD_ prefix environmental variables such as LD_PRELOAD, LD_LIBRARY_PATH.

    It examines the NEEDED field(s) in the user executable binary's DYNAMIC segment section (see below) to determine the dependencies.

    It calls _dl_init_paths (in elf/dl-load.c) to initialize the dynamic libraries search paths. According to ld.so man page and this page, the dynamic libraries are searched in the following order:

    1. The RPATH in the DYNAMIC segment if there is no RUNPATH in the DYNAMIC segment.

      RPATH can be specified when the code is compiled with gcc -Wl,-rpath=...

      Use of RPATH is deprecated because it has an obvious drawback: There is no way to override it except using LD_PRELOAD environmental variable or removing it from the DYNAMIC segment.

      Both RPATH and RUNPATH can contain $ORIGIN (or equivalently ${ORIGIN}), which will be expanded to the value of environmental variable LD_ORIGIN_PATH or the full path of the loaded object (unless the programs use setuid or setgid)

    2. The LD_LIBRARY_PATH environmental variable (unless the programs use setuid or setgid)
    3. The RUNPATH in the DYNAMIC segment.
      RUNPATH can be specified when the code is compiled with gcc -Wl,-rpath=...,--enable-new-dtags
      One can use chrpath tool to manipulate RPATH and RUNPATH settings.
    4. /etc/ld.so.cache
    5. /lib
    6. /usr/lib

    It calls _dl_map_object_from_fd (in elf/dl-load.c) to load the dynamic libraries, sets up the right read/write/execute permissions for the memory segments, (within _dl_map_object_from_fd, look at calls to mmap, mprotect and symbols such as PROT_READ, PROT_WRITE, PROT_EXEC, PROT_NONE), zeroes out BSS sections of dynamic libraries (inside _dl_map_object_from_fd function, look at calls to memset), updates the link map, and performs relocations.

    It calls _dl_relocate_object (in elf/dl-reloc.c) to perform runtime relocations (see details below).

  • When _dl_start returns, it continues to execute code in _dl_start_user (see sysdeps/x86_64/dl-machine.h)
  • _dl_start_user will call _dl_init_internal, which will call call_init to invoke initialization function of each dynamic library loaded.

    Note that _dl_init_internal is defined in elf/dl-init.c as:

    void
    internal_function
    _dl_init (struct link_map *main_map, int argc, char **argv, char **env)
    
    call_init is also in elf/dl-init.c

  • The initialization function of a dynamic library, say libfoo.so, is located at the address marked with type "INIT" in the output of readelf -d libfoo.so For Glibc, its initialization function is named _init (not to be confused with the _init inside the user's executable binary) and its source code is in sysdeps/unix/sysv/linux/x86_64/init-first.c.

    _init will do the following things:

    • Save argc, argv, envp to hidden variables __libc_argc, __libc_argv, __environ
    • Call VDSO_SETUP to set up Virtaul Dynamic Shared Objects (see here) VDSO_SETUP is a platform-dependent macro. For x86_64, this macro is defined as _libc_vdso_platform_setup in sysdeps/unix/sysv/linux/x86_64/init-first.c
    • Call __init_misc (in misc/init-misc.c) which saves argv[0] to two global variables: program_invocation_name and program_invocation_short_name
    • Call __libc_global_ctors (in elf/soinit.c) to invoke each function listed in the .ctors section (see below).

      For x86_64, .ctors section contains only one function: init_cacheinfo

  • At the end of _dl_start_user, the control transfers to user program's entry point address (use readelf -h a.out|grep Entry to see) which is usually the initial address of .text section and contains the entry of a function named _start, and in the control transfer, the finalizer function _dl_fini is passed as an argument, and the stack frames are completely clobbered, as if the user program is run without any ld.so intervention. The latter is done by manipulating the stack (see the on-stack auxiliary vector adjustment code and HAVE_AUX_VECTOR in dl_main)

Here is the call graph, which is worth a thousand words

and see here on how it is generated.

To see ld.so in action, set the environmental variable LD_DEBUG to all and then run a user program.

The above debugging information does not show mmap and mprotect calls. However, we can use strace. If we run the user program again with

strace -e trace=mmap,mprotect,munmap,open a.out
we should see something like the following:
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x2ae62c0d1000

  .... (a lot of failed attempts to open 'libpthread.so.0' using LD_LIBRARY_PATH)

open("/etc/ld.so.cache", O_RDONLY) = 3
mmap(NULL, 104801, PROT_READ, MAP_PRIVATE, 3, 0) = 0x2ae62c0d2000
open("/lib64/libpthread.so.0", O_RDONLY) = 3
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x2ae62c0ec000
mmap(0x3434e00000, 2204528, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x3434e00000     <-- Bring in the first "LOAD" segment
mprotect(0x3434e16000, 2093056, PROT_NONE) = 0     <-- Make the "hole" inaccessible
mmap(0x3435015000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x15000) = 0x3435015000     <-- Bring in the second "LOAD" segment
mmap(0x3435017000, 13168, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x3435017000
       (note: 0x3435017000 is the [anon] part which follows immediately after libpthread-2.5.so)
  ...
  .... (a lot of failed attempts to open 'libc.so.6' using LD_LIBRARY_PATH)

open("/lib64/libc.so.6", O_RDONLY) = 3
mmap(0x3434200000, 3498328, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x3434200000     <-- Bring in the first "LOAD" segment
mprotect(0x343434e000, 2093056, PROT_NONE) = 0     <-- Make the "hole" inaccessible
mmap(0x343454d000, 20480, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x14d000) = 0x343454d000     <-- Bring in the second "LOAD" segment
mmap(0x3434552000, 16728, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x3434552000
       (note: 0x3434552000 is the [anon] part which follows immediately after libc-2.5.so)
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x2ae62c0ed000
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x2ae62c0ee000
mprotect(0x343454d000, 16384, PROT_READ) = 0    <-- Make the GNU_RELRO segment read-only
mprotect(0x3435015000, 4096, PROT_READ) = 0     <-- Make the GNU_RELRO segment read-only
mprotect(0x343401b000, 4096, PROT_READ) = 0
munmap(0x2ae62c0d2000, 104801)= 0
mmap(NULL, 10489856, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS|MAP_32BIT, -1, 0) = 0x40dc7000
mprotect(0x40dc7000, 4096, PROT_NONE)   = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x2aaaaaaab000

.plt section

This section contains trampolines for functions defined in dynamic libraries. A sample disassembly (run the command objdump -M intel -dj .plt a.out) will show the following:
4003c0 <printf@plt-0x10>:
  4003c0: push   QWORD PTR [RIP+0x2004d2]        # 600898 <_GLOBAL_OFFSET_TABLE_+0x8>
  4003c6: jmp    QWORD PTR [RIP+0x2004d4]        # 6008a0 <_GLOBAL_OFFSET_TABLE_+0x10>
  4003cc: nop    DWORD PTR [RAX+0x0]

4003d0 <printf@plt>:
  4003d0: jmp    QWORD PTR [RIP+0x2004d2]        # 6008a8 <_GLOBAL_OFFSET_TABLE_+0x18>
  4003d6: push   0
  4003db: jmp    4003c0 <printf@plt-0x10>

4003e0 <__libc_start_main@plt>:
  4003e0: jmp    QWORD PTR [RIP+0x2004ca]        # 6008b0 <_GLOBAL_OFFSET_TABLE_+0x20>
  4003e6: push   1
  4003eb: jmp    4003c0 <printf@plt-0x10>
The _GLOBAL_OFFSET_TABLE_ (labeled as R_X86_64_JUMP_SLOT and starts at address 0x600890) is located in .got.plt section (to see this, run the command objdump -h a.out |grep -A 1 600890 or the command readelf -r a.out) The data in .got.plt section look like the following during runtime (use gdb to see them)
(gdb) b *0x4003d0
(gdb) run
(gdb) x/6a 0x600890
0x600890: 0x6006e8 <_DYNAMIC>     0x32696159a8
0x6008a0: 0x326950aa20 <_dl_runtime_resolve>     0x4003d6 <printf@plt+6>
0x6008b0: 0x326971c3f0 <__libc_start_main>       0x0
When printf is called the first time in the user program, the jump at 4003d0 will jump to 4003d6, which is just the next instruction (push 0) The it jumps to 4003c0, which does not have a function name (so it is shown as <printf@plt-0x10>). At 4003c6, it will jumps to _dl_runtime_resolve. This function (in Glibc's source file sysdeps/x86_64/dl-trampoline.S) is a trampoline to _dl_fixup (in Glibc's source file elf/dl-runtime.c). _dl_fixup again, is part of Glibc runtime linker ld.so. In particular, it will change the address stored at 6008a8 to the actual address of printf in libc.so.6. To see this, set up a hardware watchpoint
(gdb) watch *0x6008a8
(gdb) cont
Continuing.
Hardware watchpoint 2: *0x6008a8

Old value = 4195286
New value = 1769244016
0x000000326950abc2 in fixup () from /lib64/ld-linux-x86-64.so.2
If we continue execution, printf will be called, as expected. When printf is called again in the user program, the jump at 4003d0 will bounce directly to printf:
(gdb) x/6a 0x600890
0x600890: 0x6006e8 <_DYNAMIC>     0x32696159a8
0x6008a0: 0x326950aa20 <_dl_runtime_resolve>     0x3269748570 <printf>
0x6008b0: 0x326971c3f0 <__libc_start_main>       0x0

.init, .fini, .preinit_array, .init_array and .fini_array sections

.init and .fini sections contain code to do initialization and termination, as specified by the System V Application Binary Interface. If the code is compiled by GCC, then one will see the following code in .init and .fini sections, respectively:
4003a8 <_init>:
  4003a8: sub    RSP, 8
  4003ac: call   call_gmon_start
  4003b1: call   frame_dummy
  4003b6: call   __do_global_ctors_aux
  4003bb: add    RSP, 8
  4003bf: ret

400618 <_fini>:
  400618: sub    RSP, 8
  40061c: call   __do_global_dtors_aux
  400621: add    RSP, 8
  400625: ret
There is only one function: _init, in .init section, and likewise, only one function: _fini in .fini section. Both _init and _fini are synthesized at compile time by the compiler/linker. Glibc provides its own prolog and epilog for _init and _fini, but the compiler is free to choose how to use them and add more code into _init and _fini.

In Glibc, the source file sysdeps/generic/initfini.c (and some system dependent ones, such as sysdeps/x86_64/elf/initfini.c) is compiled into two files: /usr/lib64/crti.o for prolog and /usr/lib64/crtn.o for epilog.

For the compiler part, GCC uses different prolog and epilog files, depending on the compiler command-line options. To see them, execute gcc -dumpspec, and one can see

...

*endfile:
  %{ffast-math|funsafe-math-optimizations:crtfastmath.o%s}
  %{mpc32:crtprec32.o%s}
  %{mpc64:crtprec64.o%s}
  %{mpc80:crtprec80.o%s}
  %{shared|pie:crtendS.o%s;:crtend.o%s}
  crtn.o%s

...

*startfile:
  %{!shared: %{pg|p|profile:gcrt1.o%s;pie:Scrt1.o%s;:crt1.o%s}}
  crti.o%s
  %{static:crtbeginT.o%s;shared|pie:crtbeginS.o%s;:crtbegin.o%s}

...
The detailed explanation of GCC spec file is here. For above snippet, it means, for example, if compiler command-line option -ffast-math is used, include GCC's crtfastmath.o file (this file can be found under /usr/lib/gcc/<arch>/<version>/) at the end of the linking process. Glibc's crtn.o is always included at the end of linking. The %s means this preceding file is a startup file. (GCC allows to skip startup files during linking using -nostartfiles compiler option)

Similarly, if -shared compiler command-line option is not used, then always include Glibc's crt1.o at the start of the linking process. crt1.o contains the function _start in .text section (not .init section!) _start is the function that is executed before anything else... see below. Next, include Glibc's crti.o in the linking. Finally, include either crtbeginT.o, crtbeginS.o, or crtbegin.o (both are part of GCC, of course), depending on whether -static or -shared (or neither) is used.

So, for example, if a program is compiled using dynamic linking (which is default), no profiling, no fast math optimizations, then the linking will include the following files in the following order:

  1. crt1.o (part of Glibc)
  2. crti.o (part of Glibc and contributes the code at 4003a8, 4003ac, 400618, and the body of call_gmon_start)
  3. crtbegin.o (part of GCC and contributes the code at 4003b1 and 40061c, and the body of frame_dummy and __do_global_dtors_aux)
  4. user's code
  5. crtend.o (part of GCC and contributes the code at 4003b6 and the body of __do_global_ctors_aux)
  6. crtn.o (part of Glibc and contributes the code at 4003bb, 4003bf, 400621, 400625)
Why __do_global_ctors_aux is in crtend*.o and __do_global_dtors_aux is in crtbegin*.o ? Recall the order of invocation of destructors should be the reverse order of invocation of constructors. Therefore, GCC doing so will ensure __do_global_ctors_aux is called as late as possible in .init section and __do_global_dtors_aux is called as early as possible in .fini section.

Now back to the 4003a8 <_init>.

call_gmon_start is part of the Glibc prolog /usr/lib64/crti.o. It initializes gprof related data structures.

frame_dummy is in GCC code gcc/crtstuff.c and it is used to set up excepion handling and Java class registration (JCR) information.

The most interesting code is __do_global_ctors_aux (in GCC's gcc/crtstuff.c and gcc/gbl-ctors.h) What it does is to call functions which are marked as __attribute__ ((constructor)) (and static C++ objects' constructors) one by one:

  __SIZE_TYPE__ nptrs = (__SIZE_TYPE__) __CTOR_LIST__[0];
  unsigned i;

  if (nptrs == (__SIZE_TYPE__)-1)
    for (nptrs = 0; __CTOR_LIST__[nptrs + 1] != 0; nptrs++);

  for (i = nptrs; i >= 1; i--)
    __CTOR_LIST__[i] ();
The array __CTOR_LIST__ is stored in a special section called .ctors. Suppose a function called foo is marked as __attribute__ ((constructor)), then the runtime call stack trace would be
(gdb) break foo
(gdb) run
(gdb) bt
#0  0x00000000004004d8 in foo ()
#1  0x0000000000400606 in __do_global_ctors_aux ()
#2  0x00000000004003bb in _init ()
#3  0x00000000004005a0 in ?? ()
#4  0x0000000000400561 in __libc_csu_init ()
#5  0x000000326971c46f in __libc_start_main ()
#6  0x000000000040041a in _start ()
Similarly, the __do_global_dtors_aux in _fini function will invoke all functions which are marked as __attribute__ ((destructor)). __do_global_dtors_aux code is also in GCC's source tree at gcc/crtstuff.c. If a function called foo is marked as __attribute__ ((destructor)) (and static C++ objects' destructors), then the runtime call stack trace would be
(gdb) bt
#0  0x0000000000400518 in foo ()
#1  0x00000000004004ca in __do_global_dtors_aux ()
#2  0x0000000000400641 in _fini ()
#3  0x00000032699367e8 in ?? () from /lib64/tls/libc.so.6
#4  0x0000003269730c95 in exit () from /lib64/tls/libc.so.6
#5  0x000000326971c4d2 in __libc_start_main () from /lib64/tls/libc.so.6
#6  0x000000000040045a in _start ()
The array __DTOR_LIST__ contains the addresses of these destructors and it is stored in a special section called .dtors.

What user functions will be executed before main and at program exit?

As above call strack trace shows, _init is NOT the only function to be called before main. It is __libc_csu_init function (in Glibc's source file csu/elf-init.c) that determines what functions to be run before main and the order of running them. Its code is like this
   void __libc_csu_init (int argc, char **argv, char **envp)
   {
   #ifndef LIBC_NONSHARED
     {
       const size_t size = __preinit_array_end - __preinit_array_start;
       size_t i;
       for (i = 0; i < size; i++)
         (*__preinit_array_start [i]) (argc, argv, envp);
     }
   #endif

     _init ();

     const size_t size = __init_array_end - __init_array_start;
     for (size_t i = 0; i < size; i++)
         (*__init_array_start [i]) (argc, argv, envp);
   }
(Symbols such as __preinit_array_start, __preinit_array_end, __init_array_start, __init_array_end are defined by the default ld script; look for PROVIDE and PROVIDE_HIDDEN keywords in the output of ld -verbose command.)

The __libc_csu_fini function has similar code, but what functions to be executed at program exit are actually determined by exit:

    void __libc_csu_fini (void)
    {
    #ifndef LIBC_NONSHARED
      size_t i = __fini_array_end - __fini_array_start;
      while (i-- > 0)
        (*__fini_array_start [i]) ();

      _fini ();
    #endif
    }

To see what's going on, consider the following C code example:

   #include <stdio.h>
   #include <stdlib.h>

   void preinit(int argc, char **argv, char **envp) {
     printf("%s\n", __FUNCTION__);
   }

   void init(int argc, char **argv, char **envp) {
     printf("%s\n", __FUNCTION__);
   }

   void fini() {
     printf("%s\n", __FUNCTION__);
   }

   __attribute__((section(".init_array"))) typeof(init) *__init = init;
   __attribute__((section(".preinit_array"))) typeof(preinit) *__preinit = preinit;
   __attribute__((section(".fini_array"))) typeof(fini) *__fini = fini;

   void  __attribute__ ((constructor)) constructor() {
     printf("%s\n", __FUNCTION__);
   }

   void __attribute__ ((destructor)) destructor() {
     printf("%s\n", __FUNCTION__);
   }

   void my_atexit() {
     printf("%s\n", __FUNCTION__);
   }

   void my_atexit2() {
     printf("%s\n", __FUNCTION__);
   }

   int main() {
     atexit(my_atexit);
     atexit(my_atexit2);
   }
The output will be
   preinit
   constructor
   init
   my_atexit2
   my_atexit
   fini
   destructor
The .preinit_array and .init_array sections must contain function pointers (NOT code!) The prototype of these functions must be
void func(int argc,char** argv,char** envp)
__libc_csu_init execute them in the following order:
  1. Function pointers in .preinit_array section
  2. Functions marked as __attribute__ ((constructor)), via _init
  3. Function pointers in .init_array section
The .fini_array section must also contain function pointers and the prototype is like the destructor, i.e. taking no arguments and returning void. If the program exits normally, then the exit function (Glibc source file stdlib/exit.c) is called and it will do the following:
  1. In reverse order, functions registered via atexit or on_exit
  2. Function pointers in .fini_array section, via __libc_csu_fini
  3. Functions marked as __attribute__ ((destructor)), via __libc_csu_fini (which calls _fini after Step 2)
  4. stdio cleanup functions

It is not advisable to put a code in .init section, e.g.

void __attribute__((section(".init"))) foo() {
  ...
}
because doing so will cause __do_global_ctors_aux NOT to be called. The .init section will now look like this:
4003a0 <_init>:
  4003a0: sub    RSP, 8
  4003a4: call   call_gmon_start
  4003a9: call   frame_dummy

4003ae <foo>:
  4003ae: push   RBP
  4003af: mov    RBP, RSP

     ....  (foo's body)

  4003b2: leave
  4003b3: ret
  4003b4: call   __do_global_ctors_aux
  4003b9: add    RSP, 8
  4003bd: ret

Now .init section contains more than one function, but the epilog of _init is distorted by the insertion of foo

Similarly, it is not advisable to put a code in .fini section, because otherwise the code will look like this:

4006d8 <_fini>:
  4006d8: sub    RSP, 8
  4006dc: call   __do_global_dtors_aux

4006e1 <foo>:
  4006e1: push   RBP
  4006e2: mov    RBP, RSP

     ....  (foo's body)

  4006ef: leave
  4006f0: ret
  4006f1: add    RSP, 8
  4006f5: ret
Now the epilog of _fini is distorted by the insertion of foo, so the stack frame pointer will not be adjusted (add RSP, 8 is not executed), causing segmentation fault.

What do _start and __libc_start_main do?

The above call stack traces show that _start calls __libc_start_main, which runs all of the code before main.

_start is part of Glibc code, as in sysdeps/x86_64/elf/start.S. As mentioned earlier, it is compiled as /usr/lib64/crt1.o and is statically linked to user's executable binary during compilation. To see this, run gcc with -v command, and the last line would be something like:

.../collect2 ... /usr/lib64/crt1.o /usr/lib64/crti.o ...  /usr/lib64/crtn.o
_start is always placed at the beginning of .text section, and the default ld script specifies "Entry point address" (in ELF header, use readelf -h ld.so|grep Entry command to see) to be the address of _start (use ld -verbose | grep ENTRY command to see), so _start is guaranteed to be run before anything else. (This is changeable, however, at compile time one can specify a different initial address by -e option)

_start does only one thing: It sets up the arguments needed by __libc_start_main and then call it. __libc_start_main's source code is csu/libc-start.c and its function prototype is:

__libc_start_main (int (*main) (int, char **, char **),
                   int argc,
                   char *argv,
                   int  (*init) (int, char **, char **),
                   void (*fini) (void),
                   void (*rtld_fini) (void),
                   void *stack_end)
                  )
__libc_start_main does quite a lot of work in addition to kicking off __libc_csu_init:
  1. Set up argv and envp
  2. Initialize the thread local storage by calling __pthread_initialize_minimal (which only calls __libc_setup_tls).

    __libc_setup_tls will initialize Thread Control Block and Dynamic Thread Vector.

  3. Set up the thread stack guard
  4. Register the destructor (i.e. the rtld_fini argument passed to __libc_start_main) of the dynamic linker (by calling __cxa_atexit) if there is any
  5. Initialize Glibc inself by calling __libc_init_first
  6. Register __libc_csu_fini (i.e. the fini argument passed to __libc_start_main) using __cxa_atexit
  7. Call __libc_csu_init (i.e. the init argument passed to __libc_start_main)
    1. Call function pointers in .preinit_array section
    2. Execute the code in .init section, which is usually _init function. What _init function does is compiler-specific. For GCC, _init executes user functions marked as __attribute__ ((constructor)) (in __do_global_dtors_aux)
    3. Call function pointers in .init_array section
  8. Set up data structures needed for thread unwinding/cancellation
  9. Call main of user's program.
  10. Call exit
So if the last line of user program's main is return XX, then the XX will be passed to exit at Step #11 above. If the last line is not return XX or is simply return, then the value passed to exit would be undefined.

Of course, if the user program calls exit or abort, then exit will gets called.

Here is the call graph, which is worth a thousand words

and see here on how it is generated.

If one tries to build a program which does not contain main, then one should see the following error:

/usr/lib/crt1.o: In function `_start': (.text+0x20): undefined reference to `main'
collect2: ld returned 1 exit status
As mentioned earlier, crt1.o (part of Glibc) contains the function _start, which will call __libc_start_main and pass main (a function pointer) as one of the arguments. If one uses
nm -u /usr/lib/crt1.o
then it will show main is a undefined symbol in crt1.o. Now let's disassemble crt1.o:
$ objdump -M intel -dj .text /usr/lib/crt1.o

crt1.o:     file format elf64-x86-64

Disassembly of section .text:

0000000000000000 <_start>:
   0:   31 ed                   xor    ebp,ebp
   2:   49 89 d1                mov    r9,rdx
   5:   5e                      pop    rsi
   6:   48 89 e2                mov    rdx,rsp
   9:   48 83 e4 f0             and    rsp,0xfffffffffffffff0
   d:   50                      push   rax
   e:   54                      push   rsp
   f:   49 c7 c0 00 00 00 00    mov    r8,0x0
  16:   48 c7 c1 00 00 00 00    mov    rcx,0x0
  1d:   48 c7 c7 00 00 00 00    mov    rdi,0x0
  24:   e8 00 00 00 00          call   29 <_start+0x29>
  29:   f4                      hlt
  ...
Above shows .text+0x20 refers to the 4 bytes of an mov instruction. This means during the linking, the address of main should be resolved and then inserted at the right memory location: .text+0x20. Now let's cross reference the relocation table:
$ readelf -p /usr/lib/crt1.o

Relocation section '.rela.text' at offset 0x410 contains 4 entries:
  Offset          Info           Type           Sym. Value    Sym. Name + Addend
000000000012  00090000000b R_X86_64_32S      0000000000000000 __libc_csu_fini + 0
000000000019  000b0000000b R_X86_64_32S      0000000000000000 __libc_csu_init + 0
000000000020  000c0000000b R_X86_64_32S      0000000000000000 main + 0
000000000025  000f00000002 R_X86_64_PC32     0000000000000000 __libc_start_main - 4
Above shows where 0x20 comes from.

How to find the address of main of an executable binary ?

When an ELF executable binary is stripped off symbolic information, it is not clear where the main is located.

From above analysis, it's possible to find out the address of main (which is NOT the "Entry point address" seen from the output of readelf -h a.out | grep Entry command. "Entry point address" is the address of _start)

Since the address of main is the first argument to the call to __libc_start_main, we can extract the value of the first argument as follows.

On 64-bit x86, the calling convention requires that the first argument goes to RDI register, so the address can be extracted by

objdump -j .text -d a.out | grep -B5 'call.*__libc_start_main' | awk '/mov.*%rdi/ { print $NF }'
On 32-bit x86, the C calling convention ("cdecl") is that the first argument is the last item to be pushed onto the stack before the call, so the address can be extracted by
objdump -j .text -d a.out | grep -B2 'call.*__libc_start_main' | awk '/push.*0x/ { print $NF }'

PIC, TLS, and AMD64 code models

Relocation is the process of connecting symbolic references with symbolic definitions. The runtime relocation is done by ld.so, as in elf_machine_rela function in Glibc's source file sysdeps/x86_64/dl-machine.h. The link-time relocation is done by the link-editor ld, which uses the relocation table in the object file (.rela.text section). Each symbolic reference has an entry in the relocation table, and each entry contains three fields: offset, info (relocation type, symbol table index), and addend. The relocation types are:

Relocation type Meaning Used when
R_X86_64_16 Direct 16 bit zero extended
R_X86_64_32 Direct 32 bit zero extended
R_X86_64_32S Direct 32 bit sign extended
R_X86_64_64 Direct 64 bit Large code model
R_X86_64_8 Direct 8 bit sign extended
R_X86_64_COPY Copy symbol at runtime
R_X86_64_DTPMOD64 ID of module containing symbol TLS
R_X86_64_DTPOFF32 Offset in TLS block TLS
R_X86_64_DTPOFF64 Offset in module's TLS block TLS
R_X86_64_GLOB_DAT .got section, which contains addresses to the actual functions in DLL
R_X86_64_GOT32 32 bit GOT entry
R_X86_64_GOT64 64-bit GOT entry offset PIC & Large code model
R_X86_64_GOTOFF64 64-bit GOT offset PIC & Large code model
R_X86_64_GOTPC32 32-bit PC relative offset to GOT
R_X86_64_GOTPC32_TLSDESC 32-bit PC relative to TLS descriptor in GOT TLS
R_X86_64_GOTPC64 64-bit PC relative offset to GOT PIC & Large code model
R_X86_64_GOTPCREL 32 bit signed PC relative offset to GOT PIC
R_X86_64_GOTPCREL64 64-bit PC relative offset to GOT entry PIC & Large code model
R_X86_64_GOTPLT64 Like GOT64, indicates that PLT entry needed PIC & Large code model
R_X86_64_GOTTPOFF 32 bit signed PC relative offset to GOT entry for IE symbol TLS
R_X86_64_JUMP_SLOT .got.plt section, which contains addresses to the actual functions in DLL DLL
R_X86_64_PC16 16 bit sign extended PC relative
R_X86_64_PC32 PC relative 32 bit signed
R_X86_64_PC64 64-bit PC relative Large code model
R_X86_64_PC8 8 bit sign extended PC relative
R_X86_64_PLT32 32 bit PLT address
R_X86_64_PLTOFF64 64-bit GOT relative offset to PLT entry PIC & Large code model
R_X86_64_RELATIVE Adjust by program base
R_X86_64_SIZE32
R_X86_64_SIZE64
R_X86_64_TLSDESC 2 by 64-bit TLS descriptor TLS
R_X86_64_TLSDESC_CALL Relaxable call through TLS descriptor TLS
R_X86_64_TLSGD 32 bit signed PC relative offset to two GOT entries for GD symbol TLS & PIC
R_X86_64_TLSLD 32 bit signed PC relative offset to two GOT entries for LD symbol TLS
R_X86_64_TPOFF32 Offset in initial TLS block TLS
R_X86_64_TPOFF64 Offset in initial TLS block TLS & Large code model

According to Chapter 3.5 of AMD64 System V Application Binary Interface, there are four code models and they differ in addressing modes (absolute versus relative):

  • Small: All compile- and link-time addresses and symbols are assumed to fit in 32-bit immediate operands. This model restricts code and global data to the low 2 GB of the address space (to be exact, between 0x0 and 0x7eff ffff, which is 2031 MB)

    The compiler can encode symbolic references

    • In sign-extended immediate operands for offsets in the range of 0x8000 0000 (-231) to 0x100 0000 (224)
    • In zero-extended immediate operands for offsets in the range of 0x0 to 0x7f00 0000 (231 - 224)
    • In RIP relative addressing mode for offsets in the range 0xff00 0000 (-224 = -16 MB) to 0x100 0000 (224 = 16 MB)

    This mode is the default mode for most compilers.

  • Large: No restrictions are placed on the size or placement of code and data. The max virtual memory space is 48 bits (256 TB).
  • Medium: Like the Small code model, except the data sections are split into two parts, e.g. instead of having just .data, .rodata, .bss sections, there would also be .ldata, .lrodata, .lbss sections. The smaller .data, etc are still the same as in the Small code model, and the larger .ldata, etc are as in the Large code model.
  • Kernel: Like the Small code model, but the 2 GB address space spans from 0xffff ffff 8000 0000 (264-231) to 0xffff ffff ff00 0000 (264-224) Because of this, the offsets which can be encoded using sign-extended and zero-extended immediate operands change.
Now consider the following C code
extern int esrc[100];
       int gsrc[100];
static int ssrc[100];

void foo() {
  int k;
  k = esrc[5];
  k = gsrc[5];
  k = ssrc[5];
}
  • Small code model, no PIC (i.e. compiled with just gcc -c):
    k = esrc[5];     mov EAX, DWORD PTR[RIP+0x0]
                     mov DWORD PTR[RBP-0x4], EAX
    k = gsrc[5];     mov EAX, DWORD PTR[RIP+0x0]
                     mov DWORD PTR[RBP-0x4], EAX
    k = ssrc[5];     mov EAX, DWORD PTR[RIP+0x0]
                     mov DWORD PTR[RBP-0x4], EAX
    
    and the relocation table is (use readelf -r foo.o command)
    type           Sym. Name + Addend
    R_X86_64_PC32  esrc + 10
    R_X86_64_PC32  gsrc + 10
    R_X86_64_PC32  .bss + 10
    
    All of the 0x0's in the generated assembly will be filled at link-time with their relative offsets in respective sections, as indicated by the relocation table.

  • Large code model, no PIC (i.e. compiled with gcc -c -mcmodel=large)
    k = esrc[5];     mov RAX, 0x0
                     mov EAX, DWORD PTR[RAX+0x10]
                     mov DWORD PTR[RBP-0x4], EAX
    k = gsrc[5];     mov RAX, 0x0
                     mov EAX, DWORD PTR[RAX+0x10]
                     mov DWORD PTR[RBP-0x4], EAX
    k = ssrc[5];     mov RAX, 0x0
                     mov EAX, DWORD PTR[RAX+0x10]
                     mov DWORD PTR[RBP-0x4], EAX
    
    and the relocation table is:
    type         Sym. Name + Addend
    R_X86_64_64  esrc + 0
    R_X86_64_64  gsrc + 0
    R_X86_64_64  .bss + 0
    
    All of the 0x0's in the generated assembly will be filled at link-time with their (64-bit) absolute addresses.

  • Small code model, PIC (i.e. compiled with gcc -c -fPIC. Note that adding -shared or not will not affect the generated code)
    k = esrc[5];     mov RAX, QWORD PTR[RIP+0x0]
                     mov EAX, DWORD PTR[RAX+0x10]
                     mov DWORD PTR[RBP-0x4], EAX
    k = gsrc[5];     mov RAX, QWORD PTR[RIP+0x0]
                     mov EAX, DWORD PTR[RAX+0x10]
                     mov DWORD PTR[RBP-0x4], EAX
    k = ssrc[5];     mov EAX, DWORD PTR[RIP+0x0]
                     mov DWORD PTR[RBP-0x4], EAX
    
    and the relocation table is:
    type              Sym. Name + Addend
    R_X86_64_GOTPCREL esrc - 4
    R_X86_64_GOTPCREL gsrc - 4
    R_X86_64_PC32     .bss + 10
    
    The first two 0x0's in the generated assembly will be filled with the relative offset of _GLOBAL_OFFSET_TABLE_ (i.e. the .got.plt section)

  • Large code model, PIC (i.e. compiled with gcc -c -fPIC -mcmodel=large)
                     lea RBX, [RIP-0x7]
                     mov R11, 0x0
                     add RBX, R11
    k = esrc[5];     mov RAX, 0x0
                     mov RAX, QWORD PTR[RBX+RAX*1]
                     mov EAX, DWORD PTR[RAX+0x10]
                     mov DWORD PTR[RBP-0x4], EAX
    k = gsrc[5];     mov RAX, 0x0
                     mov RAX, QWORD PTR[RBX+RAX*1]
                     mov EAX, DWORD PTR[RAX+0x10]
                     mov DWORD PTR[RBP-0x4], EAX
    k = ssrc[5];     mov RAX, 0x0
                     mov RAX, QWORD PTR[RBX+RAX*1]
                     mov EAX, DWORD PTR[RAX+0x10]
                     mov DWORD PTR[RBP-0x4], EAX
    
    The first 0x0 is in the generated assembly will be filled with the absolute address of _GLOBAL_OFFSET_TABLE_

_GLOBAL_OFFSET_TABLE_, .got.plt section, and DYNAMIC segment

Earlier we see that the _GLOBAL_OFFSET_TABLE_ is located in .got.plt section:
(gdb) b *0x4003d0
(gdb) run
(gdb) x/6a 0x600890
0x600890: 0x6006e8 <_DYNAMIC>     0x32696159a8
0x6008a0: 0x326950aa20 <_dl_runtime_resolve>     0x4003d6 <printf@plt+6>
0x6008b0: 0x326971c3f0 <__libc_start_main>       0x0
According to Chapter 5.2 of AMD64 System V Application Binary Interface, the first 3 entries of this table are reserved for special purposes. The first entry is set up during compilation by the link editor ld. The second and third entries are set up during runtime by the runtime linker ld.so (see function _dl_relocate_object in Glibc source file elf/dl-reloc.c and in particular, notice the ELF_DYNAMIC_RELOCATE macro, which calls function elf_machine_runtime_setup in sysdeps/x86_64/dl-machine.h)

The first entry _DYNAMIC has value 6006e8, and this is exactly the starting address of .dynamic section (or DYNAMIC segment, in ELF's "execution view".) The runtime linker ld.so uses this section to find the all necessary information needed for runtime relocation and dynamic linking.

To see DYNAMIC segment's content, use readelf -d a.out command, or objdump -x a.out, or just use x/50a 0x6006e8 in gdb. The readelf -d a.out command will show something like this:

Dynamic section at offset 0x6e8 contains 21 entries:
  Tag        Type                 Name/Value
 0x0000000000000001 (NEEDED)     Shared library: [libc.so.6]  <-- dependent dynamic library name
 0x000000000000000c (INIT)       0x4003a8     <-- address of .init section
 0x000000000000000d (FINI)       0x400618     <-- address of .fini section
 0x0000000000000004 (HASH)       0x400240     <-- address of .hash section
 0x000000006ffffef5 (GNU_HASH)   0x400268     <-- address of .gnu.hash section
 0x0000000000000005 (STRTAB)     0x4002e8     <-- address of .strtab section
 0x0000000000000006 (SYMTAB)     0x400288     <-- address of .symtab section
 0x000000000000000a (STRSZ)      63 (bytes)   <-- size of .strtab section
 0x000000000000000b (SYMENT)     24 (bytes)   <-- size of an entry in .symtab section
 0x0000000000000015 (DEBUG)      0x0          <-- see below
 0x0000000000000003 (PLTGOT)     0x600860     <-- address of .got.plt section
 0x0000000000000002 (PLTRELSZ)   48 (bytes)   <-- total size of .rela.plt section
 0x0000000000000014 (PLTREL)     RELA         <-- RELA or REL ?
 0x0000000000000017 (JMPREL)     0x400368     <-- address of .rela.plt section
 0x0000000000000007 (RELA)       0x400350     <-- address of .rela.dyn section
 0x0000000000000008 (RELASZ)     24 (bytes)   <-- total size of .rela.dyn section
 0x0000000000000009 (RELAENT)    24 (bytes)   <-- size of an entry in .rela.dyn section
 0x000000006ffffffe (VERNEED)    0x400330     <-- address of .gnu.version_r section
 0x000000006fffffff (VERNEEDNUM) 1            <-- number of needed versions
 0x000000006ffffff0 (VERSYM)     0x400328     <-- address of .gnu.version section
 0x0000000000000000 (NULL)       0x0          <-- marks the end of .dynamic section
Each entry in DYNAMIC segment is a struct of only two members: "tag" and "value". The NEEDED, INIT ... above are "tags" (see /usr/include/elf.h)

Other tags of interest are:

BIND_NOW           The same as BIND_NOW in FLAGS. This has been superseded by
                   BIND_NOW in FLAGS

CHECKSUM           The checksum value used by prelink.

DEBUG              At runtime ld.so will fill its value with the runtime
                   address of r_debug structure (see elf/rtld.c)
                   and this info is used by GDB (see elf_locate_base function
                   in GDB's source tree).

FINI               Address of .fini section
FINI_ARRAY         Address of .fini_array section
FINI_ARRAYSZ       Size of .fini_array section

FLAGS              Additional flags, such as BIND_NOW, STATIC_TLS, TEXTREL..

FLAGS_1            Additional flags used by Solaris, such as NOW (the same as BIND_NOW), INTERPOSE..

GNU_PRELINKED      The timestamp string when the binary object is last prelinked.

INIT               Address of .init section
INIT_ARRAY         Address of .init_array section
INIT_ARRAYSZ       Size of .init_array section

INTERP             Address of .interp section

PREINIT_ARRAY      Address of .preinit_array section
PREINIT_ARRAYSZ    Size of .preinit_array section

RELACOUNT          Number of R_X86_64_RELATIVE entries in RELA segment (.rela.dyn
                   section)

RPATH              Dynamic library search path, which has higher precendence than
                   LD_LIBRARY_PATH. RPATH is ignored if RUNPATH is present.

                   Use of RPATH is deprecated.

                   When one uses "gcc -Wl,-rpath=... " to build binaries, the info
                   is stored here.

RUNPATH            Dynamic library search path, which has lower precendence than
                   LD_LIBRARY_PATH.

                   When one uses "gcc -Wl,-rpath=...,--enable-new-dtags"
                   to build binaries, the info is stored here.
                   (See here for details.)

                   One can use chrpath
                   tool to manipulate RPATH and RUNPATH settings.


SONAME             Shared object (i.e. dynamic library) name. When one uses
                   "gcc -Wl,-soname=... " to build binaries, the info is
                   stored here.

TEXTREL            Relocation might modify .text section.

VERDEF             Address of .gnu.version_d section
VERDEFNUM          Number of version definitions.

Runtime Relocation

After exploring DYNAMIC segment, we can explain how ld.so performs runtime relocation.

First, before ld.so loads all dependent libraries of a dynamic executable, it needs to run its own relocation! Even if ld.so is a statically-linked binary, it also has a DYNAMIC segment and thus PLTREL (.rela.dyn section) and JMPREL (.rela.plt section) tags:

$ readelf -a `readelf -p .interp /bin/sh | awk '/ld/ {print $3}'`

 ....

Dynamic section at offset 0x14e18 contains 22 entries:
  Tag        Type                         Name/Value
 0x000000000000000e (SONAME)             Library soname: [ld-linux-x86-64.so.2]
 0x0000000000000004 (HASH)               0x3269500190
 0x0000000000000005 (STRTAB)             0x3269500578
 0x0000000000000006 (SYMTAB)             0x3269500260
 0x000000000000000a (STRSZ)              388 (bytes)
 0x000000000000000b (SYMENT)             24 (bytes)
 0x0000000000000003 (PLTGOT)             0x3269614f98
 0x0000000000000002 (PLTRELSZ)           120 (bytes)
 0x0000000000000014 (PLTREL)             RELA
 0x0000000000000017 (JMPREL)             0x32695009a0
 0x0000000000000007 (RELA)               0x32695007c0
 0x0000000000000008 (RELASZ)             480 (bytes)
 0x0000000000000009 (RELAENT)            24 (bytes)
 0x000000006ffffffc (VERDEF)             0x3269500740
 0x000000006ffffffd (VERDEFNUM)          4
 0x0000000000000018 (BIND_NOW)
 0x000000006ffffffb (FLAGS_1)            Flags: NOW
 0x000000006ffffff0 (VERSYM)             0x32695006fc
 0x000000006ffffff9 (RELACOUNT)          19
 0x000000006ffffdf8 (CHECKSUM)           0x4c4e099e
 0x000000006ffffdf5 (GNU_PRELINKED)      2010-08-26T08:13:28
 0x0000000000000000 (NULL)               0x0

Relocation section '.rela.dyn' at offset 0x7c0 contains 20 entries:
  Offset          Info           Type           Sym. Value    Sym. Name + Addend
003269614cf0  000000000008 R_X86_64_RELATIVE                    000000326950dd80
  ....
003269615820  000000000008 R_X86_64_RELATIVE                    0000003269501140
003269614fe0  001e00000006 R_X86_64_GLOB_DAT 0000003269615980 _r_debug + 0

Relocation section '.rela.plt' at offset 0x9a0 contains 5 entries:
  Offset          Info           Type           Sym. Value    Sym. Name + Addend
003269614fb0  000b00000007 R_X86_64_JUMP_SLO 000000326950f1b0 __libc_memalign + 0
003269614fb8  000c00000007 R_X86_64_JUMP_SLO 000000326950f2b0 malloc + 0
003269614fc0  001200000007 R_X86_64_JUMP_SLO 000000326950f2c0 calloc + 0
003269614fc8  001800000007 R_X86_64_JUMP_SLO 000000326950f340 realloc + 0
003269614fd0  002000000007 R_X86_64_JUMP_SLO 000000326950f300 free + 0
Note that the ld.so is prelinked. On Fedora and Red Hat Enterprise Linux (RHEL) systems, prelink is run every two weeks. To see if your Linux has similar setup, check /etc/sysconfig/prelink and /etc/prelink.conf

What does this prelink do? It changes the base address of a dynamic library to the actual address in the user program's address space when it is loaded into memory. Of course, ld.so recognizes GNU_PRELINKED tag and will load a dynamic library to its this base address (recall the first argument of mmap is the preferred address; of course, this is subject to the operating system.)

Normally, a dynamic library is built as position independent code, i.e. the -fPIC compiler command-line option, and thus the base address is 0. For example, a normal libc.so has ELF program header as follows (readelf -l command):

Program Headers:
  Type  Offset             VirtAddr           PhysAddr
        FileSiz            MemSiz              Flags  Align
  LOAD  0x0000000000000000 0x0000000000000000 0x0000000000000000
        0x0000000000179058 0x0000000000179058  R E    200000
  LOAD  0x0000000000179730 0x0000000000379730 0x0000000000379730
        0x0000000000004668 0x00000000000090f8  RW     200000
   ....
And when calling mmap with address 0 (i.e. NULL) the operating system can choose any address it feels appropriate.

A prelinked one, on the other hand, has its ELF program header as follows:

Program Headers:
  Type Offset             VirtAddr           PhysAddr
       FileSiz            MemSiz              Flags  Align
  LOAD 0x0000000000000000 0x0000003433e00000 0x0000003433e00000
       0x000000000001bb80 0x000000000001bb80  R E    200000
  LOAD 0x000000000001bb90 0x000000343401bb90 0x000000343401bb90
       0x0000000000000f58 0x00000000000010f8  RW     200000
What is the advantage of prelinking? ld.so will not process R_X86_64_RELATIVE relocation types since they are already in the "right" place in user program's address space. The extra benefit of this is the memory regions which ld.so would have written to (if R_X86_64_RELATIVE needs processing) will not incur any Copy-On-Writes and thus can be made Read-Only.

According to this post, for GUI programs, which tend to link against dozens of dynamic libraries and use lengthy C++ demangled names, the speed up can be an order of magnitude.

How to disable prelinking at runtime? Run the user program with LD_USE_LOAD_BIAS environmental variable set to 0.

How does ld.so process its own relocation?

The relocation is done by _dl_relocate_object function in Glibc's elf/dl-reloc.c, which will call elf_machine_rela function in sysdeps/x86_64/dl-machine.h to do the majority of work.

First to be processed is the .rela.dyn relocation table, which contains a bunch of R_X86_64_RELATIVE types and one R_X86_64_GLOB_DAT type (the variable _r_debug)

If prelink is used, i.e. ld.so is indeed loaded to the desired address, then R_X86_64_RELATIVE relocation types will be ignored. If not, then the address calculation for R_X86_64_RELATIVE types is

Base Address + Value Stored at [Base Address + Offset]
For example, in ld.so's case, its base address is 2a95556000 (can be obtained from pmap command; inside ld.so, it calls elf_machine_load_address function to get this value)
0000400000    4K r-x--  /tmp/a.out
0000500000    4K rw---  /tmp/a.out
2a95556000   92K r-x--  /lib64/ld.so
2a9556d000    8K rw---  [ anon ]
2a95599000    4K rw---  [ anon ]
2a9566c000    4K r----  /lib64/ld.so
2a9566d000    4K rw---  /lib64/ld.so
3269700000 1216K r-x--  /lib64/libc-2.3.4.so
...
And ld.so's .rela.dyn relocation table is (no prelinked! If ld.so is prelinked, the offset will be in a much higher address)
Relocation section '.rela.dyn' at offset 0x7c0 contains 20 entries:
  Offset          Info           Type           Sym. Value    Sym. Name + Addend
000000116d50  000000000008 R_X86_64_RELATIVE                    000000000000e250
...
so the relocation for 000000116d50 is processed as
0x2a95556000 + *(0x116d50+0x2a95556000)
and this new value is stored at 0x2a9566cd50 (=0x116d50+0x2a95556000)

As R_X86_64_RELATIVE types do not require symbol lookups, they are handled in a tight loop in elf_machine_rela_relative function in sysdeps/x86_64/dl-machine.h

Any relocation types other than R_X86_64_RELATIVE need to go through symbol resolution first.

So what about R_X86_64_GLOB_DAT relocation type in ld.so ? First, RESOLVE_MAP (a macro defined within elf/dl-reloc.c) is called (with r_type = R_X86_64_GLOB_DAT) to find out which ELF binary (could be the user's program or its dependent dynamic libraries) contains this symbol. Then R_X86_64_GLOB_DAT relocation type is calculated as

Base Address + Symbol Value + Addend
where Base Address is the base address of ELF binary which contains the symbol, and Symbol Value is the symbol value from the symbol table of ELF binary which contains the symbol.

So for ld.so,

Relocation section '.rela.dyn' at offset 0x7c0 contains 20 entries:
  Offset          Info           Type           Sym. Value    Sym. Name + Addend
  ....
000000116fe0  001e00000006 R_X86_64_GLOB_DAT 00000000001179c0 _r_debug + 0
The relocation for 000000116fe0 is processed as
0x2a95556000 + 0x1179c0 + 0
because ld.so determines _r_debug can be found from itself. The calculated value is stored at 0x2a9566cfe0 (=0x116fe0+0x2a95556000).

The next to be processed by ld.so is its own .rela.plt relocation table, which contains a bunch of R_X86_64_JUMP_SLOT types. This reloction type is handled exactly the same way as R_X86_64_GLOB_DAT.

After ld.so finishes its own relocation, it loads user program's dependent libraries and process their relocations one by one. First, ld.so handles libc.so's relocation. libc.so has two relocation types we have not covered so far: R_X86_64_64 and R_X86_64_TPOFF64.

R_X86_64_64 relocation type is processed by first looking up the symbol's runtime absolute address, and then calculating

Absolute Address + Addend
And the R_X86_64_TPOFF64 relocation type is calculated as
Symbol Value + Addend - TLS Offset
which usually results in a negative value.

R_X86_64_COPY relocation type

R_X86_64_COPY relocation type is used when a dynamic binary refers to an initialized global variable (not a function!) defined in a dynamic link library. Unlike functions, for variables, there is no lazy binding, and the trampoline trick used in .plt section does not work. Instead, the global variable will actually be allocated in dynamic binary's .bss section.

To see how R_X86_64_COPY relocation type works, consider the following two code:

foo.c

    int foo=4;

    void foo_access() {
      foo=5;
    }

bar.c

    #include <stdio.h>
    extern int foo;

    int main() {
       printf("foo=%d\n",foo);
    }
Now compile them as follows:
$ gcc -shared -fPIC -Wl,-soname=libfoo.so foo.c -o /tmp/libfoo.so
$ gcc bar.c -o bar -L/tmp -lfoo
And run them as
$ LD_PRELOAD=/tmp/libfoo.so ./bar
Before explaining what happened during runtime, we need to examine the binaries first.

The foo_access in libfoo.so is like this:

69c <foo_access>:
 69c:  push   rbp
 69d:  mov    rbp,rsp
 6a0:  mov    rax,QWORD PTR [rip+0x100269]  # 100910 <_DYNAMIC+0x198>
 6a7:  mov    DWORD PTR [rax],0x5
 6ad:  leave
 6ae:  ret
So for libfoo.so, the address of variable foo is in its .got section, not .data section:
$ readelf -a /tmp/libfoo.so

Section Headers:
  [Nr] Name              Type             Address           Offset
       Size              EntSize          Flags  Link  Info  Align
...
  [18] .got              PROGBITS         0000000000100908  00000908
       0000000000000020  0000000000000008  WA       0     0     8
  [19] .got.plt          PROGBITS         0000000000100928  00000928
       0000000000000020  0000000000000008  WA       0     0     8
...
  [20] .data             PROGBITS         0000000000100948  00000948
       0000000000000014  0000000000000000  WA       0     0     8
...

Relocation section '.rela.dyn' at offset 0x520 contains 6 entries:
  Offset          Info           Type           Sym. Value    Sym. Name + Addend
000000100948  000000000008 R_X86_64_RELATIVE                    0000000000100948
000000100950  000000000008 R_X86_64_RELATIVE                    0000000000100768
000000100908  000f00000006 R_X86_64_GLOB_DAT 0000000000000000 __cxa_finalize + 0
000000100910  001100000006 R_X86_64_GLOB_DAT 0000000000100958 foo + 0
....
But what about the address 0x100958 ? This address is in libfoo.so's .data section! Well, 0x100958 has the initial value of foo (in our example, 4) At runtime, ld.so will copy this value to bar's .bss section:
$ objdump -sj .data libfoo.so

libfoo.so:     file format elf64-x86-64

Contents of section .data:
 100948 48091000 00000000 68071000 00000000  H.......h.......
 100958 04000000                             ....

Next, disassemble the main function of bar:

4005f8 <main>:
  4005f8:  push   rbp
  4005f9:  mov    rbp,rsp
  4005fc:  mov    esi,DWORD PTR [rip+0x1003de] # 5009e0 <__bss_start>
  400602:  mov    edi,0x40070c
  400607:  mov    eax,0x0
  40060c:  call   400528 <printf@plt>
  400611:  leave
  400612:  ret
So the variable foo is indeed located in bar's .bss section. Let's double check with nm:
$ nm -n bar | grep 5009e0

00000000005009e0 A __bss_start
00000000005009e0 A _edata
00000000005009e0 B foo
(Symbols such as __bss_start and _edata are defined by the default ld script; one can search them in the output of ld -verbose command.)

The dynamic relocation table of bar is:

Relocation section '.rela.dyn' at offset 0x490 contains 2 entries:
  Offset          Info           Type           Sym. Value    Sym. Name + Addend
000000500998  000c00000006 R_X86_64_GLOB_DAT 0000000000000000 __gmon_start__ + 0
0000005009e0  000700000005 R_X86_64_COPY     00000000005009e0 foo + 0
Now what happens during runtime is this: After ld.so loads all dependent dynamic libraries, it starts processing their relocations. When it sees foo of libfoo.so, it calls RESOLVE_MAP with r_type = R_X86_64_GLOB_DAT to get the Base Address, which is 0, and Symbol Value, which is 5009e0. Next it sees foo of libfoo.so has R_X86_64_GLOB_DAT relocation type, so it calculates the new address as 5009e0 = 0 + 5009e0 + 0 (addend) and stores the result somewhere inside .got section.

After ld.so has processed relocations of all dynamic libraries, it starts processing the relocation table of bar. When it sees foo of bar, it calls RESOLVE_MAP again, but with r_type = R_X86_64_COPY. This time, the address returned is the runtime address of foo in libfoo.so's .data section. As mentioned earlier, this address holds the initial value of foo. Next it sees foo of bar has R_X86_64_COPY relocation type, so it uses memcpy to copy data to 5009e0 (see the Sym. Value of .rela.dyn section of bar above) from the runtime address of foo in libfoo.so's .data section (see Glibc source file sysdeps/x86_64/dl-machine.h)

The above example also illustrates the difference between .got section and .got.plt section. For the runtime linker ld.so, all it knows is entries in PLTREL segment, i.e. .rela.dyn section, (which corresponds to .got section) must be resolved/relocated immediately, while entries in JMPREL segment, i.e. .rela.plt section, (which corresponds to .got.plt section) can use lazy binding. For x86_64 architecture, the relocation is actually not needed for R_X86_64_JUMP_SLOT relocation types (albeit the symbol resolution is still needed)

PIC or no PIC

When building a dynamic library, we are told to always compile the code with -fPIC option.

What's the difference then ?

Consider the following simple code:

#include <stdio.h>
int bar;

void foo() {
  printf("%d\n",bar);
}
Compile the above code in 32-bit mode with and without -fPIC:
$ gcc -shared -m32 foo.c -o nopic.so
$ gcc -shared -m32 -fPIC foo.c -o pic.so
(If you try to compile the above in 64-bit mode, GCC will stop and insist you should compile with -fPIC option, i.e. you are going to see error message such as relocation R_X86_64_PC32 against symbol `XXXYYY' can not be used when making a shared object; recompile with -fPIC) The sections and relocation tables of nopic.so and pic.so are shown at left and right hand side, respectively:
Section Headers:                                                 Section Headers:
[Nr] Name           Type     Addr                                [Nr] Name              Type     Addr
[ 0]                NULL     0000                                [ 0]                   NULL     0000
  ...                                                              ...
[ 8] .init          PROGBITS 02f8                                [ 8] .init             PROGBITS 02f0
[ 9] .plt           PROGBITS 0310                                [ 9] .plt              PROGBITS 0308
[10] .text          PROGBITS 0340                                [10] .text             PROGBITS 0350
[11] .fini          PROGBITS 0488                                [11] .fini             PROGBITS 04a8
[12] .rodata        PROGBITS 04a4                                [12] .rodata           PROGBITS 04c4
  ...                                                              ...
[17] .dynamic       DYNAMIC  14c0                                [17] .dynamic          DYNAMIC  14e0
[18] .got           PROGBITS 1590                                [18] .got              PROGBITS 15a8
[19] .got.plt       PROGBITS 159c                                [19] .got.plt          PROGBITS 15b8
[20] .data          PROGBITS 15b0                                [20] .data             PROGBITS 15d0

  ...                                                              ...

Relocation section '.rel.dyn' at offset 0x2b0                    Relocation section '.rel.dyn' at offset 0x2b0
contains 7 entries:                                              contains 5 entries:
 Offset     Info    Type            Sym.Value  Sym. Name         Offset     Info    Type            Sym.Value  Sym. Name
00000439  00000008 R_386_RELATIVE                                000015d0  00000008 R_386_RELATIVE
000015b0  00000008 R_386_RELATIVE                                000015a8  00000106 R_386_GLOB_DAT    000015dc   bar
00000434  00000101 R_386_32          000015bc   bar                     ...
00000445  00000602 R_386_PC32        00000000   printf
 ...

Relocation section '.rel.plt' at offset 0x2e8:                   Relocation section '.rel.plt' at offset 0x2d8
contains 2 entries:                                              contains 3 entries:
 Offset     Info    Type            Sym.Value  Sym. Name         Offset     Info    Type            Sym.Value  Sym. Name
000015a8  00000207 R_386_JUMP_SLOT   00000000   __gmon_start__   000015c4  00000207 R_386_JUMP_SLOT   00000000   __gmon_start__
000015ac  00000a07 R_386_JUMP_SLOT   00000000   __cxa_finalize   000015c8  00000607 R_386_JUMP_SLOT   00000000   printf
                                                                      ...
When we compile with -fPIC we can see the variable bar has the right relocation type (R_386_GLOB_DAT) and the relocation takes place in the right section (.got) The same for printf.

Without -fPIC, the relocations of the format string "\n", bar and printf all take place inside the .text section! But we know .text section is in a Read-Only LOAD segment, so what ld.so would do ?

As expected, ld.so will make .text section writeable, patch the bytes, and make it Read-Only again. Since the relocation of both bar and printf are in .rel.dyn, their relocations are performed immediately (no lazy binding), so this approach is feasible.

So how does ld.so handle R_386_RELATIVE, R_386_32 and R_386_PC32 relocation types ?

Let's look at the disassembly:

0000042c <foo>:
 42c: 55                push   ebp
 42d: 89 e5             mov    ebp,esp
 42f: 83 ec 18          sub    esp,0x18
 432: 8b 15 00 00 00 00 mov    edx,DWORD PTR ds:0x0   <-- reference to bar
 438: b8 a4 04 00 00    mov    eax,0x4a4              <-- reference to "%d\n" format string in .rodata
 43d: 89 54 24 04       mov    DWORD PTR [esp+0x4],edx
 441: 89 04 24          mov    DWORD PTR [esp],eax
 444: e8 fc ff ff ff    call   445 <foo+0x19>  <-- reference to printf
 449: c9                leave
 44a: c3                ret
How would the 4 bytes starting at 445 (R_386_PC32 type) be patched ? Suppose at runtime, our nopic.so is loaded into memory with base address 8000, and the 4 bytes to be patched are now at 8000 + 445 = 8445. Furthermore, suppose ld.so has determined the entry address of printf to be 10000, then ld.so calculates the relative offset as follows:
10000 - 8445 + fffffffc = 7bb7
(fffffffc is -4) so ld.so replaces fc ff ff ff with b7 7b 00 00

To patch the 4 bytes starting at 434 (R_386_32 type) is simpler. ld.so will simply overwrite the 4 bytes with the runtime absolute address of bar.

To patch the 4 bytes starting at 439 (R_386_RELATIVE type) ld.so calculates the address as

10000 + 4a4 = 104a4
so ld.so replaces a4 04 00 00 with a4 04 01 00

Finally, what about the R_386_RELATIVE relocation at 15b0 ? 15b0 is the starting address of .data section, and the first 4 bytes of .data section stores its own address, 15b0. So it has to be relocated and patched as 115b0.

In conclusion, R_386_RELATIVE means "32-bit relative to base address", R_386_PC32 means the "32-bit IP-relative offset" and R_386_32 means the "32-bit absolute."

Troubleshooting ld.so

What is "error while loading shared libraries: requires glibc 2.5 or later dynamic linker" ?

The cause of this error is the dynamic binary (or one of its dependent shared libraries) you want to run only has .gnu.hash section, but the ld.so on the target machine is too old to recognize .gnu.hash; it only recognizes the old-school .hash section.

This usually happens when the dynamic binary in question is built using newer version of GCC. The solution is to recompile the code with either -static compiler command-line option (to create a static binary), or the following option:

-Wl,--hash-style=both
This tells the link editor ld to create both .gnu.hash and .hash sections.

According to ld documentation here, the old-school .hash section is the default, but the compiler can override it. For example, the GCC (which is version 4.1.2) on RHEL (Red Hat Enterprise Linux) Server release 5.5 has this line:

$ gcc -dumpspecs
....
*link:
%{!static:--eh-frame-hdr} %{!m32:-m elf_x86_64} %{m32:-m elf_i386} --hash-style=gnu   %{shared:-shared}   ....
...

For more information, see here.

What is "Floating point exception" ?

The cause of this error is the same as the previous question. On certain systems, e.g. RHEL, the old version ld.so is backported to emit "error while loading shared libraries: requires glibc 2.5 or later dynamic linker", but this is not always the case, and you will see this error instead.

What is ".../libc.so.6: version `GLIBC_2.4' not found " ?

As the error message says, some of the symbols need Glibc version 2.4 or higher. This can also be seen by
$ objdump -x foo | grep 'Version References' -A10

Version References:
  required from libc.so.6:
    0x0d696914 0x00 03 GLIBC_2.4
    0x09691a75 0x00 02 GLIBC_2.2.5

...
The fix is to recompile the code with -static compiler command-line option.

What is "FATAL: kernel too old" ?

Even if you recompile the code with -static compiler command-line option to avoid any dependency on the dynamic Glibc library, you could still encounter the error in question, and your code will exit with Segmentation Fault error.

This kernel version check is done by DL_SYSDEP_OSCHECK macro in Glibc's sysdeps/unix/sysv/linux/dl-osinfo.h It calls _dl_discover_osversion to get current kernel's version.

To wit, run your code (suppose it is not stripped) inside gdb,

(gdb) run
Starting program: foo
FATAL: kernel too old

Program received signal SIGSEGV, Segmentation fault.
0x00000000004324a9 in ptmalloc_init ()
(gdb) call _dl_discover_osversion()
$1 = 132617
(gdb) p/x $1
$2 = 0x20609
(gdb)
Here 0x20609 means the current kernel version is 2.6.9.

The fix (or hack) is to add the following function in your code:

int _dl_discover_osversion() { return 0xffffff; }
and compile your code with -static compiler command-line option.

Exploring Glibc's pthread_t

When one creates a thread using the Pthread API, one will get a pthread_t object as a handle. In Glibc, pthread_t is actually a pointer pointing to a pthread struct, which is opaque. Its definition can be found in Glibc's source tree at nptl/descr.h. The first member of pthread struct is yet another struct called tcbhead_t defined in system-dependent header files such as nptl/sysdeps/x86_64/tls.h. It holds TLS related information. It contains at least an integer member called multiple_threads which indicates if the process is running in multi-thread mode.

The second member of pthread struct is also a struct called list_t defined in nptl/sysdeps/pthread/list.h.

The third and fourth members of pthread struct are thread ID and thread group ID (both are of pid_t type).

Other members of pthread struct which are of interest: int cancelhandling for cancellation information, int flags for thread attributes, start_routine for start position of the code to be executed for the thread, void *arg for the argument to start_routine void *stackblock and size_t stackblock_size for thread-specific stack information.

Since pthread struct is opaque, how can one obtain the above information, or more precisely, how can one obtain the offsets of these members within the pthread struct ? We can use the known information and search for the memory region pointed by pthread_t, as in this code snippet.