Creating and using debug symbol tables with CMake and gdb

28 Jan 2020 14:36 UTC Embedded

Introduction

When working with a big project on a resource constrained embedded hardware, it might be difficult to debug it properly on the target board. Executables and libraries compiled in Debug mode are big, bloated and slow. To proper debug them in GDB, you need compile symbols, otherwise you’ll not be able to understand what a stack trace means.

Sometimes it may be useful to structure your project so that it can be debugged with emulators, mocks and simulated devices on a PC; it is a wise choice because emulators and mocks speed up debugging and developing when other parts (especially hardware and firmware) are not ready yet.

But this is not always possible, especially when you are debugging a piece of code that works with specific hardware that can’t be emulated in software or is not available on your development box.

Figure it: you have a specific chip on a i2c bus, and of course, it can’t be connected on any other hardware. So native debugging is your only choice.

But embedded hardware is often resource constrained. For example, installing the unstripped executables can be prohibitively. In my actual project, the whole project is composed of many executable and many shared libraries. While the production package is around 5Mb compressed, the unstripped version is around 70Mb and we only have 50Mb of flash space available.

How to deal with it?

ELF objects are composed of sections that includes code (.text), data (.bss and .data) and debug symbols. GDB can read symbol tables from other files, so one solution is to build the package as usual, perhaps with unoptimizing flags (-O0) and with debug symbols (-g) and then split them in executable and debug symbol files. This way code can installed, tested and used as usual and symbols only moved on board when needed.

This is just one of the solutions, maybe it is not always applicable, but why not?

This article is divided in two sections: first how to automatize the creation of the debug package, then how to load the symbol tables into gdb.

Extracting debug symbols from linked objects

The recipe for this is the same on both the library and executables. First run objcopy to extract the symbol table and build the debug symbol file, then strip the object file.

The debug symbol file is created by running this command (taken from gcc documentation)

objcopy --only-keep-debug my-object my-object.debug

my-object can be either an executable or a shared object (.so) file.

This will create a new ELF object with just the symbol table. See this note from man obj-copy:

The second step is strip the original executable. We use the usual strip command

strip --strip-debug my-object

Last step, is to link the executable to the debug symbol file, for this we run objcopy again:

objcopy --add-gnu-debuglink=my-object.dbg my-object

With this we’ll have two objects; the executable (my-object) and the symbol file (my-object.dbg). The latest can be copied on the board when needed, and loaded into dbg.

But first: let’s automate this into CMake.

Automating packaging with CMake

Everything can be automated into CMake at packaging time. Since I’m using CPack, I’ll create a new function called package() and it will be called instead of install(), this way:

package(TARGET hal LIBRARY DESTINATION ${CMAKE_INSTALL_LIBDIR} COMPONENT libraries)

Originally, I used install(TARGETS hal LIBRARY DESTINATION ........) Note the pretty similar syntax. This is the recipe:

<code>function(package)
    set(options RUNTIME LIBRARY)
    set(oneValueArgs DESTINATION COMPONENT TARGET)
    set(multiValueArgs "")
    cmake_parse_arguments(PACKAGE "${options}" "${oneValueArgs}"
            "${multiValueArgs}" ${ARGN})

    add_custom_command(TARGET ${PACKAGE_TARGET} POST_BUILD
            COMMAND ${CMAKE_OBJCOPY} --only-keep-debug $<TARGET_FILE:${PACKAGE_TARGET}> $<TARGET_FILE:${PACKAGE_TARGET}>.dbg
            COMMAND ${CMAKE_STRIP} --strip-debug $<TARGET_FILE:${PACKAGE_TARGET}>
            COMMAND ${CMAKE_OBJCOPY} --add-gnu-debuglink=$<TARGET_FILE:${PACKAGE_TARGET}>.dbg $<TARGET_FILE:${PACKAGE_TARGET}>
            )

    if (${PACKAGE_RUNTIME})
        install(TARGETS ${PACKAGE_TARGET}
                RUNTIME DESTINATION ${PACKAGE_DESTINATION}
                COMPONENT ${PACKAGE_COMPONENT})
    elseif (${PACKAGE_LIBRARY})
        install(TARGETS ${PACKAGE_TARGET}
                RUNTIME DESTINATION ${CMAKE_INSTALL_LIBDIR}
                LIBRARY DESTINATION ${CMAKE_INSTALL_LIBDIR}
                COMPONENT ${PACKAGE_COMPONENT})
    endif ()

    install(FILES $<TARGET_FILE:${PACKAGE_TARGET}>.dbg
            DESTINATION ${CMAKE_INSTALL_LIBDIR}/debug
            COMPONENT debugsym
            )

endfunction(package)</code>

Of course you need to define the “Debugsym” package as usual.

Loading symbols in GDB

Now it’s debug time, your executable is crashing and you don’t know why.

See the following example.

(gdb) terminate called after throwing an instance of 'std::bad_function_call'
   what():  bad_function_call
 Thread 8 "hald" received signal SIGABRT, Aborted.
 [Switching to Thread 0xb2efe450 (LWP 469)]
 0xb65dd368 in raise () from /lib/libc.so.6
 (gdb)bt
 0  0xb65dd368 in raise () from /lib/libc.so.6
 1  0xb65c91c8 in abort () from /lib/libc.so.6
 2  0xb6829db0 in __gnu_cxx::__verbose_terminate_handler() ()
 from /lib/libstdc++.so.6
 3  0xb6827aec in __cxxabiv1::__terminate(void (*)()) ()
 from /lib/libstdc++.so.6
 4  0xb6827b60 in std::terminate() () from /lib/libstdc++.so.6
 5  0xb6827ec8 in __cxa_throw () from /lib/libstdc++.so.6
 6  0xb6855358 in std::__throw_bad_function_call() () from /lib/libstdc++.so.6
 7  0xb6cb3214 in boost::variant<…….> () from /usr/lib/libhal.so

As you can see, the backtrace can’t display the source code level information. This is because all the executable are stripped.

We can use then the gdb symbol-file command. The debug symbol files have been copied in /usr/lib/debug/ but any place will work.

(gdb) symbol-file /usr/lib/debug/halcl.debug Reading symbols from /usr/lib/debug/halcl.debug…done.

The backtrace command now shows the source code lines up to the border of the object for which we have loaded the symbols, that is, only for executable. But the crash apparently happens in a shared library, and gdb can’t find the source lines. We’ll need to load the symbol file for libhal.so, using add-symbol-file. But this command requires the address to which the symbol applies. This can be found using info sharedlibrary.

(gdb) info sharedlibrary 
 From        To          Syms Read   Shared Object Library
 0xb6f06a00  0xb6f21e34  Yes ()     /lib/ld-linux-armhf.so.3 0xb6ea9e40  0xb6eec7e0  Yes ()     /usr/lib/libboost_program_options.so.1.66.0
 0xb6e73204  0xb6e744fc  Yes ()     /usr/lib/libboost_system.so.1.66.0 0xb6dde2b8  0xb6e4d090  Yes ()     /usr/lib/libboost_log.so.1.66.0
 0xb6d7f478  0xb6d8f664  Yes ()     /usr/lib/libboost_thread.so.1.66.0 0xb6d29bf0  0xb6d50484  Yes ()     /usr/lib/libos.so
 0xb6c7d328  0xb6cc8b5c  Yes ()     /usr/lib/libhal.so 0xb6b58d48  0xb6b649a4  Yes ()     /usr/lib/libusb-1.0.so.0
 0xb6b40780  0xb6b43cac  Yes ()     /lib/librt.so.1 0xb6aea720  0xb6af9d64  Yes ()     /lib/libpthread.so.0
 0xb6acefd8  0xb6ad33ec  Yes ()     /usr/lib/libboost_date_time.so.1.66.0 0xb6a46e98  0xb6aa66ec  Yes ()     /usr/lib/libboost_log_setup.so.1.66.0
 0xb6a10808  0xb6a1d16c  Yes ()     /usr/lib/libboost_filesystem.so.1.66.0 0xb6972600  0xb69ed224  Yes ()     /usr/lib/libboost_regex.so.1.66.0
 0xb691f4e8  0xb6921950  Yes ()     /usr/lib/libboost_chrono.so.1.66.0 0xb690b630  0xb690b7d8  Yes ()     /usr/lib/libboost_atomic.so.1.66.0
 0xb6823fa0  0xb68df07c  Yes ()     /lib/libstdc++.so.6 0xb672a1e8  0xb675b9f8  Yes ()     /lib/libm.so.6
 0xb6703450  0xb6711480  Yes (*)     /lib/libgcc_s.so.1

The crashing object is libhal.so, and the address it’s loaded in is 0xb6c7d328. This is what we’ll use for add-symbol-file.

(gdb) add-symbol-file /usr/lib/debug/libhal.so.dbg 0xb6c7d328 add symbol table from file "/usr/lib/debug/libhal.so.dbg" at .text_addr = 0xb6c7d328 (y or n) y Reading symbols from /usr/lib/debug/libhal.so.dbg..

When you run backtrace now, you’ll see the lines of the source code.

(gdb) bt
 0  0xb65dd368 in raise () from /lib/libc.so.6
 1  0xb65c91c8 in abort () from /lib/libc.so.6
 2  0xb6829db0 in __gnu_cxx::__verbose_terminate_handler() ()
 from /lib/libstdc++.so.6
 3  0xb6827aec in __cxxabiv1::__terminate(void (*)()) ()
 from /lib/libstdc++.so.6
 4  0xb6827b60 in std::terminate() () from /lib/libstdc++.so.6
 5  0xb6827ec8 in __cxa_throw () from /lib/libstdc++.so.6
 6  0xb6855358 in std::__throw_bad_function_call() () from /lib/libstdc++.so.6
 7  0xb6cb3214 in std::function<...>::o
 perator()() const (
     __args#0=…, this=)
     at /home/happycactus/...../ext-toolchain/arm-linux-gnueabihf/include/c++/
 8.2.1/bits/std_function.h:682
 8  bm3::hal::VariantFactory<...>::create
  (ts#0=…, index=)
     at /home/happycactus/....os/hal/lib/include/hal/modules/proto/Factories.h:71
 9  bm3::hal::details::createFromFactory<...> (buffer=…)
     at /home/happycactus/..../os/hal/lib/include/hal/modules/proto/VariantTypes.h:54

That’s all. Some thing must be considered:

embedded systems have constrained resources. Not only storage space: also memory. And symbol tables eat a lot of resources. You’ll not be able to work as you do on your development box, but perhaps you’ll be able to make some debugging. You should load only the symbol table you absolutely need. * Symbol table can be discarded. Use remove-symbol-file to free up resources. * Post-mortem debugging with core files is often an option, even on the development Box. Just setup your board to produce core dumps. * It is possible, using the toolchain tools, to recover the source line from addresses. objdump, addr2line, readelf and nm are your friend. * also remote debugging is an option, if you have a network interface (I hadn’t in my case).

Happy Coding, or, in this case, happy debugging!