Remote Debugging EMAC OE SDK Projects with gdbserver

Table 1: Conventions
target_program	The name of the application being debugged. This is the result of the Makefile build process.
target_machine	Connection information for the target machine. This can either be a serial port (ie. /dev/ttyS2) or a TCP connection in the form of HOST:PORT.
/path/to/sdk/	Represents the development system path to the EMAC OE SDK.

Sometimes a program has no technical errors that cause the compile to fail, but fails to meet the developer's expectations when run. This is typically due to algorithm or data structure design errors which can be difficult to find with just visual inspection of the code. Because of this, it can be beneficial to run a debugger targeting the binary resulting from the compile process. Debugging is the process of watching what is going on inside of another program while it is running. When a program is compiled with debug symbols included in the binary, it is possible to observe the source code and corresponding assembly while running the debugger.

When working with embedded systems the binary is usually compiled on a development machine with a different CPU architecture than what is on the target machine. This can be a problem when, as is typically the case, the target machine lacks the system resources to run a debugger. In these cases, it is possible to use the GNU debugger, or GDB, on the development machine to remotely debug the target machine provided it has a program called gdbserver. All EMAC OE builds are packaged with gdbserver to simplify the setup process for developers.

This guide is intended to build a basic understanding of how to use gdbserver with EMAC products. It is not intended as a general guide to debugging computer programs. For help with that, see the GDB man pages on the development system or read [this manual] on debugging with GDB.

Setup

Using gdbserver involves setting up both the target machine and the development machine. This requires that the binary application be present on both development and target machines. The development machine copy of the application must be compiled with debug flags whereas this is not strictly necessary for the target machine. See the [Optional global.properties Modifications Section] on the New EMAC OE SDK Project Guide for more information. See the [[[EMAC OE Getting Started Guide]]] for more information on how to connect to the target EMAC product using a serial port or Ethernet connection.

Target Machine

Because EMAC OE builds are distributed with gdbserver, installation is not a concern. The only setup necessary is to run gdbserver with target_program:

1. If the target application is already running, use the attachpid option to connect gdbserver to the application as shown below. The PID argument can be determined using pidof.

developer@ldc:~$ pidof target_program
developer@ldc:~$ gdbserver target_machine --attach PID

1. If the target application is not already running, the name of the binary may be included as an argument to the gdbserver program call.

<snytaxhighlight lang="bash"> developer@ldc:~$ gdbserver target_machine target_program [ARGS] </syntaxhighlight>

This establishes a gdbserver port on the target machine that listens for incoming connections from GDB on the development machine. In debug terminology, gdbserver is “attached” to the process ID of the program being debugged. In reality, though, GDB is attached to the process ID of a proxy which passes the messages to and from the remote device under test.

The next step is to run GDB on the development machine using the target_program/

Development Machine

1. First, cd to the directory where the targe executable is stored.
2. Run the EMAC OE SDK GDB:

developer@ldc:~$ /path/to/sdk/EMAC-OE-arm-linux-gnueabi-SDK_4.0/gcc-4.2.4-arm-linux-gnueabi/bin/arm-linux-gnueabi-gdb target_program

1. Run the following commands in GDB to prepare for the debug session:

(gdb) target remote target_machine

Sample GDB Session

This example GDB session uses the EMAC OE SDK example project named pthread_demo. It consists of the single source file pthread_demo.c. The program is called with a single integer argument indicating how many reader threads the user wishes to create. The following describes the tasks of the main thread:

1. The main thread performs user input validation. It prints a usage message according to the argument passed to it on the command line. The function expects the user to pass a number indicating how many threads should be spawned.
2. The main thread initiates a new thread which uses the generator() function to perform the following tasks:
   1. Checks to see if the number of reader threads matches the number of times a reader thread has acquired the mutex lock and performed its task. If the two values do match, then the generator thread unlocks the mutex, breaks out of the while loop and moves on to line 167 to gracefully exit. If the two values do not match, then the generator thread continues through the rest of the while loop described in steps 2.2 and 2.3.
   2. Generates random data to be stored in the data struct shared by all the threads. To do this, it protects the data struct with the use of a mutex variable.
   3. Sleeps after giving up its lock on the mutex so that another thread might have a chance to acquire the lock.
3. After creating the generator thread the main thread iteratively creates as many reader threads as indicated by the single integer argument. Each reader thread performs the following tasks:
   1. Waits for a chance to acquire the mutex lock. Once the mutex lock is acquired, it prints the value of the random number generated by the generator thread in its last run.
   2. Increments an integer in the data struct to indicate that it has completed its task.
   3. Gives up its lock on the mutex and exits.
4. After creating the prescribed number of reader threads, the main thread then waits for each thread created to exit gracefully.
5. The main thread exists.

The SDK version of pthread_demo.c works according to the description above with a MAX_THREAD value of 100. However, for the purpose of this example debug session it is instructive to use a faulty version of the same program. Replace lines 75-80 in pthread_demo.c with the code snippet shown in Listing 1 below.

if ((data.num_threads < 1) || (data.num_threads < MAX_THREAD)) {
        fprintf(stderr,
                "The number of thread should between 1 and %d\n",
                MAX_THREAD);
        exit(EXIT_FAILURE);
}

Useful GDB Commands

The following is a brief description of some essential GDB commands. Each description is followed by a link to the official GDB documentation page that has more specific information about what the command does and how to use it. Please note that the official GDB documentation is targeted for the latest GDB release which at the time of writing this documentation is 7.4. The version of GDB that EMAC distributes with the OE products, however, is version 6.8. Because of this, the links to documentation below may provide slightly different information. The biggest difference between the two version of GDB, however, is in the support for debugging programs with multiple threads. This is reflected in the documentation as well. Because of this, EMAC has set up ftp access to GDB 6.8 documentation on its web server. It is highly recommended that the GDB 6.8 documentation be referenced in cases where the program does not seem to support commands or options specified in the current official documentation.

Command	Description
start/run	These commands are used to start the debugged program with the only difference being that start automatically pauses execution at the beginning of the program's main function whereas run must be told explicitly where to pause using the breakpoint command listed below. See also [Debugging with GDB, Section 4.2: Starting your Program]
kill	Used to kill the currently-running instance of target_program. See also [Debugging with GDB, Section 4.9: Killing the Child Process]
print	Used to print the value of an expression. See also [Debugging with GDB, Section 10: Examining Data]
list	List contents of function or specified line. See also [Debugging with GDB, Section 9: Examining Source Files]
layout	This is a TUI (Text User Interface) command that enables the programmer to view multiple debug views at once including source code, assembly, and registers. See also [Debugging with GDB, Section 25.4: TUI Commands]
disassemble	This command allows the programmer to see assembler instructions. See also [Debugging with GDB, Section 9.6: Source and Machine Code]
break	This command specifies a function name, line number, or instruction at which GDB is to pause execution. See also [Debugging with GDB, Section 5.1: Breakpoints]
next/nexti, step/stepi	Allow the programmer to step through a program without specifying breakpoints. The next/nexti commands step over function calls, stopping on the next line of the same stack frame; step/stepi, step into function calls, stopping on the first line in the next stack frame. The difference between step/next and stepi/nexti is that the i indicates instruction-by-instruction stepping at the assembly language level. See also [Debugging with GDB, Section 5.2: Continuing and Stepping]
continue	Used to continue program execution from the address where it was last stopped. See the Debugging with GDB link for next/step for more information about the continue command.
bt	Short for "backtrace," which displays to the programmer a brief summary of execution up to the current point in the program. This is useful because it shows a nested list of stack frames starting with the current one. See also [Debugging with GDB, Section 8.2: Backtrace]
quit	This will quit the debugging session, and return you to the shell. The Control-D key combination is another way to accomplish this.