View source for Remote Debugging EMAC OE SDK Projects with gdbserver DEBUG 2019

{{todo|SEOKWREV (11.07.13-15:25->MD:reassigned to KY)(11.18.13-14:20->KY+);(11.19.13-20:05->MD+);(03.04.14-16:20->BS-);(03.20.14-15:55->BS+);(04.14.14-14:45->BS+)|Klint Youngmeyer|project=oe 4,oe 5,ky,md,bs,SEOKWREV}}

{{#seo:
|title=Remote Debugging EMAC OE SDK Projects with gdbsever
|titlemode=append
|keywords=EMAC Remote Debugging,gdbserver,GDB Commands
|description=Remote Debugging EMAC OE SDK Projects with gdbserver.
}}
{{EMAC OE SDK Conventions}}

Sometimes a program has no technical errors that cause the compile to fail, but fails to meet the developer's expectations when run. This is typically due to algorithm or data structure design errors which can be difficult to find with just visual inspection of the code. Because of this, it can be beneficial to run a debugger targeting the executable process. Debugging is the process of watching what is going on inside of another program while it is running. When a program is compiled with debug symbols included in the binary, it is possible to observe the source code and corresponding assembly code while running the debugger.

When working with embedded systems the binary is usually compiled on a development machine with a different CPU architecture than what is on the target machine. This can be a problem when, as is typically the case, the target machine lacks the system resources to run a debugger. In these cases, it is possible to use the GNU debugger, or GDB, on the development machine to remotely debug the target machine provided it has a program called gdbserver. All EMAC OE builds are packaged with gdbserver to simplify the setup process for developers.

This guide is intended to build a basic understanding of how to use gdbserver with EMAC products. It is not intended as a general guide to debugging computer programs. For help with that, see the GDB man pages on the development system or read [http://sourceware.org/gdb/current/onlinedocs/gdb.html this manual] on debugging with GDB.

==Setup==

Using <code>gdbserver</code> involves setting up both the target machine and the development machine. This requires that the binary application be present on both development and target machines. The development machine copy of the application must be compiled with debug flags whereas this is not strictly necessary for the target machine. See the [[Creating_a_New_EMAC_OE_SDK_Project#Optional_global.properties_Modifications | Optional <code>global.properties</code> Modifications Section]] on the New EMAC OE SDK Project Guide for more information. See the [[Serial Connections]] or [[Network Connections]] page for more information on how to connect to the target EMAC product using a serial port or Ethernet connection.

===Building a Debug-friendly Executable for the Target Device=== 

A debug-friendly executable must be constructed, before remote debugging can occur.

<cl>
1. This section demonstrates how to use the EMAC CMake tool to generate CMake files automatically for a project. When using the EMAC SDK there are currently two options for cross compiling:
*arm
*x86

For the purposes of this guide, the <tt>arm</tt> option will be used for the listed examples.

Navigate to the directory where the project will be located. Then run the CMake new project script.
{{clo}}
{{clio | username=developer | hostname=ldc | pwd=~ |cd projects }}
{{clio | username=developer | hostname=ldc | pwd=~/projects | 1 = export PATH=/opt/emac/5.1/bin:$PATH}}
{{clio | username=developer | hostname=ldc | pwd=~/projects |oe_init_project -n hello.c }}
If desired, please enter a name for this project, otherwise press Enter to use the default: hello_emac <br />

-- Creating new project directory... <br />
-- Creating new source file... <br />
-- Building custom CMakeLists.txt file... <br />
-- Done. <br />

Do you want to create a build directory for this project? (y/n) y <br />

-- Creating build directory... <br />

Do you want to run cmake for this project? (y/n) y <br />

-- Using system compiler <br />
-- The C compiler identification is GNU 4.8.2 <br />
-- The CXX compiler identification is GNU 4.8.2 <br />
-- Check for working C compiler: /usr/bin/cc <br />
-- Check for working C compiler: /usr/bin/cc -- works <br />
-- Detecting C compiler ABI info <br />
-- Detecting C compiler ABI info - done <br />
-- Check for working CXX compiler: /usr/bin/c++ <br />
-- Check for working CXX compiler: /usr/bin/c++ -- works <br />
-- Detecting CXX compiler ABI info <br />
-- Detecting CXX compiler ABI info - done <br />
-- Configuring done <br />
-- Generating done <br />
-- Build files have been written to: /home/developer/projects/hello_emac/hello_emac-build <br />

Do you want to compile this project? (y/n) y

Scanning dependencies of target hello_emac <br />
[100%] Building C object CMakeFiles/hello_emac.dir/hello.c.o <br />
Linking C executable hello_emac <br />
[100%] Built target hello_emac <br />
{{clos}}

The executable, in this case,  is now inside the <code> hello_emac/hello_emac-build </code> directory.


* In the terminal navigate to the base directory of the project

{{note | If the target board being used is <tt>x86</tt>, then change all occurrences of <tt>arm</tt> in the following sections below to <tt>x86</tt>.}}


* Create a build directory for cross compiling
{{cli | username=developer | hostname=ldc | pwd=~/projects/hello_emac | mkdir hello_emac-build-arm }}
* Change directories into the newly created directory
{{cli | username=developer | hostname=ldc | pwd=~/projects/hello_emac | cd hello_emac-build-arm }}
* Run cmake using the target achitecture
{{cli | username=developer | hostname=ldc | pwd=~/projects/hello_emac/hello_emac-build-arm | 1 = cmake .. -DARCH:STRING=arm -DCMAKE_BUILD_TYPE{{=}}Debug }} 
* Compile the code using make
{{cli | username=developer | hostname=ldc | pwd=~/projects/hello_emac/hello_emac-build-arm | make }} 

{{note | Using cmake without specifying the build-type results in the project being built with the ''last'' build-type specified. For this reason it is recommended to always specify the build-type, '''-DCMAKE_BUILD_TYPE{{=}}build-type'''  when using cmake. Build-type options are ''Release'', ''Debug'', ''RelWithDebInfo'', and ''MinSizeRel''.}}
* If the '''target device's''' IP address is unknown, it can be found by using the following command on the target terminal
{{cli | ifconfig | hostname=ipac9x25}}
* If required, set the '''target device's''' file system to read write (Required when transferring the executable to any directory other than the ''tmp'' or ''home'' directories)
{{cli | oemntrw | hostname=ipac9x25}}
* Assuming an IP address of ''10.0.6.221'', from the '''development machine''' transfer the executable (More transfer options are given here)
{{cli | username=developer | hostname=ldc | pwd=~/projects/hello_emac/hello_emac-build-arm | scp hello_emac root@10.0.6.221: }}

</cl>

===Target Machine===

Because EMAC OE builds are distributed with <code>gdbserver</code>, installation is not a concern. The only setup necessary is to run <code>gdbserver</code> with <code>target_program</code>: 



If the target application is not already running, the name of the binary may be included as an argument to the <code>gdbserver</code> program call.
<syntaxhighlight lang="console">
root@emac-oe:~# gdbserver :2159 target_program [ARGS]
</syntaxhighlight>
Where '''2159''' is the registered TCP port number for remote GDB


This establishes a <code>gdbserver</code> port on the target machine that listens for incoming connections from GDB on the development machine. In debug terminology, <code>gdbserver</code> is “attached” to the process ID of the program being debugged. In reality, though, GDB is attached to the process ID of a proxy which passes the messages to and from the remote device under test. 
<br />
<cl>
1. Continuing with our example target build, <code>cd</code> to the directory where the target executable is stored. In our example this is the home directory ('''~''')

* input the following
{{cli | gdbserver :2159 hello_emac | hostname=ipac9x25}}

The next step is to run GDB on the development machine using the <code>target_program</code>/
</cl>
===Development Machine===

<cl>
1. First, <code>cd</code> to the directory where the target executable is stored.
* Run EMAC OE SDK GDB:
<font size = 1> 
{{cli | username=developer |hostname=ldc |pwd=~/projects/hello_emac/hello_emac-build-arm |/opt/emac/5.1/sysroots/x86_64-emacsdk-linux/usr/bin/arm-emac-linux-gnueabi/arm-emac-linux-gnueabi-gdb hello_emac}}
</font>
* Assuming an IP address of ''10.0.6.221'', run the following commands in GDB to connect to the target device and establish a debug session:
<syntaxhighlight lang="gdb">
(gdb) target remote 10.0.6.221:2159
</syntaxhighlight>
 
</cl>
The gdb debug session is now established.

{{mbox | type = warning | text = Note that the location of GDB in the toolchain may differ from what is shown above depending on which version of the SDK is used.}}

<br clear=all>{{mbox | type = notice | text = If the gdb executable, or any other executable you run, is located in a directory which is in the PATH environment variable, you can simply run that command without the long path prefix. Sourcing the environment variables generated by the EMAC script for this will provide you with such a path. The script which creates the file to source also creates a symbolic link for arm-linux-gnueabi-gdb called, simply, gdb. With your shell environment setup this way, you could simply execute: 
<syntaxhighlight lang="console">
developer@ldc:~$ gdb target_program
</syntaxhighlight>

in place of the long command given in step 2, above.
}}


==Sample GDB Session==

This example GDB session uses the EMAC OE SDK example project named <code>pthread_demo</code>. It consists of the single source file <code>pthread_demo.c</code>. The program is called with a single integer argument indicating how many <code>reader</code> threads the user wishes to create. The following describes the tasks of the <code>main</code> thread: 

<cl>
1. The <code>main</code> thread performs user input validation. It prints a usage message according to the argument passed to it on the command line. The function expects the user to pass a number indicating how many threads should be spawned.
* The <code>main</code> thread initiates a new thread which uses the <code>generator()</code> function to perform the following tasks:
 i. Checks to see if the number of <code>reader</code> threads matches the number of times a <code>reader</code> thread has acquired the mutex lock and performed its task. If the two values do match, then the <code>generator</code> thread unlocks the mutex, breaks out of the while loop and moves on to line 167 to gracefully exit. If the two values do not match, then the <code>generator</code> thread continues through the rest of the while loop described in steps 2.2 and 2.3.
 * Generates random data to be stored in the data struct shared by all the threads. To do this, it protects the data struct with the use of a mutex variable.
 * Sleeps after giving up its lock on the mutex so that another thread might have a chance to acquire the lock.
* After creating the <code>generator</code> thread the <code>main</code> thread iteratively creates as many <code>reader</code> threads as indicated by the single integer argument. Each <code>reader</code> thread performs the following tasks:
 i. Waits for a chance to acquire the mutex <code>lock</code>. Once the mutex <code>lock</code> is acquired, it prints the value of the random number <code>generated</code> by the generator thread in its last run.
 * Increments an integer in the <code>data</code> struct to indicate that it has completed its task.
 * Gives up its lock on the mutex and exits.
* After creating the prescribed number of <code>reader</code> threads, the <code>main</code> thread then waits for each thread created to exit gracefully. 
* The <code>main</code> thread exists.
</cl>

The SDK version of <code>pthread_demo.c</code> works according to the description above with a <code>MAX_THREAD</code> value of 100. However, for the purpose of this example debug session it is instructive to use a faulty version of the same program. Replace lines 75-80 in <code>pthread_demo.c</code> with the code snippet shown in Listing 1 below.

<syntaxhighlight lang="c">
if ((data.num_threads < 1) || (data.num_threads < MAX_THREAD)) {
        fprintf(stderr,
                "The number of thread should between 1 and %d\n",
                MAX_THREAD);
        exit(EXIT_FAILURE);
}
</syntaxhighlight>

===Useful GDB Commands===

The following is a brief description of some essential GDB commands. Each description is followed by a link to the official GDB documentation page that has more specific information about what the command does and how to use it. Please note that the official GDB documentation is targeted for the latest GDB release which at the time of writing this documentation is 7.6.1. The version of GDB that EMAC distributes with the OE products, however, is version 6.8. Because of this, the links to documentation below may provide slightly different information. The biggest difference between the two version of GDB, however, is in the support for debugging programs with multiple threads. This is reflected in the documentation as well. Because of this, EMAC has set up ftp access to GDB 6.8 documentation on its web server. It is highly recommended that the GDB 6.8 documentation be referenced in cases where the program does not seem to support commands or options specified in the current official documentation. 

{| class="wikitable"
! Command !! Description 
|-
| <code>start/run</code>
|| These commands are used to start the debugged program with the only difference being that start automatically pauses execution at the beginning of the program's main function whereas run must be told explicitly where to pause using the breakpoint command listed below.
See also [Debugging with GDB, Section 4.2: Starting your Program]
|-
| <code>kill</code>
|| Used to kill the currently-running instance of target_program.
See also [Debugging with GDB, Section 4.9: Killing the Child Process]
|-
| <code>print</code>
|| Used to print the value of an expression.
See also [Debugging with GDB, Section 10: Examining Data]
|-
| <code>list</code>
|| List contents of function or specified line.
See also [Debugging with GDB, Section 9: Examining Source Files]
|-
| <code>layout</code>
|| This is a TUI (Text User Interface) command that enables the programmer to view multiple debug views at once including source code, assembly, and registers.
See also [Debugging with GDB, Section 25.4: TUI Commands]
|-
| <code>disassemble</code>
|| This command allows the programmer to see assembler instructions.
See also [Debugging with GDB, Section 9.6: Source and Machine Code]
|-
| <code>break</code>
|| This command specifies a function name, line number, or instruction at which GDB is to pause execution.
See also [Debugging with GDB, Section 5.1: Breakpoints]
|-
| <code>next/nexti, step/stepi</code>
|| Allow the programmer to step through a program without specifying breakpoints. The <code>next/nexti</code> commands step over function calls, stopping on the next line of the same stack frame; <code>step/stepi</code>, step into function calls, stopping on the first line in the next stack frame. The difference between <code>step/next</code> and <code>stepi/nexti</code> is that the <code>i</code> indicates instruction-by-instruction stepping at the assembly language level.
See also [Debugging with GDB, Section 5.2: Continuing and Stepping]
|-
| <code>continue</code>
|| Used to continue program execution from the address where it was last stopped.
See the Debugging with GDB link for <code>next/step</code> for more information about the <code>continue</code> command.
|-
| <code>bt</code>
|| Short for "backtrace," which displays to the programmer a brief summary of execution up to the current point in the program. This is useful because it shows a nested list of stack frames starting with the current one.
See also [Debugging with GDB, Section 8.2: Backtrace]
|-
| <code>quit</code>
||  This will quit the debugging session, and return you to the shell. The Control-D key combination is another way to accomplish this.
|}

===Session Walk-through===

This debug session walk-through assumes that the program has been compiled using the modified source code above and that both the target machine and the development machine have been set up according to the above Setup section. The walk-through is divided into multiple “lessons” with the intent of first introducing the use of the commands described above and then actually running GDB to debug a known programming problem. Each lesson may be run independently of the others, but it is recommended that each be run in order starting from Lesson 1 for the first time through.

====Lesson 1: Navigation and Code Display====

This lesson assumes that <code>gdbserver</code> has been run as in the [[#TargetMachine | Target Machine Setup]] section above with an ARG value of 3. Other values are fine so long as they fall within the range of 1 to 100. The number '3' was arbitrarily chosen to avoid having to use a symbolic variable in the explanations below.

<cl>
1. Type <code>b main</code> to set a breakpoint at the main function in the source code.
* Type <code>continue</code>. This will cause the program to continue from the breakpoint set by GDB at startup. The program was passed an argument of 3, indicating that three threads should be created. 
* Type <code>b 73</code> to set a breakpoint at line 73 in the source code, which should be the line containing <code>data.num_threads = atoi(argv[1]);</code>
* Type <code>continue</code>. The program will continue execution up until line 73 in the source code. At this point, type <code>layout split</code> to view a split screen containing both the source code and the assembly-level machine instructions. Both screens show the program's current location in execution. The assembly-level display shows what the target's processor is actually executing at that point in the source code as shown in the source-level display. To view either of these without the other type <code>layout asm</code> for just assembly-level and <code>layout src</code> for just source-level.
* Type <code>nexti</code>. This will cause the program to execute the next instruction in the current stack frame, which is a mov instruction that starts preparations on the current stack for a call to the library function <code>atoi()</code>. The details of this process are beyond the scope of this tutorial; essentially, the program needs to store information about the current execution location in the stack for when the <code>atoi()</code> function finishes. Type <code>ni</code> (alias for nexti) three more times. You should end up on a <code>bl</code> instruction in the assembly view as shown in Listing 2 below. The source layout should still show the program on line 73.
<syntaxhighlight lang="asm">
B+ |0x887c <main+112>       ldr    r3, [r11, #-84]                   │
   |0x8880 <main+116>       add    r3, r3, #4      ; 0x4             │
   |0x8884 <main+120>       ldr    r3, [r3]                          │
   |0x8888 <main+124>       mov    r0, r3                            │
  >|0x888c <main+128>       bl     0x86e0 <atoi>                     │
</syntaxhighlight>
'''Listing 2. GDB Assembly Layout''' - ''Note that the assembly may look different depending on the target architecture.''
* Type <code>stepi</code>. This will cause the program to move into the next stack frame and GDB to show the assembly-level instructions of the <code>atoi()</code> call. Since the library containing <code>atoi()</code> was likely not compiled with debug symbols, the source-level layout will show the message <code>[ No Source Available ]</code>.
* Type <code>bt</code>. This will cause the program to display a human-readable version of the current stack. Each stack “frame” is represented by the name of the function call it represents with that function's location in memory. Type <code>bt full</code> to get a list of the variables local to each stack frame.
* Type <code>finish</code>. This will cause the current stack frame to return and execution to pause on the next instruction of the previous stack frame.
* Type <code>kill</code>. This will cause the current process to be killed by <code>gdbserver</code> at the target machine. <code>gdbserver</code> will also terminate at this point. In order to start a new remote debug session, start gdbserver as described in the Target Machine Setup section and re-run step 3 of the [[#Development Machine | Development Machine Setup]] section.
</cl>

{{mbox | type = warning | text = '''Note:''' <code>b</code> is an alias for the <code>break</code> command. 
<code>ni</code> is an alias for <code>nexti</code>}}

====Lesson 2: Finding the Bug====

Though this sample is contrived, it is still useful to demonstrate how to find a design mistake in an otherwise well-written (no errors or warnings) program. These types of mistakes typically have to do with the array boundary miscalculations, logic and comparison operator mistakes, or other simple mistakes. For the sake of demonstration, assume that the actual mistake is unknown. This lesson assumes that gdbserver has just been started as in the [[#TargetMachine | Target Machine Setup]] above with an <code>ARG</code> value of 5.

<cl>
1. Before starting the program in the debugger again, run it by itself on the target machine to see what the actual program output is: 
<syntaxhighlight lang="bash">
root@emac-oe:~# /tmp/pthread_demo 5
The number of threads should be between 1 and 100
</syntaxhighlight>
The program was given an input of '5' yet the output message seems to indicate that this is out of range which is obviously not true.
* Start the debugger again and connect to the target machine as described in the Setup section.
* Type <code>b main</code> to set a breakpoint at the main function in the source code.
* Type <code>continue</code>. This will cause the program to continue from the breakpoint set by GDB at startup. 
* Type <code>n</code>. This will cause the program to step over the next line of source code. The reason for using <code>n</code> rather than <code>s</code> or one of the instruction stepping commands is because the erroneous output indicates that the coding mistake is in the programmer's source code rather than the c library functions <code>atoi()</code> or <code>fprintf()</code>. Stepping over the function will save all the time required to step through every detail of what the library functions are doing. Later passes through the code can be used to step into functions called from within that stack frame if the first pass proves unsuccessful.
* Continue to type <code>n</code> until one of the program's <code>exit()</code> calls is reached, but do not actually step into that <code>exit()</code> call. Judging by the program's output above, this should bring you to the conditional block that checks the value of the local variable n used to store the output of <code>atoi()</code> as shown in Listing 3. Note that once execution reaches line 79 of the source code, GDB will display the output of the <code>fprintf()</code> function from line 76. This may cause display problems within the text-based UI library that GDB uses which will require the command refresh to fix.
<syntaxhighlight lang="gdb">
B+ |75              if ((data.num_threads < 1) || (data.num_threads < MAX_THREAD)) {       |
   |76                      fprintf(stderr,                                                |
   |77                              "The number of thread should between 1 and %d\n",      |
   |78                              MAX_THREAD);                                           |
  >|79                      exit(EXIT_FAILURE);                                            |
   |80              }                                                                      |
</syntaxhighlight>
* Type <code>p/d data-&gt;num_threads</code>. The letter <code>p</code> is an alias for <code>print</code>, the switch <code>/d</code> tells GDB to treat the expression requested as an integer in signed decimal, and <code>data-&gt;num_threads</code> refers to the element <code>num_threads</code> within <code>struct thread_data</code>, as would be expected. This should provide the following output:
<syntaxhighlight lang="gdb">
(gdb) p/d data->num_threads
$6 = 5
</syntaxhighlight>
Note that the integer part of <code>$6</code> will increment with each call to the gdb command <code>print</code>. The above output confirms that the argument '5' was successfully passed to the program and read into a variable to be tested, indicating that one of the logical tests for the current conditional block contains a mistake. This merits a closer look at line 75:
<syntaxhighlight lang="gdb">
B+ |75              if ((data.num_threads < 1) || (data.num_threads < MAX_THREAD)) {       |
</syntaxhighlight>
Line 75 consists of a conditional test which is the logical OR of two arithmetic tests involving the values of <code>data.num_threads</code>, '1', and <code>MAX_THREAD</code>. The first test is true the input integer is less than <code>1–(data.num_threads &lt; 1)</code>. The second tests whether the input integer is less than the symbolic constant, <code>MAX_THREAD–(data.num_threads &lt; MAX_THREAD)</code>. Judging by the name of this constant and the result of the test (we know it resolves to true because the value of <code>data.num_threads</code> in this case is not less than one), we can see that the comparison operator used is the culprit. The correct interpretation is that it should be '&gt;' rather than '&lt;'.
* Type <code>kill</code>.
</cl>

This was a simple problem to solve but the method used above could apply in any situation where source code compiles and runs without errors yet provides varied or unexpected output.

====Lesson 3: Debugging With Threads====

Do not fix the programming mistake found in Lesson 2. This lesson will cover the use of the jump command to skip blocks of code and commands specific to debugging multi-threaded programs. Before getting started, see [http://sourceware.org/gdb/current/onlinedocs/gdb/Thread-Stops.html#Thread-Stops Debugging with GDB: 5. Stopping and Starting Multi-thread Programs].

This lesson assumes that <code>gdbserver</code> has just been started as in the Target Machine Setup above with an <code>ARG</code> value of 7.

<cl>
1. Start the debugger again and connect to the target machine as described in the Setup section.
* Type set <code>scheduler-locking on</code>. This command enables GDB to lock all threads save for the currently-selected thread from running when the <code>step/stepi</code> or <code>next/nexti</code> commands are given. 
* Set all the breakpoints that you will need for this session: 
 i. Type <code>b main</code> to set a breakpoint at the main function in the source code.
 * Type <code>b 75</code>. This will set a break point at the conditional block that checks the value of the program's single integer argument. If you recall from Lesson 2, this is the conditional which evaluates incorrectly in the modified version of the application.
 * Type <code>b 143</code>. This will set a breakpoint in the variable assignment in the <code>generator</code> function. Careful examination of the source code will show that this function is called from a thread created by the main thread of execution but never from the main thread itself.
 * Type <code>b 129</code>. This will set a breakpoint in the variable assignment of the <code>reader</code> function. As with the <code>generator</code> function, any time the <code>reader</code> function is called it will be inside a thread that is not the main thread.
 * Type <code>b 135</code>. This will set a breakpoint just after the <code>fflush(stdout)</code> statement in the <code>reader</code> function.
 * Type <code>b 97</code>. This will set a breakpoint in the main thread after the <code>generator</code> thread has been created but before the main thread begins creating <code>reader</code> threads.
 * Type <code>b 119</code>. This will set a breakpoint after the main thread iteratively creates the <code>reader</code> threads.
* Optional: You may want to run the <code>layout split</code> command so that you can see both the assembly and the source code during the debug session.
* Type <code>continue</code> then hit 'Enter' once. This will bring you to line 75 in the source code.
* Type <code>j 81</code>. This is an alias for <code>jump 81</code> that tells GDB to have the program jump to line 81 of the source code and resume execution at the first assembly instruction represented by line 81 of the source code. This line is labeled <code><main.c+196></code>. Note that the program effectively no longer checks the input it receives.
* Type <code>i th</code>. This will cause GDB to display a list of the application's threads currently in memory. Take a moment to consider what is happening in the program. We know that in Step 2 of this lesson we used <code>set scheduler-locking</code> to tell GDB to effectively only allow the currently-selected thread of execution to be affected by the GDB <code>step</code> and <code>next</code> commands. Others will wait at their respective breakpoints until explicitly told by the programmer to execute the next line of source code or instruction. The next breakpoint that <code>main</code> reaches occurs after the <code>generator</code> thread is created. This means that there are currently two threads of execution, the <code>main</code> thread paused at line 97 and the <code>generator</code> thread paused at line 143. 
* Type <code>thread 2</code>. This will select the <code>generator</code> thread.
* Type <code>thread apply 2 n</code>. This will tell the <code>generator</code> thread to execute the next line of source code and pause again on the line following that. Without typing any other commands into the GDB prompt, hit 'Enter' seven more times. This should bring you to line 165 of the source code:
<syntaxhighlight lang="c">
usleep(1);
</syntaxhighlight>
Notice the output of the program on the remote terminal on which <code>gdbserver</code> was run. Standard output on that terminal should show the output from the <code>printf()</code> call on line 154.
* Type <code>thread 1</code>. This will select the <code>main</code> thread.
* Type <code>continue</code>. This will cause the <code>main</code> thread to continue execution while <code>generator</code> remains paused at line 165. <code>main</code> pauses again at line 119, once the 7 <code>reader</code> threads have been created. Recall from step 3.4 that <code>b 129</code> set a breakpoint in the <code>reader</code> function so that the <code>reader</code> threads would pause at line 129.
* Type <code>i th</code>. This is an alias for <code>info threads</code>. This causes GDB to print out all the threads currently in memory. Notice that there are three types of threads, <code>main</code>, <code>generator</code>, and <code>reader</code>. The <code>info threads</code> command also shows that the <code>reader</code> threads are all paused at line 129, the <code>generator</code> thread is paused at line 165, and the <code>main</code> thread is paused at line 119.
* Type <code>thread 5</code>. This will select the third <code>reader</code> thread.
* Type <code>thread apply 5 n</code>. Then press 'enter' seven times. This will cause the third <code>reader</code> thread to complete its task and exit gracefully using the <code>pthread_exit()</code> function.
* Type <code>thread 1</code>. This will select the <code>main</code> thread.
* Type <code>p data</code>. This will show the current state of the data structure that was passed to each thread. Note that each thread contains a pointer to the same data structure. This requires the use of what is known as a mutex (MUTual EXclusion) variable which is used to protect the data structure from concurrent modifications. In other words, any time the data structure is read or written to by one of the threads, they must first call the function pthread_<code>mutex_lock()</code> to ensure that no other thread currently “has” the lock. When a thread is done with the shared data structure, it calls the function <code>pthread_mutex_unlock()</code> to make the lock available to other threads. Notice that nothing about the mutex's inclusion in the data structure requires it to be used in order to read or write to the data structure. This means that any programmer who wishes to use threads to implement concurrent programming must pay close attention to code that accesses shared data structures to ensure concurrent modifications do not occur.
* Perform the previous four steps for as many of the <code>reader</code> threads as you want. Notice that each one prints a message to standard out providing information about the state of the shared <code>data</code> variable at the time that it has the lock. By switching to the <code>generator</code> thread once the mutex variable is unlocked, that code can be stepped through to generate a new random number for the data structure. '''IMPORTANT:''' Do not execute a line of source code containing a call to <code>pthread_mutex_lock()</code> without first ensuring that the mutex variable is unlocked. To do this, carefully perform the following steps:
 i. Type <code>p data-&gt;lock</code>. This will show the values of the mutex variable in the <code>data</code> structure variable. Make note of the value of the <code>owner</code> field.
 * Type <code>i th</code>. This will show the current list of threads in the program. What follows is a possible output from these two commands:
<syntaxhighlight lang="gdb">
(gdb) p data->lock
$3 = {__data = {__lock = 1, __count = 0, __owner = 1288, __kind = 0, __nusers = 1, {__spins = 0, __list = {__next = 0x0}}},
  __size = "\001\000\000\000\000\000\000\000\b\005\000\000\000\000\000\000\001\000\000\000\000\000\000", __align = 1}
(gdb) i th
  9 Thread 1289  reader (arg=0xbeddec8c) at pthread_demo.c:129
  8 Thread 1288  reader (arg=0xbeddec8c) at pthread_demo.c:134
  7 Thread 1287  reader (arg=0xbeddec8c) at pthread_demo.c:129
  6 Thread 1286  0x00008b04 in reader (arg=0xbeddec8c) at pthread_demo.c:129
  3 Thread 1283  0x00008b04 in reader (arg=0xbeddec8c) at pthread_demo.c:129
* 2 Thread 1282  generator (arg=0xbeddec8c) at pthread_demo.c:165
  1 Thread 1281  0x00008a64 in main (argc=2, argv=0xbeddee24) at pthread_demo.c:119
(gdb)
</syntaxhighlight>
Analysis of this output requires the understanding that the third column of the <code>i th</code> output indicates that particular thread's process ID. The important part of the <code>p data-&gt;lock</code> output is the <code>owner</code> field, whose value will always either be zero or correspond to the process ID of one of the currently-running threads. In this case, the <code>lock-&gt;__owner</code> field clearly indicates that thread 8 currently owns the <code>data-&gt;lock</code> mutex variable. This would indicate that thread 8 should be stepped through until it has called the <code>pthread_mutex_unlock()</code> function before stepping into a call to <code>pthread_mutex_lock()</code> in any other function.
To summarize, always ensure that the <code>owner</code> field of the mutex variable is equal to zero before using <code>pthread_mutex_lock()</code> while debugging. If the lock is currently owned by another thread, GDB will hang until sent an interrupt signal which will require that the entire debug process be started over.
* The walktrhough is complete. There are two ways to end the debug session gracefully:
 * Type <code>monitor exit</code>, <code>kill</code> (confirm with 'y'), then <code>quit</code>.
 * Type <code>set scheduler-locking off</code>, <code>delete</code>, <code>thread 1</code>, <code>continue</code>, then <code>monitor exit</code>, <code>kill</code> (confirm with 'y'), <code>quit</code>. This will allow the program to finish executing before ending the session.
</cl>

===GNU GDB Documentation===

Again, for more information on how to debug with GDB, refer to the [http://sourceware.org/gdb/current/onlinedocs/gdb.html GDB Manual]. This is a valuable resource for anyone just learning to debug software and will go into much greater detail than is possible in this guide.

==Next Steps==

If you have not done so already, be sure to check out the [[Using_EMAC_OE_SDK_Example_Projects | EMAC OE SDK Example Projects]] or learn to create your own [[Creating_a_New_EMAC_OE_SDK_Project | New Project]].

<!--[[Category:EMAC OE SDK]]-->