Custom Linux Kernel Development

TODO: {{#todo:Complete (03.18.14-13:40->MD+)(03.20.14-11:35->MD+);(03.27.14-16:25->BS+);(04.14.14-15:00->BS+);(11.10.15-15:30->MG+)(11.11.2015-18:15->MD+)|Travis Stratman|oe 5,TS,bs,Complete,mg,mw,md}}

The ability to easily customize and expand any portion of the kernel is a feature of Linux that makes it very well suited for embedded systems development. In the embedded environment, specialized hardware, protocols, and systems may require a look into the kernel internals, custom configuration, feature additions, or driver development. This article aims to provide information on the most common kernel development tasks for EMAC OE Linux systems as well as additional kernel development resources.

Prerequisites

This article assumes some basic familiarity with Linux and C programming competency.

Before continuing, you should ensure that git is installed on your development machine and review the Building the Linux Kernel document.

Background

The Linux kernel is the heart of the GNU/Linux OS and is developed by a community of thousands of contributors across the globe. Linux follows an evolutionary development model. The needs of the community of users and developers drive feature development, while Linus Torvalds -- the "father" of Linux -- integrates and releases these features and improvements into the mainline ("vanilla") kernel branch.

Linux is a monolithic kernel, meaning that the kernel is responsible for defining a complete interface to the system hardware. However, kernel drivers and features may be compiled as modules which may be loaded or unloaded dynamically on a running system. The kernel includes support for many different architectures, and has a feature set including preemption, virtual memory, shared libraries, virtualization, and complete networking stacks.

The Kernel Source

Source code for EMAC kernels is provided through our Git server. Refer to the documentation for your system to determine the correct source to use.

NOTE

For this guide, the 2.6.30-at91 kernel tree will be used. This kernel tree currently covers many of EMAC OE 4's SoM-based ARM products.

Clone the Git Repository

To clone the git repository over anonymous HTTP, run the following commands:

developer@ldc:~# git clone http://git.emacinc.com/public/source/linux-2.6.30-at91.git

Once the command has completed, the entire source should be contained in the linux-2.6.30-at91 directory. The master branch will be checked out automatically. Because EMAC uses the master branch for all releases, this is the correct branch to use, but other branches or tags may be checked out if directed or required.

Kernel Source Structure

Within the kernel source tree that was downloaded, you will see several top-level directories. Table 1 gives a brief description of each of these directories which will be referred to throughout this document.

Directory name	Description
arch	Architecture-specific kernel code with subdirectories for each architecture
block	Block device driver code
crypto	Encryption support algorithms
Documentation	Text documentation files on the kernel and APIs
drivers	Device driver code, with a subdirectory structure for each device type
firmware	Code for building firmware required to communicate with devices
fs	File systems code
include	Include (header) files required to build the kernel code
init	Kernel initialization code
ipc	Interprocess communications code
kernel	Main internal kernel code
lib	Library code
mm	Memory management code
net	Kernel networking code
samples	Code examples and drivers that have not been fully developed
scripts	Various scripts used for the configuration and build process as well as standalone utility scripts
security	Kernel security support
sound	Sound and audio driver code
usr	Code used during kernel image creation
virt	Virtualization support code

Configuration

The kernel uses a configuration system that specifies how every aspect of the kernel is built.

Configuration Structure

The kernel configuration is generated based on selections by the user combined with dependency information built into the Kconfig structure. You will find a file named Kconfig in most source directories within the kernel. These files follow the language described in Documentation/kbuild/kconfig-language.txt and control the structure and operation of the kernel configuration utility.

Once created, the current kernel configuration is stored in a file named .config in the top level of the kernel source tree. This file is read and updated by the configuration utility and is used by the make system during the build process. The first step in configuring the kernel often involves initializing this file with a default configuration providing sane values for the target platform. These default configuration files are stored under the arch directory. For example, arch/arm/configs/ holds default configuration files for ARM targets.

NOTE

Note that the .config file is a hidden file since its filename begins with a '.'. The ls utility will not show this file by default unless the -a option is passed to show hidden files.

In this example, copy the arch/arm/configs/som9g45-som210_defconfig file to the top level of the kernel source as .config:

developer@ldc:~# cp arch/arm/configs/som9g45-som210_defconfig .config

This is the default configuration file for the EMAC SoM-9G45 module using an SoM-210ES carrier board.

The configuration file defines all CONFIG values that are set. One of the following values can be used:

• y (yes)
• m (module)
• a numeric value
• a string value

Options not configured are entered into the file as a comment with the option followed by the phrase, "is not set." An example of each of these formats is shown below:

#
# Automatically generated make config: don't edit
# Linux kernel version: 2.6.30
# Tue Jan 10 17:42:58 2012
#
CONFIG_ARM=y
CONFIG_SYS_SUPPORTS_APM_EMULATION=y
CONFIG_GENERIC_GPIO=y
CONFIG_GENERIC_TIME=y
CONFIG_GENERIC_CLOCKEVENTS=y
CONFIG_MMU=y
# CONFIG_NO_IOPORT is not set
...
CONFIG_DEFCONFIG_LIST="/lib/modules/$UNAME_RELEASE/.config"
...
CONFIG_INIT_ENV_ARG_LIMIT=32
...
CONFIG_SDIO_UART=m

NOTE

While editing the configuration file by hand is not recommended because it is easy to break dependencies that the configuration utility will catch, viewing the configuration file as text can be helpful in quickly exposing the current configuration selections. The grep utility is often helpful when checking a kernel configuration file to determine the setting of a specific option.

Configuring the Kernel

The kernel supports several methods of generating configuration values including command line utilities, a menu-based utility, a Qt-based utility, and a GTK+-based utility. These utilities are instantiated as make targets. The menu-based utility, invoked by the menuconfig command, is the only option supported by the EMAC Linux kernel build script. See here for more information on how to initiate the configuration process using the EMAC Linux kernel build script.

Menuconfig

The top-level interface to the kernel menuconfig utility is shown in Figure 1. Note that the navigation keys and legend are shown at the top of the page.

Figure 1: Linux Kernel Menuconfig Main Page

Any menu item with a ---> will lead to a submenu with further options. For example, to set the EMAC carrier board used with this SoM, you would use the arrow keys to highlight System Type and press Enter to select. From this menu, select Atmel AT91 System-on-Chip followed by EMAC Carrier Board Selection. This will initiate a menu allowing a single selection with all of the carrier boards supported by the currently selected SoM module. If, for example, you wanted to target the SoM-200ES carrier board, you could highlight SOM-200ES Carrier Support and press the space bar or Enter to select it.

Once all desired changes have been made, select Exit and press enter from the main menu. You will be prompted to save the kernel configuration as shown in Figure 2.

Figure 2: Linux Kernel Menuconfig Save Confirmation

Building and Deployment

If configuration options are the only customization required for your project, you may continue by building the kernel. If only loadable module options were changed from the original configuration that your running kernel was built with, you will only need to load the modules onto the target system as described here. If other built-in options were modified, you will need to reload the kernel image as well, which requires a different procedure depending on the bootloader:

• U-Boot: Loading Images with U-Boot
• RedBoot: Loading Images with RedBoot
• LILO: Installing LILO

Adding Support for a New Carrier Board

One common kernel development task when working with EMAC systems is adding support for a new custom carrier board for use with an EMAC SoM. In general, the procedure for this task is to use one of the existing EMAC carrier board support files as a base and make adjustments as needed to support the custom hardware. Additions to the Makefiles and Kconfig are also generally required to add new selections to the kernel configuration and build system. This section covers an example of the work required for this using the SoM-9G45 module as an example.

NOTE

Note that the process described below is somewhat tailored to the 2.6.30-at91 kernel tree and will not be applicable directly to systems which use different kernels without some modification. The intent is to familiarize the reader with the task of adding support for a custom carrier board using this as a specific example. Contact EMAC if you require more information on how to do this for your target system.

Machine-Specific Support Files

Board-level support files are found under the machine-specific directories within the kernel structure for ARM devices. For the Atmel AT91-based devices, this is found under arch/arm/mach-at91. Within this directory, you will see some CPU-specific support and includes, as well as a collection of files prefixed with board-. These files contain configuration and initialization code to support a specific board -- only one of which may be included in a single kernel binary.

The board support file for the SoM9G45 is board-som9m10g45.c. You will see that it contains code for devices inherent to the module, such as the flash, LCD, Ethernet PHY, and micro-SD (MMC), as well as references to the cspec ("carrier specification") structure. The carrier specification structure is defined by code in the carrier board file. It determines which devices are enabled and how they are configured to match the features of the target carrier board.

NOTE

Note that this method of board support configuration changed dramatically in more recent kernel series with the Device Tree structure.

For example, in the som_board_init() function, SPI support is configured with the following lines:

/* SPI */
at91_add_device_spi(som_spi_devices, ARRAY_SIZE(som_spi_devices));

if (cspec.use_periph & CARRIER_USES_SPI)
	at91_add_device_spi(cspec.carrier_spi_info, cspec.carrier_spi_cnt);

The first call to at91_add_device_spi() adds devices present on the SoM itself, while the next lines check to see if the carrier board specification indicates that SPI is utilized and add all devices in the carrier board spi configuration. The carrier support functions and definitions are included through arch/arm/mach-at91/include/mach/emac-carrier.h.

Carrier Board-Specific Support

The emac-carrier directory contains code and configuration specifically for support of EMAC carrier boards. This code is used to generate the carrier specification values used in the board-som9m10g45.c file as shown above. The Kconfig file in the emac-carrier directory is sourced from the mach-at91/Kconfig file and provides selections for all available carrier boards. A snippet of the emac-carrier/Kconfig file is shown below.

if MACH_SOM9M10G45

choice
	prompt "EMAC Carrier Board Selection"
....
config SOM200_CARRIER
	bool "SOM-200ES Carrier Support"

config SOM210_CARRIER
	bool "SOM-210ES (PPC-E4) Carrier Support"

config SOM212_CARRIER
	bool "SOM-212ES Carrier Support"

config SOM250_CARRIER
	bool "SOM-250ES (PPC-E10/PPC-E7+) Carrier Support"

endchoice

This code generates the selection menu discussed in the Menuconfig section. If "SOM-210ES (PPC-E4) Carrier Support" is selected in the menu, the variable CONFIG_SOM210_CARRIER will be defined. Looking at emac-carrier/Makefile, you will see entries for each carrier board as well as other related options. For example, the line below specifies that board-som210.c will be compiled and included in the image if the SOM-210ES carrier support option is selected.

obj-$(CONFIG_SOM210_CARRIER)	+= board-som210.o

Taking SOM-210ES as an example, look at the board-som210.c. You will see a good portion of software that is very similar to the full board support files in the mach-at91 directory, since this file defines all peripheral support required by the carrier board hardware not included in the base code for the SoM itself. At the end of the file, you will find the carrier_init() function, which is called from the SoM initialization code and sets up the carrier_spec structure. This function is shown below for reference:

int __init carrier_init(struct carrier_spec *spec)
{
	spec->use_periph  = CARRIER_USES_USBH | CARRIER_USES_MCI | \
			    CARRIER_USES_TSADCC | CARRIER_USES_PWM | \
			    CARRIER_USES_LCD | CARRIER_USES_SPI | \ 
			    CARRIER_USES_I2C0;

	spec->carrier_map_io = carrier_map_io;
	spec->carrier_boardspec = carrier_device_boardspec;
	spec->carrier_spi_info = carrier_spi_devices;
	spec->carrier_spi_cnt = ARRAY_SIZE(carrier_spi_devices);

	return 0;

}

Note that some of the carrier_spec parameters are function pointers, such as spec->carrier_map_io and spec->carrier_boardspec. The signature of these two function pointers is:

void (*func)(void)

Example New Carrier Board Support Files

Suppose that support for a new carrier board is required and utilizes hardware very similar to the SoM-210ES. For the purposes of this example, assume that the name of the product is FTS. To implement support for this hardware, start by copying the emac-carrier/board-som210.c file to emac-carrier/board-fts.c and follow the steps below to add support to the kernel for this new board.

developer@ldc:~/linux/arch/arm/mach-at91/emac-carrier# cp board-som210.c board-fts.c

Kconfig and Makefile Additions

An entry must be added to the Kconfig file in order for it to be an available option in the configuration menu. To do this, edit emac-carrier/Kconfig and add the entry as shown below under the EMAC Carrier Board Selection selection menu.

if MACH_SOM9M10G45

choice
	prompt "EMAC Carrier Board Selection"

config FTS_CARRIER
	bool "FTS Carrier Support"

....
endchoice

A corresponding Makefile entry is required to trigger compilation and linking of the board-fts.c file that was created when the CONFIG_FTS_CARRIER option is selected. Add the following line to emac-carrier/Makefile:

obj-$(CONFIG_FTS_CARRIER)         += board-fts.o

Source Code Modifications

The final step in adding support for the FTS carrier board involves modifying the C source in the emac-carrier/board-fts.c file that was created to match the hardware. As this is a fictional product, the modifications below will make assumptions as to what features are included.

Serial Support

The carrier_map_io() function performs initialization and mapping of the serial ports on the system. The SOM210 carrier has three serial ports using the DBGU, USART1, and USART2 controllers. Assume that the FTS system has one additional serial port mapped to USART3 on the CPU with no handshaking. The resulting code is shown below.

static void __init carrier_map_io(void)
{
	/* DGBU on ttyS0. (Rx & Tx only) */
	at91_register_uart(0, 0, 0);

	at91_register_uart(AT91SAM9G45_ID_US1, 1, ATMEL_UART_CTS | ATMEL_UART_RTS);
	at91_register_uart(AT91SAM9G45_ID_US2, 2, ATMEL_UART_CTS | ATMEL_UART_RTS);
	at91_register_uart(AT91SAM9G45_ID_US3, 3, 0);

	/* set serial console to ttyS0 (ie, DBGU) */
	at91_set_serial_console(0);
}

SPI Support

The next section in the carrier board file specifies the SPI devices on the system. Assume that the FTS carrier board will have support for the CS4271 audio codec and one spare general purpose SPI software interface. The AD7766 SPI device in the SoM210ES carrier board file will not be required. The resulting SPI specification is as follows:

static struct spi_s spi0cs1 = 
{
	.name = "spi0cs1",
	.subclass = 0,
	.tip = lsi2esc_spi_tip,
	.xmit = lsi2esc_spi_xmit,
	.confwrite = lsi2esc_spi_confwrite,
	.confread = lsi2esc_spi_confread,
	.speedread = lsi2esc_spi_speedread,
	.speedwrite = lsi2esc_spi_speedwrite,
	.gpio_name = NULL,
	.gpio_create = NULL,
	.gpio_data = NULL,
};

static struct spi_board_info carrier_spi_devices[] = {
#if defined(CONFIG_SND_SOC_CS4271) || defined(CONFIG_SND_SOC_CS4271_MODULE)
	{	/* CS4271 codec */
		.modalias	= "cs4271",
		.chip_select	= 2,
		.bus_num	= 1,
		.max_speed_hz	= 1 * 1000 * 1000,
	},
#endif
	{ /* SPI0 CS1 (Spare) */
		.modalias = "lsi2esc",
		.chip_select = 1,
		.controller_data = (void *)AT91_PIN_PD28,
		.bus_num = 0,
		.max_speed_hz = 1e6,
		.platform_data = &spi0cs1,
	},
};

NOTE

Note that the spi0cs1 device is an EMAC SPI Class device. See EMAC SPI Programming for more information on how to utilize this interface.

Programmable Clocks

The next function in the carrier board file is set_tc_clocks(), which configures the timer/counter blocks for constant frequency clock outputs on the two dedicated clock output pins on the module. If these are not required for the hardware design, they may be disabled. Assume that the FTS board does not utilize these clocks and remove the function from the file altogether.

GPIO Support

The next section of the carrier board support file sets up GPIO devices using the EMAC GPIO Class. This involves setting up a collection of functions and arrays for each GPIO device and using the GPIO_SET_FUNCTION_CREATE() macro as defined in gpio-wrapper.h to define a complete GPIO class device. The following GPIO class devices are included in the SoM-210ES carrier support code:

1. sdsw creates a GPIO device for toggling power to the SD card on the carrier board
2. gpio creates a generic device using the lines labeled Handy GPIO pins on the SoM210ES header
3. rs232_4xx controls the serial port configuration lines on the carrier board
4. beeper creates a method to toggle the input line to the beeper on the carrier board

Assume that the FTS hardware requires the sdsw and rs232_4xx features, but does not have the beeper or generic "Handy" GPIO header. Instead, a 3:8 analog multiplexer is provided on the SoM lines GPIO11, GPIO12, and GPIO13, which correspond to CPU pins PB14, PB15, and PB16 (see the SoM-9G45M manual). The resulting GPIO support code is as follows:

/* SD card power switch */
static unsigned sdsw_gpio_set[] = { AT91_PIN_PD10 };

static unsigned sdsw_direction = 0x01; /* output */
static unsigned sdsw_value = 0x00; /* low */

GPIO_SET_FUNCTION_CREATE(sdsw)

/* Analog mux control */
static unsigned amux_gpio_set[] = {
	AT91_PIN_PB14, /* SOM GPIO11 */
	AT91_PIN_PB15, /* SOM GPIO12 */
	AT91_PIN_PB16, /* SOM GPIO13 */
};

static unsigned amux_direction = 0x0F; /* Default: All outputs */
static unsigned amux_value = 0x00; /* Default: Output low */

GPIO_SET_FUNCTION_CREATE(amux)

static unsigned rs232_4xx_gpio_set[] = {
	AT91_PIN_PB24,
	AT91_PIN_PB25,
	AT91_PIN_PE0,
};

static unsigned rs232_4xx_direction = 0x07;
static unsigned rs232_4xx_value = 0x01; /* Default to RS232 */

GPIO_SET_FUNCTION_CREATE(rs232_4xx)

Boardspec

The EMAC boardspec platform driver is used to specify board specific initialization code, such as GPIO class devices. A function pointer is passed into the platform device structure for the driver and called when the driver is loaded. carrier_classes() is utilized in this case as seen below.

/*
 * Boardspec IOEX
 */
static int carrier_classes(void)
{
	/* SD Power */
	sdsw_gpio_class_create();

	/* RS-232/422/485 Control */
	rs232_4xx_gpio_class_create();

        /* Analog mux */
	amux_gpio_class_create();
	
	return 0;
}

static struct platform_device boardspec_device = {
	.name = "boardspec",
	.id = 2,
	.dev		= {
		.platform_data	= &carrier_classes,
	},
};

static inline void carrier_device_boardspec(void)
{
	at91_set_gpio_output(AT91_PIN_PE1, 1);	/* LCD Enable */
	platform_device_register(&boardspec_device);
}

Carrier Initialization

The final function in the carrier board specification file is carrier_init(). This function is called from the SoM board file and initializes the carrier_spec structure. This includes setting the use_periph mask value to denote which devices are supported as well as initializing function pointers and other data specific to this carrier definition. The code from the board-som210.c file may be used without modification to support the FTS board. This code is listed below.

int __init carrier_init(struct carrier_spec *spec)
{
	spec->use_periph  = CARRIER_USES_USBH | CARRIER_USES_MCI | \
			    CARRIER_USES_TSADCC | CARRIER_USES_PWM | \
			    CARRIER_USES_LCD | CARRIER_USES_SPI | \
			    CARRIER_USES_I2C0;

	spec->carrier_map_io = carrier_map_io;
	spec->carrier_boardspec = carrier_device_boardspec;
	spec->carrier_spi_info = carrier_spi_devices;
	spec->carrier_spi_cnt = ARRAY_SIZE(carrier_spi_devices);

	return 0;
}

Testing

After making the development changes required to support a new carrier board, you must change the kernel configuration to specify the new carrier board. Using the menu configuration utility described in the Menuconfig section, navigate to System Type -> Atmel AT91 System-on-Chip -> EMAC Carrier Board Selection. The list should now have a choice for FTS Carrier Support. Highlight this option and press enter. Finally, exit and save the configuration.

Following the steps described in Building and Deployment above, compile the kernel and load it to your target hardware. Test all functionality, particularly new devices that were added to the carrier board support file, and make adjustments as necessary.

Driver Development

Developing and examining device driver code is often required through the course of embedded systems development, particularly due to the specialized devices used. This section provides a quick overview of device driver development on the Linux Kernel.

Overview

Device drivers provide a direct interface to a hardware device through a well-defined programming interface. This makes the device accessible to userspace and other kernel code without requiring any knowledge of what is happening at the hardware level. For example, a CAN bus controller may be memory mapped to the processor, while another connects through the SPI bus. The user does not need to know the underlying hardware interface because the CAN API is used to abstract this and the same application code may be used with either device. Furthermore, the driver for the SPI-based CAN chip may be used on different systems without porting as it uses the standard SPI API to interface with the SPI controller through the SPI driver written for the processor it is being used on.

Linux Driver Structure

As an open source operating system, all of the source code for device drivers on the system (except for any proprietary binary modules as discussed in Licensing below) are available for reference and analysis. There are many types of drivers in the Linux kernel, and many drivers can be classified in several different ways. Three major classes of device drivers apply in some way to most drivers in the Linux kernel as described below (see Linux Device Drivers Edition 3, Chapter 1 for more detailed information):

Character device drivers

Character devices, also known as char devices, are able to be accessed similar to a file, where a stream of data can be used as an abstraction to represent the device. Serial ports are one example of character devices.

Block device drivers

Block devices require data transfer in a specific block size, such as storage devices. Although the kernel interface to a block device is specialized, the interface to the user works exactly like a character device in that it accepts reading and writing a stream of any number of bytes.

Network interface drivers

Network interfaces are capable of communication across a network and are generally hardware devices, such as the "MACB" controller on Atmel hardware or a NIC on a PC. A network interface driver handles packet transmission, and identifies each device with a unique name on the system, such as eth0.

The Linux kernel device driver system implements a layered approach. This means that specific devices are separated into subsystems, which implement a common API and share core code between all drivers. Among other benefits, using this approach speeds development time and maintenance, ensures a common interface, and encourages code reuse.

A simple example of the driver model is the SPI subsystem, which provides a core SPI API with support for SPI controller drivers and SPI protocol drivers. Controller drivers provide direct access to the hardware, transmitting and receiving data on the physical pins. Protocol drivers pass messages through the controller driver to communicate with other devices using a specific protocol. Kernel and userlevel APIs are provided for initializing, declaring, and accessing SPI devices. Creating a new SPI controller driver is a matter of providing code to support the functions required by the SPI controller core that behave as specified and registering the driver during initialization. See Documentation/spi/, drivers/spi/, and include/linux/spi/ to examine this structure.

Driver Development Process

The first steps in developing or expanding a driver for a particular device include obtaining information about the device and determining what drivers already exist for this device or similar devices. This process may start through Internet searches, requesting information from the manufacturer, and searching through the kernel source as well as the source for the most recent kernels on the kernel.org site. Review of any existing drivers should provide a good idea of where your driver should fall in the kernel, what APIs it should use, and its basic structure. Depending on the device there may be several valid options which will need to be evaluated based on the needs of the application.

Once the location and structure of the driver have been decided, the code may be implemented. Several points should be considered as part of the driver development:

1. Where appropriate, use existing drivers from the stable kernel as a model for implementing new code.
2. Add Kconfig and Makefile entries as needed to support configuration of the driver.
3. While programming, follow the Linux kernel coding style as described in Documentation/CodingStyle to keep formatting and standards consistent and ease adoption.
4. During and following code development, the driver should be fully tested on the hardware.

Share

Once you have developed a kernel driver, you may want to consider becoming involved in the Linux community and submitting the driver as a patch for inclusion into the mainline kernel. Documentation/SubmittingDrivers has more information on this process and requirements.

Licensing

It is important to note that all code built-in to the kernel automatically must use an Open Source license compatible with the terms of the GPL. For loadable kernel modules, there is controversy over whether modules are derived works of the kernel. While proprietary and other non-GPL-compatible modules are generally permitted, several restrictions apply: The "taint" flag will be set on the kernel as soon as the module is loaded, and certain symbols are only available to modules marked with a GPL-compatible license.

EMAC recommends that you seek legal advice if you are unsure of the license requirements for your software.

See the following resources for more information:

• Open Source Licensing
• Linux Kernel COPYING File
• TLDP.org LKM Copyright Article

Where to go for Additional Information

There is an abundance of information available both online and in-print on the topic of Linux kernel development. Some of these resources are discussed below.

Kernel.org

Kernel.org is the site hosting the Linux kernel. Source code, news, links, and other helpful information related to the Linux kernel can be found on this site.

Linux Device Drivers, Third Edition (LDD3)

This book is available online in free PDF form through the LWN.net site. It is also available in-print through O'Reilly. LDD3 covers most of the topics required for Linux kernel development and includes examples. Note that some of the APIs covered have changed since 2.6.10 when the book was written so adaptation may be required to use the example code.

Kernel Newbies

The Kernel Newbies website provides information for people new to working with the Linux kernel and device drivers. The site is still a work in progress, but has some valuable information for working with the Linux kernel. It also maintains a Kernel Changelog, which is a page that gives a plain English description of the features and bugfixes which went into the latest kernel release. The changelog also includes links to pages on other sites which provide more details on specific features and/or fixes which went into the kernel version being described. The changelogs which were made for previous kernel versions can be found here.

Mailing Lists

There are many very active mailing lists (see http://vger.kernel.org/vger-lists.html) that can be used to ask questions or search through archives for solutions. The archive of the Linux Kernel mailing list is available at LKML.org and archives to other lists can be found through a web search as well as links noted in the list here.

Linux Kernel in a Nutshell

This book is written by Greg Kroah-Hartman, one of the primary developers and maintainers of the kernel and covers the process and tools for building, configuration, and installation of the Linux kernel. It is available for free through the authors site in electronic format or in-print through O'Reilly.

EMAC Support

EMAC provides custom Linux kernel development and kernel development support for EMAC customers on a contractual basis. Contact EMAC Support for more information.