Difference between revisions of "System Logging"

From wiki.emacinc.com
Jump to: navigation, search
m (Checking page in while switching tasks.)
m (Saving to keep from losing it, while going for lunch.)
Line 96: Line 96:
 
{{:Templateimpl:geninfo | initials=MD | title=System Logging | desc=The page describes how to system log. | project=OE 5.0 }}
 
{{:Templateimpl:geninfo | initials=MD | title=System Logging | desc=The page describes how to system log. | project=OE 5.0 }}
  
=====Your Daemons=====
+
====Your Daemons====
  
 
When developing a software daemon, syslog is a very useful tool for monitoring and debugging your software.  The syslog daemon is highly configurable, allowing filters to be altered to suit the current needs of a developer without recompiling software.  This capability allows developers to deploy software which sends ample debugging messages to the syslog server without fear of overwhelming the system with debug messages.  Using the eight <code>local</code> facilities syslog recognizes, these debug messages can be categorized by developers to allow them to selectively choose the amount of log information they store from each of up to eight different subsystems.  This is extremely useful when a system in the field is experiencing intermittent issues.  By enabling maximum debugging output for only the subsystem responsible for the feature which is misbehaving, the developers have the greatest chance of seeing the messages they need to diagnose the source of the problem without overwhelming the system (or the storage) with messages.  If eight facilities is not enough, the developers can always repurpose dedicated facilities which aren't in use by their intended systems.  For instance, the line printing facility is rarely used on an embedded system, and may be safely repurposed by developers.
 
When developing a software daemon, syslog is a very useful tool for monitoring and debugging your software.  The syslog daemon is highly configurable, allowing filters to be altered to suit the current needs of a developer without recompiling software.  This capability allows developers to deploy software which sends ample debugging messages to the syslog server without fear of overwhelming the system with debug messages.  Using the eight <code>local</code> facilities syslog recognizes, these debug messages can be categorized by developers to allow them to selectively choose the amount of log information they store from each of up to eight different subsystems.  This is extremely useful when a system in the field is experiencing intermittent issues.  By enabling maximum debugging output for only the subsystem responsible for the feature which is misbehaving, the developers have the greatest chance of seeing the messages they need to diagnose the source of the problem without overwhelming the system (or the storage) with messages.  If eight facilities is not enough, the developers can always repurpose dedicated facilities which aren't in use by their intended systems.  For instance, the line printing facility is rarely used on an embedded system, and may be safely repurposed by developers.
  
=====Web Development=====
+
====Web Development====
  
 
When providing a web interface for an embedded system, the syslog daemon is often the only way to debug a variety of issues with the web server.  When links don't work, certain browsers have issues, pages load slowly, or any of a myriad of other issues crop up, the logs are generally the first place people turn for answers.  This is just as true on embedded systems as it is on any large scale web server or server farm.  On embedded systems, however, the configuration for the logging system may need to be altered due to the volatile nature of the logs in the default configuration.  This document provides information on how to solve the issues which may be encountered as a result of this.
 
When providing a web interface for an embedded system, the syslog daemon is often the only way to debug a variety of issues with the web server.  When links don't work, certain browsers have issues, pages load slowly, or any of a myriad of other issues crop up, the logs are generally the first place people turn for answers.  This is just as true on embedded systems as it is on any large scale web server or server farm.  On embedded systems, however, the configuration for the logging system may need to be altered due to the volatile nature of the logs in the default configuration.  This document provides information on how to solve the issues which may be encountered as a result of this.
  
=====Machine Startup=====
+
====Machine Startup====
  
 
When configuring a machine to start up all the necessary services in the correct sequential order, when the machine is powered on, the syslog server is very helpful in performing the following tasks:
 
When configuring a machine to start up all the necessary services in the correct sequential order, when the machine is powered on, the syslog server is very helpful in performing the following tasks:
Line 115: Line 115:
 
Startup scripts log their activity to the syslog daemon.  Any daemon process launched by these startup scripts should also log startup messages to indicate that they're starting, how they're configured, and that they've finished the startup sequence and are now running.  The system logs enable the engineer to assure that a machine has started as intended.
 
Startup scripts log their activity to the syslog daemon.  Any daemon process launched by these startup scripts should also log startup messages to indicate that they're starting, how they're configured, and that they've finished the startup sequence and are now running.  The system logs enable the engineer to assure that a machine has started as intended.
  
=====Malfunctioning Systems=====
+
====Malfunctioning Systems====
  
 
When a proven system is malfunctioning, the system logs can be essential in determining the cause of the malfunction.  In cases such as this, the system logs are helpful for:
 
When a proven system is malfunctioning, the system logs can be essential in determining the cause of the malfunction.  In cases such as this, the system logs are helpful for:
Line 124: Line 124:
 
Without these logs, it may be nearly impossible to diagnose these malfunctions.
 
Without these logs, it may be nearly impossible to diagnose these malfunctions.
  
=====In Brief=====
+
====In Brief====
  
 
This guide provides enough background information to enable you to handle any of the situations described above.  Using the information contain herein, you will be able to approach any of the above listed problems with the tools you need to solve the problems you are facing.
 
This guide provides enough background information to enable you to handle any of the situations described above.  Using the information contain herein, you will be able to approach any of the above listed problems with the tools you need to solve the problems you are facing.
Line 136: Line 136:
 
EMAC OE Linux 5.0 uses Busybox's syslog daemon, <code>syslogd</code>, by default.  However, the <code>syslog-ng</code> package can be installed to provide a much more capable system logging daemon, when needed.
 
EMAC OE Linux 5.0 uses Busybox's syslog daemon, <code>syslogd</code>, by default.  However, the <code>syslog-ng</code> package can be installed to provide a much more capable system logging daemon, when needed.
  
=====Default Syslog Daemon=====
+
===Default Syslog Daemon===
  
The Busybox syslog daemon is configured to log to files in the RAMDISK mounted on <code>/var/log/</code>. The files contained in this directory will be capped in size and rotated according to the configuration of the syslog daemon.  The standard syslog daemon supports the following configuration capabilities:
+
The Busybox syslog daemon is used by default.
 +
 
 +
====About====
 +
 
 +
The standard syslog daemon supports the following configuration capabilities:
 
* Maximum size limit for log files.
 
* Maximum size limit for log files.
 
* Maximum number of log files to keep, when rotated.
 
* Maximum number of log files to keep, when rotated.
Line 145: Line 149:
 
* Log to remote and local files simultaneously, just to local files, or to a memory buffer which needs to be read by a special utility.
 
* Log to remote and local files simultaneously, just to local files, or to a memory buffer which needs to be read by a special utility.
  
=====Optional <code>syslog-ng</code> With <code>logrotate</code>=====
+
====Working With====
 +
 
 +
The Busybox syslog daemon is configured, by default, to log to files in the RAMDISK mounted on <code>/var/log/</code>.  The files contained in this directory will be capped in size and rotated according to the configuration of the syslog daemon.
 +
 
 +
The Busybox syslog daemon is configured with the file, <code>/etc/syslog-startup.conf</code>.  This file is used for backwards compatibility, because the Busybox syslog daemon recently began reading the <code>/etc/syslog.conf</code> file.  The <code>/etc/syslog-startup.conf</code> comes from OE rather than from Busybox.
 +
 
 +
The standard <code>/etc/syslog-startup.conf</code> configuration is shown below:
 +
 
 +
<syntaxhighlight lang=bash>
 +
# This configuration file is used by the busybox syslog init script,
 +
# /etc/init.d/syslog[.busybox] to set syslog configuration at start time.
 +
 
 +
DESTINATION=file                # log destinations (buffer file remote)
 +
LOGFILE=/var/log/messages      # where to log (file)
 +
REMOTE=loghost:514              # where to log (syslog remote)
 +
REDUCE=no                      # reduce-size logging
 +
DROPDUPLICATES=no              # whether to drop duplicate log entries
 +
#ROTATESIZE=0                  # rotate log if grown beyond X [kByte]
 +
#ROTATEGENS=3                  # keep X generations of rotated logs
 +
BUFFERSIZE=64                  # size of circular buffer [kByte]
 +
FOREGROUND=no                  # run in foreground (don't use!)
 +
#LOGLEVEL=5                    # local log level (between 1 and 8)
 +
</syntaxhighlight>
 +
 
 +
;DESTINATION
 +
: This configures where the syslog messages will be sent. 
 +
:;buffer
 +
:: The <code>buffer</code> option will cause the messages to be stored in an internal circular buffer, in RAM.  This buffer must be read with <code>logread</code>.
 +
:;file
 +
:: The <code>file</code> option will cause the messages to be stored in a file in the location specified by the <code>LOGFILE</code> parameter, listed below.
 +
:;remote
 +
:: The <code>remote</code> option will cause syslog messages to be sent to a remote syslog daemon.
 +
;LOGFILE
 +
: The <code>LOGFILE</code> parameter specifies the location of the logfile which should be used with the <code>file</code> destination.
 +
;REMOTE
 +
: The <code>REMOTE</code> parameter specifies the host and port to which to send remote log messages.  The two part format for this variable provides the destination host followed by the destination port number, with a colon as the delimiter between the two.  The destination may be specified as a hostname, or as an IP address.  The destination port number should be the port on which the remote syslog server listens.
 +
 
 +
{{note | If the destination is specified as a hostname, it is best to create an entry in <code>/etc/hosts</code> for the hostname so that it will resolve to an IP address even if there is a problem with the DNS servers.}}
 +
 
 +
;REDUCE
 +
: The <code>REDUCE</code> option takes a <code>yes</code> or <code>no</code> argument.  When set to <code>yes</code>, the syslog server will neglect to log the hostname, facility, and severity level with each log message.  This is only recommended for severely resource constrained systems, since the facility and severity level are often used for filtering messages through <code>grep</code>.
 +
;DROPDUPLICATES
 +
: The <code>DROPDUPLICATES</code> option takes a <code>yes</code> or <code>no</code> argument.  When set to <code>yes</code>, the syslog server will ignore messages which duplicate the preceeding message.  This can be useful when an errant process keeps sending the same message over and over again, and swamps out the other messages as a result (due to the maximum size setting for the log files).  It is best to keep this option off unless it is absolutely needed for a situation like the one just described, because having it turned on can cause other problems to be hidden (generally leading to frustration and/or embarrassment).
 +
;ROTATESIZE
 +
: The <code>ROTATESIZE</code> option takes a numeric argument.  This argument is the maximum desired size for a log file, specified in kibibytes.  When a log file has grown to the point where it exceeds this high water mark, it will be rotated.
 +
;ROTATEGENS
 +
: The <code>ROTATEGENS</code> option takes a numeric argument.  This argument specifies the number of older generations of log files to keep in existence.  By generations, it means the number of log files which have already been renamed with a sequencing number as a postfix.  See the note below for more information.
 +
{{note | Logfile rotation is a process which renames log files sequentially, to preserve older log messages while keeping incoming messages writing to the same location.  For instance, for the <code>/var/log/messages</code> file, this might look like:<code>
 +
messages -> messages.0
 +
+messages</code>
 +
Or, if one were to perform this rotation manually at a terminal, it may look like:
 +
{{clo}}
 +
{{clio | pwd=/var/log |mv messages messages.0}}
 +
{{clio | pwd=/var/log |touch messages}}
 +
{{clos}}
 +
If there already is a <code>messages.0</code> file, and the number of generations to keep is set to 2, then a manual version of the rotation might look something like this:
 +
{{clo}}
 +
{{clio | pwd=/var/log |rm -f messages.2}}
 +
{{clio | pwd=/var/log |mv messages.1 messages.2}}
 +
{{clio | pwd=/var/log |mv messages.0 messages.1}}
 +
{{clio | pwd=/var/log |mv messages messages.0}}
 +
{{clio | pwd=/var/log |touch messages}}
 +
{{clos}}
 +
}}
 +
 
 +
;BUFFERSIZE
 +
: The <code>BUFFERSIZE</code> option takes a numeric argument.  This argument specifies the size of the internal circular buffer to use for holding log messages in an internal RAM buffer.  This argument only matters when the <code>DESTINATION</code> is set to <code>buffer</code>.  The size is specified in kibibytes.
 +
;FOREGROUND
 +
: The <code>FOREGROUND</code> option takes a <code>yes</code> or <code>no</code> argument.  This argument specifies whether the <code>syslogd</code> process should run in the foreground or as a daemon.  It is strongly recommended for this option to always be set to, <code>no</code>, to run <code>syslogd</code> as a daemon.  This option mostly exists for the developers of Busybox to debug their <code>syslogd</code> implementation.
 +
;LOGLEVEL
 +
: The <code>LOGLEVEL</code> option takes a numeric argument.  This argument specifies the highest numbered severity level which should be logged (see [[System_Logging#Severity | severity]] above).  By default, this is set such that messages with a severity of '''notice''', or more severe, will be logged.
 +
 
 +
===Optional <code>syslog-ng</code> With <code>logrotate</code>===
  
 
The optional <code>syslog-ng</code> and <code>logrotate</code> option provides everything that's provided by the Busybox syslog daemon, plus:
 
The optional <code>syslog-ng</code> and <code>logrotate</code> option provides everything that's provided by the Busybox syslog daemon, plus:
Line 165: Line 241:
 
{{:Templateimpl:examples | initials=MD | title=System Logging | desc=The page describes how to system log. | project=OE 5.0 }}
 
{{:Templateimpl:examples | initials=MD | title=System Logging | desc=The page describes how to system log. | project=OE 5.0 }}
  
====Evaluating Log Messages====
+
===Evaluating Log Messages===
  
 
Log messages are stored in <code>/var/log/</code> by default.  This directory contains at least one log file, <code>messages</code>.
 
Log messages are stored in <code>/var/log/</code> by default.  This directory contains at least one log file, <code>messages</code>.
  
====Writing to Syslog from Programs and Scripts====
+
===Writing to Syslog from Programs and Scripts===
As mentioned previously, many languages support writing to the Linux system logging utility.
+
As mentioned previously, many languages support writing to the Linux system logging utility. There are many handy libraries and functions for this.  Some of the most commonly used ones are shown here.
 +
 
 +
====Shell Utility: <code>logger</code>====
 +
Whether you are writing a shell script, or simply wish to enter something into the system logs from the commandline, the <code>logger</code> utility will give you the capability you need.  This simple command takes a few options, but it can also be run with the only argument being the message to log.
 +
 
 +
=====Arguments to <code>logger</code>=====
 +
 
 +
The following options are available for <code>logger</code>:
 +
;<code>-t</code> ''TAG''
 +
: Takes a ''TAG'' as an argument.  This option specifies a tag to label the message with, which is useful for sorting through logs later.
 +
;<code>-p</code> ''Priority''
 +
: Takes a ''Priority'' as an argument.  The priority may be specified as just a numeric priority level, identifying the severity of the message, or it can be specified as a tuple containing a facility name and a priority level, delimited by a dot (<code>.</code>).  If no facility is specified, the facility defaults to, <code>user</code>.  The severity levels are specified in [[System_Logging#Severity | severity]] above.  The facility names are specified in [[System_Logging#Facilities | facilities]] above.
 +
;<code>-s</code>
 +
: This option sets a boolean which enables logging messages to <code>stdout</code> as well as to the system logger.  This is especially useful in interactive shell scripts, since it allows the developer to use one function to output a message to the screen and to the system logs simultaneously.
 +
 
 +
=====Examples of Using <code>logger</code>=====
 +
 
 +
To send a message to the system log as simply as possible:
 +
{{clo}}
 +
{{clio |logger "Default settings message example"}}
 +
{{clio |tail -n 1 /var/log/messages}}
 +
Nov 17 19:20:23 ipac9x25 user.notice root: Default settings message example{{clos}}
 +
 
 +
To send a message to the system log with severity 3 (err):
 +
{{clo}}
 +
{{clio |logger -p 3 "This is an example error message"}}
 +
{{clio |tail -n 1 /var/log/messages}}
 +
Nov 17 19:22:06 ipac9x25 user.err root: This is an example error message
 +
{{clos}}
 +
 
 +
To send a message with the facility, <code>local5</code>, and the severity 4 (warn):
 +
{{clo}}
 +
{{clio |logger -p local5.4 "This is a local5 facility warning message example."}}
 +
{{clio |tail -n 1 /var/log/messages}}
 +
Nov 17 19:23:51 ipac9x25 local5.warn root: This is a local5 facility warning message example.{{clos}}
 +
To add a tag (highly recommended) to the above, use <code>-t</code>.  Here, we use the tag, <code>EXAMPLE</code>:
 +
{{clo}}
 +
{{clio |logger -t EXAMPLE -p local5.4 "This is a local5 facility warning message example."}}
 +
{{clio |tail -n 1 /var/log/messages}}
 +
Nov 17 19:25:19 ipac9x25 local5.warn EXAMPLE: This is a local5 facility warning message example.
 +
{{clos}}
 +
Notice that the username (<code>root</code>) was replaced by the tag (<code>EXAMPLE</code>).  When using tags in a script or program, it is best to come up with a unique tag and create a function for sending messages to the system log with this tag.  That way, you can easily find all messages from just your program/script.  If you have a complex program or set of scripts to debug, you could use multiple tags, but this will be more difficult to search for unless you use the optional <code>syslog-ng</code> to redirect the messages from your software to distinct files.
 +
In an interactive script, you might use <code>logger</code> like this:
 +
{{clo}}
 +
{{clio|logger -t EXAMPLE -p local3.6 -s "Successfully wrote new configuration file."}}
 +
EXAMPLE: Successfully wrote new configuration file.
 +
{{clio |tail -n 1 /var/log/messages}}
 +
Nov 17 19:25:19 ipac9x25 local5.warn EXAMPLE: This is a local5 facility warning message example.
 +
{{clos}}
 +
Notice that the message isn't visible in the system log?  This is because the <code>syslogd</code> daemon on this machine is only configured to log messages which have a severity of 5 or lower.  We might modify the example so that it will be logged, like this:
 +
{{clo}}
 +
{{clio|logger -t EXAMPLE -p local3.5 -s "Successfully wrote new configuration file."}}
 +
EXAMPLE: Successfully wrote new configuration file.
 +
{{clio |tail -n 1 /var/log/messages}}
 +
Nov 17 19:31:37 ipac9x25 local3.notice EXAMPLE: Successfully wrote new configuration file.
 +
{{clos}}
 +
 
 +
Now the message has been saved to the system logs as well as displayed on <code>stdout</code>.
 +
 
 +
====Writing to Syslog from a C Program====
 +
There is a standard C function call on Linux for writing to the system logger.
 +
 
  
 
<!-- /*********************************************************************************************************/ -->
 
<!-- /*********************************************************************************************************/ -->

Revision as of 14:40, 17 November 2015

TODO: {{#todo: InProgress (04.13.2015-14:44->BS+)(11.10.2015-18:00->MD+)(11.11.2015-10:30->MD+)|Mike Dean|OE 5.0,BS,MD}}

This document provides an overview of how system logging works in Linux, and provides guidance on how to work with and configure system logging so that it meets your needs.

Background

System Logging services exist on Linux systems to provide a central logging facility. This central logging service can store logged messages in files on the local machine, can forward the log messages to a remote machine for remote storage and display, and can output the log messages as they arrive to a terminal.

The syslog facility is standardized by POSIX and, as a result, there is a great deal of support for the syslog facility. There are APIs for logging to the syslog facility from C, C++, Python, Java, PHP, Perl, Bash, the commandline, and many other languages. Using the commandline tool, it is even possible to write to the syslog facility from the commandline.

Facilities

In syslog terminology, a facility is a category for log messages. There are generally at least seven standard syslog facilities on any Linux syslog system. The following list shows the standard seven as well as some which are rarely used anymore (especially in embedded systems). The name of the facility is followed, in parenthesis, by the integer facility code associated with the facility, which is then followed by a description of the facility. The ones you should not expect to find on an embedded system are demarcated with an asterisk (*):

kern(0)
Kernel messages
user(1)
User Level Messages
mail(2)*
Mail System Messages
daemon(3)
System Daemon Messages
auth(4)
Security/Authentication Messages
syslog(5)
Internally Generated Syslog Messages
lpr(6)*
Line Printer Subsystem Messages
news(7)*
Network News Subsystem Messages
uucp(8)*
UUCP Subsystem Messages
cron(15)
Cron Scheduling Daemon Messages
local(16-23)
Locally Defined Usage, Messages

Severity

Each of these facilities accepts messages which will have an associated severity level. The severity level is important for filtering messages. The following list shows the eight available severity levels:

emerg(0)
Emergency - System is now unusable
alert(1)
Alert - Immediate Attention is Required
crit(2)
Critical - The System is in a Critical Condition, which may be caused by a failure in the primary appliction.
err(3)
Error - An Error has occurred.
warning(4)
Warning - An unusual, but not erroneous, event has occurred. For example: "Warning: Could not check for updates."
notice(5)
Notice - Uncommon but generally expected messages. For example: "Network Interface eth0 received IP address of 192.168.0.100 from DHCP server."
info(6)
Informational - Common, expected events generate these messages. For example: "Application XYZ started successfully."
debug(7)
Debug - Debugging messages belong at this level.

Storage

By default, the log messages on an EMAC OE Linux machine are stored in a local ramdisk. This type of storage prevents excessive wear of the flash, and works when all of the flash partitions are mounted read-only. However, by storing the messages in a ramdisk, all messages in the ramdisk will be lost as soon as the machine resets, powers off, or reboots for any reason.

Alternate Logging Locations

The syslog server can be configured to log to alternate locations. This is useful for configuring the server to log to persistent storage, such as a writable partition on the local flash or a partition on an SD card. The alternate logging locations can even be remote machines.

Remote Logging

The /var/log directory is the standard location for holding local log messages. This directory usually contains either one file per facility or one directory per facility. If a facility has a directory, it may contain different files which all belong to the facility. The system logging facility directs messages to these files based on rules set in its configuration. The configured rules use the facility level and the severity level to direct the syslogger's filters regarding to which file to send any particular message. There are several different implementations of the syslogger facility, each with its own configuration mechanisms and level of flexibility.

The system logger is also able to send messages to a remote syslog server over a network. The remote syslog server will be able to filter the messages using all of the same criteria as above, but also by source. A remote syslog server configuration can be very useful for remotely debugging deployed machines. While an embedded machine stores its logs on a ramdisk, the remote machine may store them to a traditional hard disk or SSD, allowing them to be preserved through power cycles.

The remote syslog server can be on any normal server, it can be on a developer's desktop, or it can be on another embedded machine. Sending debug messages to a central syslog server ensures the debug messages will be available even if the machine of interest freezes or resets. The use of a remote syslog server can also assist with looking for trends and outliers among a group of deployed machines since the log messages for the entire group of machines can be located in one place.

Log Rotation

Syslog servers incorporate a feature called log rotation. Log rotation is a process which prevents log files from growing beyond a certain size while preserving log messages for later reference. The syslog server (or logrotate system) can be configured with a maximum file size and a maximum number of rotated logs to keep. When the maximum file size is reached, the file will be "rotated." When the maximum number of logs to keep is exceeded, the oldest one will be deleted. There is more information on how this process works below.


General Information

Your Daemons

When developing a software daemon, syslog is a very useful tool for monitoring and debugging your software. The syslog daemon is highly configurable, allowing filters to be altered to suit the current needs of a developer without recompiling software. This capability allows developers to deploy software which sends ample debugging messages to the syslog server without fear of overwhelming the system with debug messages. Using the eight local facilities syslog recognizes, these debug messages can be categorized by developers to allow them to selectively choose the amount of log information they store from each of up to eight different subsystems. This is extremely useful when a system in the field is experiencing intermittent issues. By enabling maximum debugging output for only the subsystem responsible for the feature which is misbehaving, the developers have the greatest chance of seeing the messages they need to diagnose the source of the problem without overwhelming the system (or the storage) with messages. If eight facilities is not enough, the developers can always repurpose dedicated facilities which aren't in use by their intended systems. For instance, the line printing facility is rarely used on an embedded system, and may be safely repurposed by developers.

Web Development

When providing a web interface for an embedded system, the syslog daemon is often the only way to debug a variety of issues with the web server. When links don't work, certain browsers have issues, pages load slowly, or any of a myriad of other issues crop up, the logs are generally the first place people turn for answers. This is just as true on embedded systems as it is on any large scale web server or server farm. On embedded systems, however, the configuration for the logging system may need to be altered due to the volatile nature of the logs in the default configuration. This document provides information on how to solve the issues which may be encountered as a result of this.

Machine Startup

When configuring a machine to start up all the necessary services in the correct sequential order, when the machine is powered on, the syslog server is very helpful in performing the following tasks:

  • Ensuring that process X really did start before process Y.
  • Checking to see if any errors or warnings were reported by processes starting up.
  • Ensuring that processes really did start, at all.
  • Checking to see what configuration was chosen by a process which is capable of choosing its configuration dynamically.
  • Checking the messages reported by device drivers as they start up.
  • Determining the IP address used by a system which gets its network configuration via DHCP.

Startup scripts log their activity to the syslog daemon. Any daemon process launched by these startup scripts should also log startup messages to indicate that they're starting, how they're configured, and that they've finished the startup sequence and are now running. The system logs enable the engineer to assure that a machine has started as intended.

Malfunctioning Systems

When a proven system is malfunctioning, the system logs can be essential in determining the cause of the malfunction. In cases such as this, the system logs are helpful for:

  • Looking to see who logged into a machine, and when.
  • Looking for signs of attacks directed at the machine.
  • Looking for signs of failing hardware.
  • Looking for error and warning messages coming from system processes, daemons, and/or your custom application.

Without these logs, it may be nearly impossible to diagnose these malfunctions.

In Brief

This guide provides enough background information to enable you to handle any of the situations described above. Using the information contain herein, you will be able to approach any of the above listed problems with the tools you need to solve the problems you are facing.



NOTE
To make best use of the log files, a working knowledge of grep is highly beneficial. Experience with crafting and using regular expressions with grep prior to writing system logging messages in your application and init scripts can make the diagnosis process simpler, since a little forethought into a syntax for the messages can make filtering the messages through grep much easier to accomplish correctly.


System Logging

EMAC OE Linux 5.0 uses Busybox's syslog daemon, syslogd, by default. However, the syslog-ng package can be installed to provide a much more capable system logging daemon, when needed.

Default Syslog Daemon

The Busybox syslog daemon is used by default.

About

The standard syslog daemon supports the following configuration capabilities:

  • Maximum size limit for log files.
  • Maximum number of log files to keep, when rotated.
  • Minimum priority level for messages to log.
  • Duplicate dropping (not usually recommended).
  • Log to remote and local files simultaneously, just to local files, or to a memory buffer which needs to be read by a special utility.

Working With

The Busybox syslog daemon is configured, by default, to log to files in the RAMDISK mounted on /var/log/. The files contained in this directory will be capped in size and rotated according to the configuration of the syslog daemon.

The Busybox syslog daemon is configured with the file, /etc/syslog-startup.conf. This file is used for backwards compatibility, because the Busybox syslog daemon recently began reading the /etc/syslog.conf file. The /etc/syslog-startup.conf comes from OE rather than from Busybox.

The standard /etc/syslog-startup.conf configuration is shown below:

# This configuration file is used by the busybox syslog init script,
# /etc/init.d/syslog[.busybox] to set syslog configuration at start time.

DESTINATION=file                # log destinations (buffer file remote)
LOGFILE=/var/log/messages       # where to log (file)
REMOTE=loghost:514              # where to log (syslog remote)
REDUCE=no                       # reduce-size logging
DROPDUPLICATES=no               # whether to drop duplicate log entries
#ROTATESIZE=0                   # rotate log if grown beyond X [kByte]
#ROTATEGENS=3                   # keep X generations of rotated logs
BUFFERSIZE=64                   # size of circular buffer [kByte]
FOREGROUND=no                   # run in foreground (don't use!)
#LOGLEVEL=5                     # local log level (between 1 and 8)
DESTINATION
This configures where the syslog messages will be sent.
buffer
The buffer option will cause the messages to be stored in an internal circular buffer, in RAM. This buffer must be read with logread.
file
The file option will cause the messages to be stored in a file in the location specified by the LOGFILE parameter, listed below.
remote
The remote option will cause syslog messages to be sent to a remote syslog daemon.
LOGFILE
The LOGFILE parameter specifies the location of the logfile which should be used with the file destination.
REMOTE
The REMOTE parameter specifies the host and port to which to send remote log messages. The two part format for this variable provides the destination host followed by the destination port number, with a colon as the delimiter between the two. The destination may be specified as a hostname, or as an IP address. The destination port number should be the port on which the remote syslog server listens.



NOTE
If the destination is specified as a hostname, it is best to create an entry in /etc/hosts for the hostname so that it will resolve to an IP address even if there is a problem with the DNS servers.


REDUCE
The REDUCE option takes a yes or no argument. When set to yes, the syslog server will neglect to log the hostname, facility, and severity level with each log message. This is only recommended for severely resource constrained systems, since the facility and severity level are often used for filtering messages through grep.
DROPDUPLICATES
The DROPDUPLICATES option takes a yes or no argument. When set to yes, the syslog server will ignore messages which duplicate the preceeding message. This can be useful when an errant process keeps sending the same message over and over again, and swamps out the other messages as a result (due to the maximum size setting for the log files). It is best to keep this option off unless it is absolutely needed for a situation like the one just described, because having it turned on can cause other problems to be hidden (generally leading to frustration and/or embarrassment).
ROTATESIZE
The ROTATESIZE option takes a numeric argument. This argument is the maximum desired size for a log file, specified in kibibytes. When a log file has grown to the point where it exceeds this high water mark, it will be rotated.
ROTATEGENS
The ROTATEGENS option takes a numeric argument. This argument specifies the number of older generations of log files to keep in existence. By generations, it means the number of log files which have already been renamed with a sequencing number as a postfix. See the note below for more information.



NOTE
Logfile rotation is a process which renames log files sequentially, to preserve older log messages while keeping incoming messages writing to the same location. For instance, for the /var/log/messages file, this might look like:

messages -> messages.0 +messages Or, if one were to perform this rotation manually at a terminal, it may look like:

root@som9x25:/var/log# mv messages messages.0
root@som9x25:/var/log# touch messages

If there already is a messages.0 file, and the number of generations to keep is set to 2, then a manual version of the rotation might look something like this:

root@som9x25:/var/log# rm -f messages.2
root@som9x25:/var/log# mv messages.1 messages.2
root@som9x25:/var/log# mv messages.0 messages.1
root@som9x25:/var/log# mv messages messages.0
root@som9x25:/var/log# touch messages


BUFFERSIZE
The BUFFERSIZE option takes a numeric argument. This argument specifies the size of the internal circular buffer to use for holding log messages in an internal RAM buffer. This argument only matters when the DESTINATION is set to buffer. The size is specified in kibibytes.
FOREGROUND
The FOREGROUND option takes a yes or no argument. This argument specifies whether the syslogd process should run in the foreground or as a daemon. It is strongly recommended for this option to always be set to, no, to run syslogd as a daemon. This option mostly exists for the developers of Busybox to debug their syslogd implementation.
LOGLEVEL
The LOGLEVEL option takes a numeric argument. This argument specifies the highest numbered severity level which should be logged (see severity above). By default, this is set such that messages with a severity of notice, or more severe, will be logged.

Optional syslog-ng With logrotate

The optional syslog-ng and logrotate option provides everything that's provided by the Busybox syslog daemon, plus:

  • Sophisticated filters for directing log traffic.
  • The ability to act as a central syslog server which receives log messages from remote syslog machines.
  • Enhanced control over the destination(s) for log messages, allowing (for example) messages to be logged to a local RAMDISK, a local SD card (if available), and to a remote syslog server (if available) simultaneously.
  • The ability to configure logrotate to automatically e-mail log files when they're rotated.
  • Same syslog server software is readily available for Linux desktops and servers.

As can be expected, the potential drawback of syslog-ng is the steeper learning curve required to make use of its more sophisticated features. However, those already familiar with syslog-ng will be able to benefit from its extra sophistication immediately.

The official manual page for syslog-ng has more details: http://www.syslog.org/syslog-ng/v1/

Please contact EMAC for details on how to install syslog-ng and logrotate if you have trouble finding and installing them with opkg.

Examples

Evaluating Log Messages

Log messages are stored in /var/log/ by default. This directory contains at least one log file, messages.

Writing to Syslog from Programs and Scripts

As mentioned previously, many languages support writing to the Linux system logging utility. There are many handy libraries and functions for this. Some of the most commonly used ones are shown here.

Shell Utility: logger

Whether you are writing a shell script, or simply wish to enter something into the system logs from the commandline, the logger utility will give you the capability you need. This simple command takes a few options, but it can also be run with the only argument being the message to log.

Arguments to logger

The following options are available for logger:

-t TAG
Takes a TAG as an argument. This option specifies a tag to label the message with, which is useful for sorting through logs later.
-p Priority
Takes a Priority as an argument. The priority may be specified as just a numeric priority level, identifying the severity of the message, or it can be specified as a tuple containing a facility name and a priority level, delimited by a dot (.). If no facility is specified, the facility defaults to, user. The severity levels are specified in severity above. The facility names are specified in facilities above.
-s
This option sets a boolean which enables logging messages to stdout as well as to the system logger. This is especially useful in interactive shell scripts, since it allows the developer to use one function to output a message to the screen and to the system logs simultaneously.
Examples of Using logger

To send a message to the system log as simply as possible:

root@som9x25:~# logger "Default settings message example"
root@som9x25:~# tail -n 1 /var/log/messages
Nov 17 19:20:23 ipac9x25 user.notice root: Default settings message example

To send a message to the system log with severity 3 (err):

root@som9x25:~# logger -p 3 "This is an example error message"
root@som9x25:~# tail -n 1 /var/log/messages

Nov 17 19:22:06 ipac9x25 user.err root: This is an example error message

To send a message with the facility, local5, and the severity 4 (warn):

root@som9x25:~# logger -p local5.4 "This is a local5 facility warning message example."
root@som9x25:~# tail -n 1 /var/log/messages
Nov 17 19:23:51 ipac9x25 local5.warn root: This is a local5 facility warning message example.

To add a tag (highly recommended) to the above, use -t. Here, we use the tag, EXAMPLE:

root@som9x25:~# logger -t EXAMPLE -p local5.4 "This is a local5 facility warning message example."
root@som9x25:~# tail -n 1 /var/log/messages

Nov 17 19:25:19 ipac9x25 local5.warn EXAMPLE: This is a local5 facility warning message example.

Notice that the username (root) was replaced by the tag (EXAMPLE). When using tags in a script or program, it is best to come up with a unique tag and create a function for sending messages to the system log with this tag. That way, you can easily find all messages from just your program/script. If you have a complex program or set of scripts to debug, you could use multiple tags, but this will be more difficult to search for unless you use the optional syslog-ng to redirect the messages from your software to distinct files. In an interactive script, you might use logger like this:

root@som9x25:~# logger -t EXAMPLE -p local3.6 -s "Successfully wrote new configuration file."

EXAMPLE: Successfully wrote new configuration file.

root@som9x25:~# tail -n 1 /var/log/messages

Nov 17 19:25:19 ipac9x25 local5.warn EXAMPLE: This is a local5 facility warning message example.

Notice that the message isn't visible in the system log? This is because the syslogd daemon on this machine is only configured to log messages which have a severity of 5 or lower. We might modify the example so that it will be logged, like this:

root@som9x25:~# logger -t EXAMPLE -p local3.5 -s "Successfully wrote new configuration file."

EXAMPLE: Successfully wrote new configuration file.

root@som9x25:~# tail -n 1 /var/log/messages

Nov 17 19:31:37 ipac9x25 local3.notice EXAMPLE: Successfully wrote new configuration file.

Now the message has been saved to the system logs as well as displayed on stdout.

Writing to Syslog from a C Program

There is a standard C function call on Linux for writing to the system logger.


Conclusion

Further Information

Where to Go Next
Pages with Related Content