Difference between revisions of "System Logging"
(Created page with "{{ subst:Pgtempl | initials={{subst::Templateimpl:getinitials}} | project=OE 5.0 | title=System Logging | desc=The page describes how to system log. }}") |
m (Checking page in while switching tasks.) |
||
Line 1: | Line 1: | ||
− | {{todo| | + | {{todo| InProgress (04.13.2015-14:44->BS+)(11.10.2015-18:00->MD+)(11.11.2015-10:30->MD+)| MD | project=OE 5.0,BS,MD }} |
{{#seo: | {{#seo: | ||
|title=System Logging | |title=System Logging | ||
Line 10: | Line 10: | ||
<!-- /**************************************** Page Description Text ****************************************/ --> | <!-- /**************************************** Page Description Text ****************************************/ --> | ||
<!-- /*********************************************************************************************************/ --> | <!-- /*********************************************************************************************************/ --> | ||
− | + | ||
__TOC__ | __TOC__ | ||
Line 17: | Line 17: | ||
<!-- /***************************************** Background Information ****************************************/ --> | <!-- /***************************************** Background Information ****************************************/ --> | ||
<!-- /*********************************************************************************************************/ --> | <!-- /*********************************************************************************************************/ --> | ||
− | {{:Templateimpl:bg | initials= | + | {{:Templateimpl:bg | initials=MD | title=System Logging | desc=The page describes how to system log. | project=OE 5.0 }} |
+ | System Logging services exist on Linux systems to provide a central logging facility. This central logging service can store logged messages in files on the local machine, can forward the log messages to a remote machine for remote storage and display, and can output the log messages as they arrive to a terminal. | ||
+ | |||
+ | The syslog facility is standardized by POSIX and, as a result, there is a great deal of support for the syslog facility. There are APIs for logging to the syslog facility from C, C++, Python, Java, PHP, Perl, Bash, the commandline, and many other languages. Using the commandline tool, it is even possible to write to the syslog facility from the commandline. | ||
+ | |||
+ | ==== Facilities ==== | ||
+ | |||
+ | In syslog terminology, a facility is a category for log messages. There are generally at least seven standard syslog facilities on any Linux syslog system. The following list shows the standard seven as well as some which are rarely used anymore (especially in embedded systems). The name of the facility is followed, in parenthesis, by the integer facility code associated with the facility, which is then followed by a description of the facility. The ones you should not expect to find on an embedded system are demarcated with an asterisk (*): | ||
+ | |||
+ | ;kern(0) | ||
+ | : Kernel messages | ||
+ | ;user(1) | ||
+ | : User Level Messages | ||
+ | ;mail(2)* | ||
+ | : Mail System Messages | ||
+ | ;daemon(3) | ||
+ | : System Daemon Messages | ||
+ | ;auth(4) | ||
+ | : Security/Authentication Messages | ||
+ | ;syslog(5) | ||
+ | : Internally Generated Syslog Messages | ||
+ | ;lpr(6)* | ||
+ | : Line Printer Subsystem Messages | ||
+ | ;news(7)* | ||
+ | : Network News Subsystem Messages | ||
+ | ;uucp(8)* | ||
+ | : UUCP Subsystem Messages | ||
+ | ;cron(15) | ||
+ | : Cron Scheduling Daemon Messages | ||
+ | ;local(16-23) | ||
+ | : Locally Defined Usage, Messages | ||
+ | |||
+ | ==== Severity ==== | ||
+ | |||
+ | Each of these facilities accepts messages which will have an associated severity level. The severity level is important for filtering messages. The following list shows the eight available severity levels: | ||
+ | |||
+ | ;emerg(0) | ||
+ | : Emergency - System is now unusable | ||
+ | ;alert(1) | ||
+ | : Alert - Immediate Attention is Required | ||
+ | ;crit(2) | ||
+ | : Critical - The System is in a Critical Condition, which may be caused by a failure in the primary appliction. | ||
+ | ;err(3) | ||
+ | : Error - An Error has occurred. | ||
+ | ;warning(4) | ||
+ | : Warning - An unusual, but not erroneous, event has occurred. For example: "Warning: Could not check for updates." | ||
+ | ;notice(5) | ||
+ | : Notice - Uncommon but generally expected messages. For example: "Network Interface eth0 received IP address of 192.168.0.100 from DHCP server." | ||
+ | ;info(6) | ||
+ | : Informational - Common, expected events generate these messages. For example: "Application XYZ started successfully." | ||
+ | ;debug(7) | ||
+ | : Debug - Debugging messages belong at this level. | ||
+ | |||
+ | ==== Storage ==== | ||
+ | |||
+ | By default, the log messages on an EMAC OE Linux machine are stored in a local ramdisk. This type of storage prevents excessive wear of the flash, and works when all of the flash partitions are mounted read-only. However, by storing the messages in a ramdisk, all messages in the ramdisk will be lost as soon as the machine resets, powers off, or reboots for any reason. | ||
+ | |||
+ | ===== Alternate Logging Locations ===== | ||
+ | |||
+ | The syslog server can be configured to log to alternate locations. This is useful for configuring the server to log to persistent storage, such as a writable partition on the local flash or a partition on an SD card. The alternate logging locations can even be remote machines. | ||
+ | |||
+ | ===== Remote Logging ===== | ||
+ | |||
+ | The <code>/var/log</code> directory is the standard location for holding local log messages. This directory usually contains either one file per facility or one directory per facility. If a facility has a directory, it may contain different files which all belong to the facility. The system logging facility directs messages to these files based on rules set in its configuration. The configured rules use the facility level and the severity level to direct the syslogger's filters regarding to which file to send any particular message. There are several different implementations of the syslogger facility, each with its own configuration mechanisms and level of flexibility. | ||
+ | |||
+ | The system logger is also able to send messages to a remote syslog server over a network. The remote syslog server will be able to filter the messages using all of the same criteria as above, but also by source. A remote syslog server configuration can be very useful for remotely debugging deployed machines. While an embedded machine stores its logs on a ramdisk, the remote machine may store them to a traditional hard disk or SSD, allowing them to be preserved through power cycles. | ||
+ | |||
+ | The remote syslog server can be on any normal server, it can be on a developer's desktop, or it can be on another embedded machine. Sending debug messages to a central syslog server ensures the debug messages will be available even if the machine of interest freezes or resets. The use of a remote syslog server can also assist with looking for trends and outliers among a group of deployed machines since the log messages for the entire group of machines can be located in one place. | ||
+ | |||
+ | ===== Log Rotation ===== | ||
+ | |||
+ | Syslog servers incorporate a feature called log rotation. Log rotation is a process which prevents log files from growing beyond a certain size while preserving log messages for later reference. The syslog server can be configured with a maximum file size and a maximum number of rotated logs to keep. When the maximum file size is reached, the file will be "rotated." When the maximum number of logs to keep is exceeded, the oldest one will be deleted. There is more information on how this process works below. | ||
+ | |||
<!-- /*********************************************************************************************************/ --> | <!-- /*********************************************************************************************************/ --> | ||
<!-- /***************************************** General Information *****************************************/ --> | <!-- /***************************************** General Information *****************************************/ --> | ||
<!-- /*********************************************************************************************************/ --> | <!-- /*********************************************************************************************************/ --> | ||
− | {{:Templateimpl:geninfo | initials= | + | {{:Templateimpl:geninfo | initials=MD | title=System Logging | desc=The page describes how to system log. | project=OE 5.0 }} |
+ | |||
<!-- /*********************************************************************************************************/ --> | <!-- /*********************************************************************************************************/ --> | ||
<!-- /***************************************** Using/Working With ******************************************/ --> | <!-- /***************************************** Using/Working With ******************************************/ --> | ||
<!-- /*********************************************************************************************************/ --> | <!-- /*********************************************************************************************************/ --> | ||
− | {{:Templateimpl:using | initials= | + | {{:Templateimpl:using | initials=MD | title=System Logging | desc=The page describes how to system log. | project=OE 5.0 }} |
<!-- /*********************************************************************************************************/ --> | <!-- /*********************************************************************************************************/ --> | ||
<!-- /***************************************** Examples *****************************************/ --> | <!-- /***************************************** Examples *****************************************/ --> | ||
<!-- /*********************************************************************************************************/ --> | <!-- /*********************************************************************************************************/ --> | ||
− | {{:Templateimpl:examples | initials= | + | {{:Templateimpl:examples | initials=MD | title=System Logging | desc=The page describes how to system log. | project=OE 5.0 }} |
<!-- /*********************************************************************************************************/ --> | <!-- /*********************************************************************************************************/ --> | ||
<!-- /****************************************** Conclusion ******************************************/ --> | <!-- /****************************************** Conclusion ******************************************/ --> | ||
<!-- /*********************************************************************************************************/ --> | <!-- /*********************************************************************************************************/ --> | ||
− | {{:Templateimpl:conclusion | initials= | + | {{:Templateimpl:conclusion | initials=MD | title=System Logging | desc=The page describes how to system log. | project=OE 5.0 }} |
<!-- /*********************************************************************************************************/ --> | <!-- /*********************************************************************************************************/ --> | ||
<!-- /****************************************** More Information *****************************************/ --> | <!-- /****************************************** More Information *****************************************/ --> | ||
<!-- /*********************************************************************************************************/ --> | <!-- /*********************************************************************************************************/ --> | ||
− | {{:Templateimpl:moreinfo | initials= | + | {{:Templateimpl:moreinfo | initials=MD | title=System Logging | desc=The page describes how to system log. | project=OE 5.0 }} |
* | * | ||
− | {{:Templateimpl:whatnext | initials= | + | {{:Templateimpl:whatnext | initials=MD | title=System Logging | desc=The page describes how to system log. | project=OE 5.0 }} |
* | * |
Revision as of 11:33, 11 November 2015
Contents
Background
System Logging services exist on Linux systems to provide a central logging facility. This central logging service can store logged messages in files on the local machine, can forward the log messages to a remote machine for remote storage and display, and can output the log messages as they arrive to a terminal.
The syslog facility is standardized by POSIX and, as a result, there is a great deal of support for the syslog facility. There are APIs for logging to the syslog facility from C, C++, Python, Java, PHP, Perl, Bash, the commandline, and many other languages. Using the commandline tool, it is even possible to write to the syslog facility from the commandline.
Facilities
In syslog terminology, a facility is a category for log messages. There are generally at least seven standard syslog facilities on any Linux syslog system. The following list shows the standard seven as well as some which are rarely used anymore (especially in embedded systems). The name of the facility is followed, in parenthesis, by the integer facility code associated with the facility, which is then followed by a description of the facility. The ones you should not expect to find on an embedded system are demarcated with an asterisk (*):
- kern(0)
- Kernel messages
- user(1)
- User Level Messages
- mail(2)*
- Mail System Messages
- daemon(3)
- System Daemon Messages
- auth(4)
- Security/Authentication Messages
- syslog(5)
- Internally Generated Syslog Messages
- lpr(6)*
- Line Printer Subsystem Messages
- news(7)*
- Network News Subsystem Messages
- uucp(8)*
- UUCP Subsystem Messages
- cron(15)
- Cron Scheduling Daemon Messages
- local(16-23)
- Locally Defined Usage, Messages
Severity
Each of these facilities accepts messages which will have an associated severity level. The severity level is important for filtering messages. The following list shows the eight available severity levels:
- emerg(0)
- Emergency - System is now unusable
- alert(1)
- Alert - Immediate Attention is Required
- crit(2)
- Critical - The System is in a Critical Condition, which may be caused by a failure in the primary appliction.
- err(3)
- Error - An Error has occurred.
- warning(4)
- Warning - An unusual, but not erroneous, event has occurred. For example: "Warning: Could not check for updates."
- notice(5)
- Notice - Uncommon but generally expected messages. For example: "Network Interface eth0 received IP address of 192.168.0.100 from DHCP server."
- info(6)
- Informational - Common, expected events generate these messages. For example: "Application XYZ started successfully."
- debug(7)
- Debug - Debugging messages belong at this level.
Storage
By default, the log messages on an EMAC OE Linux machine are stored in a local ramdisk. This type of storage prevents excessive wear of the flash, and works when all of the flash partitions are mounted read-only. However, by storing the messages in a ramdisk, all messages in the ramdisk will be lost as soon as the machine resets, powers off, or reboots for any reason.
Alternate Logging Locations
The syslog server can be configured to log to alternate locations. This is useful for configuring the server to log to persistent storage, such as a writable partition on the local flash or a partition on an SD card. The alternate logging locations can even be remote machines.
Remote Logging
The /var/log
directory is the standard location for holding local log messages. This directory usually contains either one file per facility or one directory per facility. If a facility has a directory, it may contain different files which all belong to the facility. The system logging facility directs messages to these files based on rules set in its configuration. The configured rules use the facility level and the severity level to direct the syslogger's filters regarding to which file to send any particular message. There are several different implementations of the syslogger facility, each with its own configuration mechanisms and level of flexibility.
The system logger is also able to send messages to a remote syslog server over a network. The remote syslog server will be able to filter the messages using all of the same criteria as above, but also by source. A remote syslog server configuration can be very useful for remotely debugging deployed machines. While an embedded machine stores its logs on a ramdisk, the remote machine may store them to a traditional hard disk or SSD, allowing them to be preserved through power cycles.
The remote syslog server can be on any normal server, it can be on a developer's desktop, or it can be on another embedded machine. Sending debug messages to a central syslog server ensures the debug messages will be available even if the machine of interest freezes or resets. The use of a remote syslog server can also assist with looking for trends and outliers among a group of deployed machines since the log messages for the entire group of machines can be located in one place.
Log Rotation
Syslog servers incorporate a feature called log rotation. Log rotation is a process which prevents log files from growing beyond a certain size while preserving log messages for later reference. The syslog server can be configured with a maximum file size and a maximum number of rotated logs to keep. When the maximum file size is reached, the file will be "rotated." When the maximum number of logs to keep is exceeded, the oldest one will be deleted. There is more information on how this process works below.