System logs are critical for monitoring the health, performance, and security of your Linux server. They provide a detailed record of events occurring on your system, from kernel operations to application activities and user actions. By effectively analyzing these logs, you can troubleshoot issues, identify security threats, and gain valuable insights into your server’s behavior. This guide will explore the fundamentals of Linux system logs, covering their locations, interpretation, and best practices for leveraging them to maintain a robust and efficient server environment.
Linux systems centralize most log files within the /var/log directory. This directory serves as the primary repository for logs generated by the operating system, system services, and various applications running on your server. Navigating this directory provides a comprehensive view of your server’s operational history.
Accessing the /var/log directory in a Linux terminal to view system logs
Within /var/log, several key files and subdirectories are essential for system administrators:
/var/log/syslogor/var/log/messages: These are general system logs that capture a wide range of system activity, including startup messages, system events, and application logs./var/log/syslogis commonly found on Debian-based distributions like Ubuntu, while/var/log/messagesis prevalent on Red Hat-based systems such as RHEL and CentOS. These logs are your first stop for investigating general server issues./var/log/auth.logor/var/log/secure: Security-related events, such as user login attempts, authentication successes and failures, and actions taken by the root user, are logged in these files. Ubuntu and Debian systems typically use/var/log/auth.log, whereas Red Hat and CentOS utilize/var/log/secure. Monitoring these logs is crucial for security auditing and threat detection./var/log/kern.log: This log file specifically records kernel-related events, warnings, and errors. It is invaluable for debugging kernel issues, hardware problems, or when working with custom kernel configurations./var/log/cron: Information about scheduled tasks managed by cron, including their execution status and any errors encountered, is stored in this log. This is helpful for verifying the proper execution of your automated server tasks.
In addition to these system-level logs, many applications create their own log directories within /var/log. For instance, Apache web server logs are usually found in /var/log/apache2 (on Debian-based systems), and MySQL/MariaDB server logs are typically located in /var/log/mysql. Furthermore, applications can also utilize Syslog for logging, which we will discuss next.
Delving into Syslog: The System Logging Protocol
Syslog is a standardized protocol designed for generating and transmitting log messages. The term “syslog” can refer to several related concepts:
- Syslog Protocol: A standard communication protocol that enables systems and applications to send log messages to a central logging server.
- Syslog Daemon: A background process (such as rsyslog or syslog-ng) running on a Linux server that collects, processes, and stores log messages.
- Syslog Message Format: A defined structure for log messages, ensuring consistency and facilitating parsing and analysis.
Syslog’s capability to forward log messages to remote servers is particularly beneficial for centralized log management. This feature allows system administrators to aggregate logs from multiple servers into a single platform like SolarWinds® Loggly® or SolarWinds Papertrail™, simplifying monitoring and analysis acrossfrastructures.
Understanding RFC 3164 and RFC 5424 Syslog Standards
While RFC 5424 represents the current Syslog protocol standard, the older RFC 3164 (also known as “BSD syslog” or “legacy syslog”) is still encountered. System administrators should be aware of both as they may encounter log messages formatted according to either standard.
Key indicators of an RFC 3164 syslog message include the absence of structured data and a timestamp format like “Mmm dd hh:mm:ss”.
Examples of BSD syslog messages (based on RFC 3164 section 5.4):
Nov 11 11:11:11 authpriv: su: 'su root' failed for user 'john' on /dev/pts/3
Dec 22 15:30:01 192.168.1.100 webserver: User 'alice' accessed /index.html
While RFC 3164 is still in use, modern systems and this guide primarily focus on the newer RFC 5424 protocol due to its enhanced features and structured approach.
Decoding the Syslog Message Format and Key Fields
Syslog messages adhering to RFC 5424 are structured with a standardized header containing several fields. These fields provide crucial context about each log event, including the timestamp, the originating application, the source location within the system, and the message priority. Although the format can be customized via syslog daemon configuration, adhering to the standard format simplifies parsing, analysis, and efficient routing of syslog events, especially when using log management tools for Linux log analysis.
Here’s an example of a syslog message in the default RFC 5424 format, originating from the SSH daemon (sshd) and indicating a failed login attempt:
Jul 15 08:45:30 yourserver sshd[12345]: Failed password for invalid user testuser from 192.168.1.10 port 22 ssh2
Syslog also allows for the inclusion of additional fields to enrich log data. Consider the following rsyslog template, which augments the standard output with priority, protocol version, and an RFC 3339 formatted timestamp:
%pri% %protocol-version% %timestamp:::date-rfc3339% %HOSTNAME% %app-name% %procid% %msgid% %msg%n
Applying this template to a similar SSH login failure event could generate a log message like this:
14 1 2023-07-15T08:45:30.123Z yourserver sshd - - pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=192.168.1.10
Let’s examine some of the most frequently used syslog fields, particularly relevant for searching and troubleshooting Linux server issues using logs.
Priority: Understanding Log Severity
The priority field, often shortened to pri (represented as <14> in the example above), signifies the urgency or severity of the logged event. It is derived from a combination of two numerical values: the facility and the severity. The facility denotes the source or type of process generating the log, ranging from 0 (kernel messages) to 23 (local applications). Severity levels, ranging from 0 to 7, indicate the impact of the event, with 0 being the highest severity (Emergency) and 7 the lowest (Debug).
The pri value can be expressed as a numerical value (prival) or a text string (pri-text). prival is calculated as (facility * 8) + severity. pri-text presents the priority as “facility.severity,” e.g., “auth.error.” While pri-text is more human-readable, prival is more compact for storage.
Timestamp: Log Event Time
The timestamp field (“2023-07-15T08:45:30.123Z” in the example) records the precise date and time when the log message was generated on the originating system. The format breakdown is as follows:
- “2023-07-15”: Year, month, and day.
- “T”: Separator between date and time, as per ISO 8601 standard.
- “08:45:30.123”: 24-hour time format including milliseconds.
- “Z”: Indicates UTC (Coordinated Universal Time). Time zone offsets like “-07:00” are also permissible, specifying deviation from UTC.
Hostname: Source Server Identification
The hostname field (“yourserver” in the example) clearly identifies the hostname or network address of the system that originally sent the log message. This is crucial in environments with multiple servers for pinpointing the source of events.
App-name: Originating Application
The app-name field (“sshd” in the example) specifies the name of the application or process that generated the log message. This allows for filtering and analyzing logs based on specific applications running on your Linux server.
Journald: Modern Logging with Systemd
Many contemporary Linux distributions utilize systemd, a system and service manager. Systemd incorporates its own logging service called journald, which can operate alongside or replace Syslog. Journald employs a fundamentally different logging approach compared to Syslog, warranting dedicated exploration. For in-depth information on journald, refer to resources like the Systemd Logging section in the Ultimate Guide to Logging and Using journalctl for Log Management.
Essential Resources for Linux Log Management
How to View and Configure Linux Logs on Ubuntu and CentOS (DigitalOcean) – A practical guide to managing Linux logs on popular distributions.
Last updated: 2023
Gain Visibility, Analyze, and Resolve Server Issues Effectively
Unlock the power of your system logs to proactively monitor your Linux server environment.
START FREE TRIAL
