Tips for viewing logs of industrial control computer systems

Industrial Control Computer System Log Review: Practical Techniques for Troubleshooting and Monitoring
System logs are critical for diagnosing issues, tracking performance, and ensuring reliability in industrial control environments. This guide provides actionable methods for accessing, analyzing, and interpreting logs on industrial control computers without relying on proprietary tools or commercial software.

1. Locating and Accessing System Logs

Default Log Paths
Industrial control systems running Linux typically store logs in /var/log/. Key directories include:

• /var/log/syslog: General system messages (e.g., kernel alerts, service startups).

• /var/log/messages: Legacy log file for system-wide events (common in older distributions).

• /var/log/dmesg: Kernel ring buffer logs, useful for hardware-related errors.

• /var/log/auth.log: Security-related events, such as user logins and sudo commands.

Real-Time Log Monitoring
Use tail to view logs as they are written, which is helpful for debugging ongoing issues:

bashtail -f /var/log/syslog

Add filters to focus on specific services (e.g., network-related logs):

bashtail -f /var/log/syslog | grep "network"

Log Rotation and Archiving
Systems often use logrotate to manage log file sizes. Check configuration files in /etc/logrotate.d/ to understand retention policies. For historical analysis, access archived logs in /var/log/ with timestamps (e.g., syslog.1.gz).

2. Analyzing Log Content for Specific Issues

Identifying Hardware Failures
Kernel logs (dmesg or /var/log/kern.log) reveal hardware-related errors. Look for patterns like:

• PCIe device errors (e.g., "PCIe Bus Error").

• Disk I/O failures (e.g., "SCSI disk error").

• USB device disconnections (e.g., "USB device disconnected").

Example command to filter kernel errors:

bashdmesg | grep -i "error\|fail\|warn"

Debugging Service Outages
Systemd-based systems log service states in /var/log/syslog or journal files. Check for failed services:

bashjournalctl -u <service-name> --no-pager -n 50

Replace <service-name> with the target service (e.g., networkd, apache2). Look for lines containing "Failed" or "Exited with code".

Security Event Analysis
Audit user activities and unauthorized access attempts via /var/log/auth.log. Common indicators include:

• Failed login attempts: "Failed password for user".

• Root access attempts: "sudo: a user is not in the sudoers file".

• SSH brute-force attacks: Multiple "Connection from <IP> port <port>" entries.

Use grep to isolate suspicious events:

bashgrep "failed\|invalid" /var/log/auth.log

3. Advanced Log Filtering and Correlation

Combining Multiple Log Sources
Cross-reference logs from different services to pinpoint root causes. For example, correlate network outages (syslog) with service restarts (journalctl):

bashgrep "network down" /var/log/syslog | while read line; do echo "$line"; journalctl -u networking --since "$(echo "$line" | awk '{print $1, $2}')" --no-pager; done

Time-Based Analysis
Use timestamps to track event sequences. For instance, identify if a hardware failure preceded a service crash:

bashawk '{print $1, $2}' /var/log/syslog | sort | uniq -c

This aggregates logs by timestamp to detect spikes in activity.

Regular Expression Patterns
Leverage regex to extract complex patterns. To find all errors related to a specific module (e.g., "eth0"):

bashgrep -E "error.*eth0|fail.*eth0" /var/log/syslog

4. Log Management Best Practices

Centralized Log Collection
For distributed industrial systems, aggregate logs to a central server using rsyslog or syslog-ng. Configure clients to forward logs:

bash# On client machine (edit /etc/rsyslog.conf)*.* @@central-log-server:514

Log Retention Policies
Define retention rules in /etc/logrotate.d/ to avoid disk saturation. Example configuration for rotating logs weekly and keeping 4 copies:

/var/log/syslog {weeklyrotate 4missingoknotifemptycompress}

Anomaly Detection Scripts
Write custom scripts to flag unusual patterns. For example, a Bash script to email alerts when error counts exceed a threshold:

bash#!/bin/bashERROR_COUNT=$(grep -c "error" /var/log/syslog)if [ "$ERROR_COUNT" -gt 10 ]; thenecho "High error count detected: $ERROR_COUNT" | mail -s "Log Alert" admin@example.comfi

5. Troubleshooting Common Log-Related Issues

Missing Logs
If logs are not being written:

• Verify rsyslog is running: systemctl status rsyslog.

• Check disk space: df -h /var/log/.

• Ensure the log facility is enabled in /etc/rsyslog.conf.

Corrupted Log Files
Recover from corrupted logs by truncating the file (after stopping the service writing to it):

bash> /var/log/syslogsystemctl restart rsyslog

Permission Errors
Ensure the log directory has correct permissions:

bashchown -R syslog:adm /var/log/chmod -R 755 /var/log/

Conclusion

Effective log review in industrial control systems hinges on understanding log locations, filtering for critical events, and correlating data across services. By mastering commands like grep, journalctl, and dmesg, engineers can proactively identify hardware faults, service disruptions, and security breaches. Implementing centralized logging and retention policies further enhances long-term system reliability.