Log Aggregation - Definition & Overview

When professional software developers build an application, they always include a built-in logging function that keeps track of events that happen within the application. When an event happens within the application, the function automatically generates a log that records the event, along with additional metadata about the conditions surrounding the event, and writes the record into a file called a log file. Log files are used by programmers in debugging to help determine the root cause of an error, but they can also be useful for users that wish to monitor the performance, security status and general behavior of an application.

As business IT environments increase in complexity and IT organizations deploy an increasing number of applications and infrastructure in public and hybrid cloud environments, there is a growing need to maintain central control of application security and performance. The IT organization may review the log file from each individual application as a way of monitoring application status, but it would be much more useful to bring all of that data together into a common platform.

Log aggregation is a software function that consolidates log data from throughout the IT infrastructure into a single centralized platform where it can be reviewed and analyzed. Log aggregation software tools may support additional functionality, such as data normalization, log search, and complex data analysis. Log aggregation is just one aspect of an overall log management process that produces real-time insights into application security and performance.

Log aggregation software tools capture event log files from applications and other sources within the IT infrastructure. Event logs are automatically computer-generated when certain types of events occur within the application. Event logs may also be classified according to the severity of event and the required urgency of response. Event logs should fall into one of the following categories:

Information

Informational log documents changes in the state of the application or changes in entities within the application. Information logs are useful for determining what actually happened in the application during a specified time period. An information log might be created when:

• A schedule batch job completes
• The application successfully loaded
• A user copied some files
• A driver initialized correctly

Information logs focus on tasks that are completed successfully, while other log classifications are used for reporting unsuccessful operations.

Errors/Application Failures

When the application experiences an error, it should automatically generate a log with the error categorization. An error means that the application is functioning incorrectly with no opportunity to recover. The error may be affecting users in the production environment, resulting in service interruptions and poor customer experience. Error logs must be addressed immediately to minimize the impact of any application error that impacts a critical service.

Warnings/Application Malfunctions

A warning log might be triggered if an application tries to do an operation and fails, but still has the opportunity to recover and deliver the service.

Warnings and errors are relatively similar, so consider the following distinction:

• If the user performs an action that calls Database X and the application crashes, the result should be an error log
• If the user performs an action that calls Database X and it takes 20 seconds longer than expected, the result should be a warning log

Warning logs are not as urgent as errors, but they should be addressed relatively quickly to avoid negatively impacting customer service.

Positive Security Events

Most applications will generate a login response to the successful completion of a security event. This includes when a user logs on to the computer when the user logs into a database or an application when the user answers a security question or completes another form of authentication (time-based one-time password, biometric data, location, out-of-band authentication, etc.)

Negative Security Events

In addition to logging success audits, log aggregation tools also keep track of failed security events. Anytime a user enters the wrong password, answers a security question incorrectly or otherwise fails to authenticate access to the system, a log will be generated that documents the event.

In addition to the event type, each event log typically includes:

• The data that the event occurred
• The time that the event occurred
• A description of the event, including an error code if applicable
• The user profile that was active when the event occurred
• The name of the computer or network endpoint where the event occurred
• An event identification number for reference
• The source of the event

Log aggregation is part of the overall log management process that helps IT organizations convert their log files into actionable insights in real-time or near real-time. The process can be described in five basic steps:

1. Instrument & Collect - The first step of log management is to start collecting logs. IT organizations must implement log collector software tools that collect data from various parts of the software stack. Many devices across platforms generate logs using the Syslog message logging standard or with other applications that can write logs directly into the log aggregation tool platform.
2. Centralize & Index - Log data needs to be normalized and indexed, making it easier to analyze and fully searchable for developers and security analysts.
3. Search & Analyze - Now that the log data is organized properly in the log aggregation tool, it can be searched and analyzed to discover patterns and identify any issues that require attention from IT operators. Human or machine learning analysis can be used to identify patterns and anomalies.
4. Monitor & Alert - Effective log monitoring is a critical aspect of the log management process. An effective log management tool should integrate with message applications to deliver timely alerts when events occur that require a prompt response.
5. Report & Dashboard - The final component of log management, reporting and dashboarding ensure that team members across departments have the necessary levels of access and visibility into application performance data.

Log aggregation allows IT organizations to bring together data from across public and hybrid cloud environments into a single platform where it can be searched and analyzed together. This process increases the visibility of cloud-based computing environments, helps security analysts respond more quickly to security threats and provides real-time insight into application performance.

Sumo Logic provides exceptional log aggregation functionality, with the ability to collect logs from almost any system and format and bring them together to support an end-to-end log management process. With our machine learning log analysis, IT organizations can turn millions of event log data points into actionable insights that support application security and performance excellence.