CISA Advisory: Best Practices for Event Logging and Threat Detection

Executive Summary

This publication defines a baseline for event logging best practices to mitigate cyber threats. It was developed by the Australian Signals Directorate’s Australian Cyber Security Centre (ASD’s ACSC) in cooperation with the following international partners: 

• United States (US) Cybersecurity and Infrastructure Security Agency (CISA), the Federal Bureau of Investigation (FBI) and the National Security Agency (NSA).
• United Kingdom (UK) National Cyber Security Centre (NCSC-UK).
• Canadian Centre for Cyber Security (CCCS).
• New Zealand National Cyber Security Centre (NCSC-NZ) and Computer Emergency Response Team (CERT NZ).
• Japan National Center of Incident Readiness and Strategy for Cybersecurity (NISC) and Computer Emergency Response Team Coordination Center (JPCERT/CC).
• The Republic of Korea National Intelligence Services (NIS) and NIS’s National Cyber Security Center (NCSC-Korea).
• Singapore Cyber Security Agency (CSA).
• The Netherlands General Intelligence and Security Service (AIVD) and Military Intelligence and Security Service (MIVD).

Event logging supports the continued delivery of operations and improves the security and resilience of critical systems by enabling network visibility. This guidance makes recommendations that improve an organization’s resilience in the current cyber threat environment, with regard for resourcing constraints. The guidance is of moderate technical complexity and assumes a basic understanding of event logging.

An effective event logging solution aims to:

• Send alerts to the network defenders responsible for monitoring when cyber security events such as critical software configuration changes are made or new software solutions are deployed.
• Identify cyber security events that may indicate a cyber security incident, such as malicious actors employing living off the land (LOTL) techniques or lateral movement post-compromise.
• Support incident response by revealing the scope and extent of a compromise.
• Monitor account compliance with organizational policies.
• Reduce alert noise, saving on costs associated with storage and query time.
• Enable network defenders to make agile and informed decisions based on prioritization of alerts and analytics.
• Ensure logs and the logging platforms are useable and performant for analysts.

There are four key factors to consider when pursuing logging best practices:

1. Enterprise-approved event logging policy.
2. Centralized event log access and correlation.
3. Secure storage and event log integrity.
4. Detection strategy for relevant threats.

To access the PDF version of this report, visit here.

Introduction

The increased prevalence of malicious actors employing LOTL techniques, such as LOTL binaries (LOLBins) and fileless malware, highlights the importance of implementing and maintaining an effective event logging solution. As demonstrated in the joint-sealed publication Identifying and Mitigating Living Off the Land Techniques, advanced persistent threats (APTs) are employing LOTL techniques to evade detection. The purpose of this publication is to detail best practice guidance for event logging and threat detection for cloud services, enterprise networks, enterprise mobility, and operational technology (OT) networks. The guidance in this publication focuses on general best practices for event logging and threat detection; however, LOTL techniques feature as they provide a great case study due to the high difficulty in detecting them.

Audience

This guidance is technical in nature and is intended for those within medium to large organizations. As such, it is primarily aimed at:

• Senior information technology (IT) and OT decision makers.
• IT and OT operators.
• Network administrators.
• Critical infrastructure providers.

Best Practices

Enterprise-approved Event Logging Policy

Developing and implementing an enterprise approved logging policy improves an organization’s chances of detecting malicious behavior on their systems and enforces a consistent method of logging across an organization’s environments. The logging policy should take into consideration any shared responsibilities between service providers and the organization. The policy should also include details of the events to be logged, event logging facilities to be used, how event logs will be monitored, event log retention durations, and when to reassess which logs are worthy of collection.

Event Log Quality

Organizations are encouraged to implement an event logging policy focused on capturing high-quality cyber security events to aid network defenders in correctly identifying cyber security incidents. In the context of cyber security incident response and threat detection, event log quality refers to the types of events collected rather than how well a log is formatted. Log quality can vary between organizations due to differences in network environments, the reason behind the need to log, differences in critical assets and the organization’s risk appetite. 

Useful event logs enrich a network defender’s ability to assess security events to identify whether they are false positives or true positives. Implementing high-quality logging will aid network defenders in discovering LOTL techniques that are designed to appear benign in nature.

Note: Capturing a large volume of well-formatted logs can be invaluable for incident responders in forensics analysis scenarios. However, organizations are encouraged to properly organize logged data into ‘hot’ data storage that is readily available and searchable, or ‘cold’ data storage that has deprioritized availability and is stored through more economical solutions – an important consideration when evaluating an organization’s log storage capacity.

For more information on how to prioritize collection of high-quality event logs please refer to CISA’s Guidance for Implementing M-21-3: Improving the Federal Government’s Investigative and Remediation Capabilities.[1] 

To strengthen detection of malicious actors employing LOTL techniques, some relevant considerations for event logging include:

• On Linux-based systems, logs capturing the use of curl, systemctl, systemd, python and other common LOLBins leveraged by malicious actors.
• On Microsoft Windows-based systems, logs capturing the use of wmic.exe, ntdsutil.exe, Netsh, cmd.exe, PowerShell, mshta.exe, rundll32.exe, resvr32.exe and other common LOLBins leveraged by malicious actors. Ensure that logging captures command execution, script block logging and module logging for PowerShell, and detailed tracking of administrative tasks.
• For cloud environments, logging all control plane operations, including API calls and end user logins. The control plane logs should be configured to capture read and write activities, administrative changes, and authentication events.

Captured Event Log Details

As a part of an organization’s event logging policy, captured event logs should contain sufficient detail to aid network defenders and incident responders. If a logging solution fails to capture data relevant to security, its effectiveness as a cyber security incident detection capability is heavily impacted.

The US Office of Management and Budget’s M-21-31[2] outlines a good baseline for what an event log should capture, if applicable:

• Properly formatted and accurate timestamp (millisecond granularity is ideal).
• Event type (status code).
• Device identifier (mac address or other unique identifier).
• Session/transaction ID.
• Autonomous system number.
• Source and destination IP (includes both IPv4 and IPv6).
• Status code.
• Response time.
• Additional headers (e.g., HTTP headers).
• The user ID, where appropriate.
• The command executed, where appropriate.
• A unique event identifier to assist with event correlation, where possible.

Note: Where possible, all data should be formatted as ‘key-value-pairs’ to allow for easier extraction.

Operational Technology Considerations

Network administrators and network operators should take into consideration the OT devices within their OT networks. Most OT devices use embedded software that is memory and/or processor constrained. An excessive level of logging could adversely affect the operation of those OT devices. Additionally, such OT devices may not be capable of generating detailed logs, in which case, sensors can be used to supplement logging capabilities. Out-of-band log communications, or generating logs based on error codes and the payloads of existing communications, can account for embedded devices with limited logging capabilities.

Additional Resources

Content and Format Consistency

When centralizing event logs, organizations should consider using a structured log format, such as JSON, where each type of log captures and presents content consistently (that is, consistent schema, format, and order). This is particularly important when event logs have been forwarded to a central storage facility as this improves a network defender’s ability to search for, filter and correlate event logs. Since logs may vary in structure (or lack thereof), implementing a method of automated log normalization is recommended. This is an important consideration for logs that can change over time or without notice such as software and software-as-a-service (SaaS) logs.

Timestamp Consistency

Organizations should consider establishing an accurate and trustworthy time source and use this consistently across all systems to assist network defenders in identifying connections between event logs. This should also include using the same date-time format across all systems. Where possible, organizations should use multiple accurate time sources in case the primary time source becomes degraded or unavailable. Note that, particularly in distributed systems, time zones and distance can influence how timestamps read in relation to each other. Network owners, system owners and cyber security incident responders are encouraged to understand how this could impact their own environments. ASD and co-authors urge organizations to consider implementing the recommendations below to help ensure consistent timestamp collection.

• Time servers should be synchronized and validated throughout all environments and set to capture significant events, such as device boots and reboots.
• Using Coordinated Universal Time (UTC) has the advantage of no time zones as well as no daylight savings, and is the preferred time standard.
  • Implement ISO 8601 formatting, with the year listed first, followed by the month, day, hour, minutes, seconds, and milliseconds (e.g., 2024-07-25T20:54:59.649Z).
• Timesharing should be unidirectional. The OT environment should synchronize time sync with the IT environment and not the other way around.
• Data historians may be implemented on some operational assets to record and store time-series data of industrial processes running on the computer system. These can provide an additional source of event log data for OT networks.

Additional Resources

• ASD has released Windows Event Logging and Forwarding guidance that details important event categories and recommendations for configurations, log retention periods and event forwarding.
• For more information about logging, please explore CISA’s Logging Made Easy (LME), a no-cost solution providing essential log management for small to medium-sized organizations, on CISA’s website or GitHub page.
• The Joint SIGINT Cyber Unit (JSCU) of the AIVD and MIVD has published a repository on GitHub with a Microsoft Windows event logging and collections baseline focused on finding balance between forensic value and optimizing retention. You can find this repository on the JSCU’s GitHub.

Event Log Retention

Organizations should ensure they retain logs for long enough to support cyber security incident investigations. Default log retention periods are often insufficient. Log retention periods should be informed by an assessment of the risks to a given system. When assessing the risks to a system, consider that in some cases, it can take up to 18 months to discover a cyber security incident and some malware can dwell on the network from 70 to 200 days before causing overt harm.[3] Log retention periods should also be compliant with any regulatory requirements and cyber security frameworks that may apply in an organization’s jurisdiction. Logs that are crucial in confirming an intrusion and its impact should be prioritized for longer retention. 

It is important to review log storage allocations, in parallel with retention periods. Insufficient storage is a common obstacle to log retention. For example, many systems will overwrite old logs when their storage allocation is exhausted. The longer that logs can be kept, the higher the chances are of determining the extent of a cyber security incident, including the potential intrusion vectors that require remediation. For effective security logging practices, organizations should implement data tiering such as hot and cold storage. This ensures that logs can be promptly retrieved to facilitate querying and threat detection.

Centralized Log Collection and Correlation

The following sections detail prioritized lists of log sources for enterprise networks, OT, cloud computing and enterprise mobility using mobile computing devices. The prioritization takes into consideration the likelihood that the logged asset will be targeted by a malicious actor, as well as the impact if the asset were to be compromised. It also prioritizes log sources that can assist in identifying LOTL techniques. Please note that this is not an exhaustive list of log sources and their threats, and their priority may differ between organizations.

Logging Priorities for Enterprise Networks

Enterprise networks face a large variety of cyber threats. These include malware, malicious insiders, and exploitation of unpatched applications and services. In the context of LOTL, enterprise networks provide malicious actors with a wide variety of native tools to exploit.

ASD and co-authors recommend that organizations prioritize the following log sources within their enterprise network:

1. Critical systems and data holdings likely to be targeted.
2. Internet-facing services, including remote access, network metadata, and their underlying server operating system.
3. Identity and domain management servers.
4. Any other critical servers.
5. Edge devices such as boundary routers and firewalls.
6. Administrative workstations.
7. Highly privileged systems such as configuration management, performance and availability monitoring (in cases where privileged access is used), Continuous Integration/Continuous Delivery (CI/CD), vulnerability scanning services, secret and privilege management.
8. Data repositories.
9. Security-related and critical software.
10. User computers.
11. User application logs.
12. Web proxies used by organizational users and service accounts.
13. DNS services used by organizational users.
14. Email servers.
15. DHCP servers.
16. Legacy IT assets (that are not previously captured in critical or internet-facing services).

ASD and co-authors recommend organizations monitor lower priority logs as well. These include:

• Underlying infrastructure, such as hypervisor hosts.
• IT devices, such as printers.
• Network components such as application gateways.

Logging Priorities for Operational Technology

Historically, IT and OT have operated separately and have provided distinct functions within organizations. Advancements in technology and digital transformation have led to the growing interconnectedness and convergence of these networks. Organizations are integrating IT and OT networks to enable the seamless flow of data between management systems and industrial operations. Their integration has introduced new cyber threats to OT networks. For example, malicious actors can access OT networks through IT networks by exploiting unpatched vulnerabilities, delivering malware, or conducting denial-of-service campaigns to impact critical services. 

ASD and co-authors recommend that organizations prioritize the following log sources in their OT environment:

1. OT devices critical to safety and service delivery, except for air-gapped systems.[4]
2. Internet-facing OT devices.
3. OT devices accessible via network boundaries.

Note that in cases where OT devices do not support logging, device logs are not available, or are available in a non-standard format, it is good practice to ensure network traffic and communications to and from the OT devices are logged.

Logging Priorities for Enterprise Mobility Using Mobile Computing Devices

Enterprise mobility is an important aspect of an organization’s security posture. Mobile device management (MDM) solutions allow organizations to manage the security of their enterprise mobility, typically including logging functionality. In the context of enterprise mobility, the aim of effective event logging is to detect compromised accounts or devices; for example, due to phishing or interactions with malicious applications and websites.

ASD and co-authors recommend organizations priorities the following log sources in their enterprise mobility solution:

1. Web proxies used by organizational users.
2. Organization operated DNS services.
3. Device security posture of organizationally managed devices.
4. Device behavior of organizationally managed devices.
5. User account behavior such as sign-ins.
6. VPN solutions.
7. MDM and Mobile Application Management (MAM) events.[5]

Additional monitoring should be implemented in collaboration with the telecommunications network provider. Such monitoring includes:

• Signaling exploitation.
• Binary/invisible SMS.
• CLI spoofing.
• SIM/eSIM activities such as SIM swapping.
• Null cipher downgrade.
• Connection downgrade (false base station).
• Network API/query against user.
• Roaming traffic protection.
• Roaming steering.

Organizations should obtain legal advice about what can be logged from any personally owned mobile devices that are enrolled in an MDM solution. For example, logging GPS location may be subject to restrictions.

Logging Priorities for Cloud Computing

ASD and co-authors recommend organizations adjust event logging practices in accordance with the cloud service that is administered, whether infrastructure-as-a-service (IaaS), platform-as-a-service (PaaS), or SaaS are implemented.  For example, IaaS would include a significant amount of logging responsibility on the tenant, whereas SaaS would place a significant amount of the logging responsibility on the provider. Therefore, organizations should coordinate closely with their cloud service provider to understand the shared-responsibility model in place, as it will influence their logging priorities. Logging priorities will also be influenced by different cloud computing service models and deployment models (that is, public, private, hybrid, community). Where privacy and data sovereignty laws apply, logging priorities may also be influenced by the location of the cloud service provider’s infrastructure. See NSA’s Manage Cloud Logs for Effective Threat Hunting guidance for additional information.

Organizations should prioritize the following log sources in their use of cloud computing services:

1. Critical systems and data holdings likely to be targeted.
2. Internet-facing services (including remote access) and, where applicable, their underlying server operating systems.
3. Use of the tenant’s user accounts that access and administer cloud services.
4. Logs for administrative configuration changes.
5. Logs for the creation, deletion and modification of all security principals, including setting and changing permissions.
6. Authentication success and/or failures to third party services (e.g., SAML/OAuth).
7. Logs generated by the cloud services, including logs for cloud APIs, all network-related events, compliance events and billing events.

Secure Storage and Event Log Integrity

ASD and co-authors recommend that organizations implement a centralized event logging facility such as a secured data lake to enable log aggregation and then forward select, processed logs to analytic tools, such as security information and event management (SIEM) solution and extended detection and response (XDR) solutions. Many commercially available network infrastructure devices have limited local storage. Forwarding event logs to a centralized and secure storage capability prevents the loss of logs once the local device’s storage is exhausted [CPG 2.U]. This can be further mitigated by ensuring default maximum event log storage sizes are configured appropriately on local devices. In the event of a cyber security incident, an absence of historical event logs will frequently have a negative impact on cyber security incident response activities. 

Secure Transport and Storage of Event Logs

ASD and co-authors recommend that organizations implement secure mechanisms such as Transport Layer Security (TLS) 1.3 and methods of cryptographic verification to ensure the integrity of event logs in-transit and at rest. Organizations should prioritize securing and restricting access to event logs that have a justified requirement to record sensitive data.

Protecting Event Logs from Unauthorized Access, Modification and Deletion

It is important to perform event log aggregation as some malicious actors are known to modify or delete local system event logs to avoid detection and to delay or degrade the efficacy of cyber security incident response. Logs may contain sensitive data that is useful to a malicious actor. As a result, users should only have access to the event logs they need to do their job.

An event logging facility should enable the protection of logs from unauthorized modification and deletion. Ensure that only personnel with a justified requirement have permission to delete or modify event logs and view the audit logs for access to the centralized logging environment.  The storage of logs should be in a separate or segmented network with additional security controls to reduce the risk of logs being tampered with in the event of network or system compromise. Events logs should also be backed up and data redundancy practices should be implemented.

Organizations are encouraged to harden and segment their SIEM solutions from general IT environments. SIEMs are attractive targets for malicious actors because they contain a wealth of information, provide an analysis function, and can be a single point of failure in an organization’s detection capability.  Organizations should consider filtering event logs before sending them to a SIEM or XDR to ensure it is receiving the most valuable logs to minimize any additional costs or capacity issues.

Centralized Event Logging Enables Threat Detection

The aggregation of event logs to a central logging facility that a SIEM can draw from enables the identification of: 

• Deviations from a baseline.
  • A baseline should include installed tools and software, user account behavior, network traffic, system intercommunications and other items, as applicable. Particular attention should be paid to privileged user accounts and critical assets such as domain controllers.
  • A baseline is derived by performing an analysis of normal behavior of some user accounts and establishing ‘always abnormal’ conditions for those same accounts.
• Cyber security events.
  • For the purpose of this document, a cyber security event is an occurrence of a system, service or network state indicating a possible breach of security policy, failure of safeguards or a previously unknown situation that may be relevant to security.
• Cyber security incidents.
  • For the purpose of this document, a cyber security incident is an unwanted or unexpected cyber security event, or a series of such events, that either has compromised business operations or has a significant probability of compromising business operations.

Timely Ingestion

Timely ingestion of event logs is important in the early detection of a cyber security events and cyber security incidents. If the generation, collection and ingestion of event logs is delayed, the organization’s ability to identify cyber security incidents is also delayed.

Detection Strategy for Relevant Threats

Detecting Living Off the Land Techniques

ASD and co-authors recommend that organizations consider implementing user and entity behavioral analytics capabilities to enable automated detection of behavioral anomalies on networks, devices, or accounts. SIEMs can detect anomalous activity by comparing event logs to a baseline of business-as-usual traffic and activity. Behavioral analytics plays a key role in detecting malicious actors employing LOTL techniques. Below is a case study that shows how threat actors leveraged LOTL to infiltrate Windows-based systems.

Case study – Volt Typhoon Since mid-2021, Volt Typhoon has targeted critical infrastructure organizations by relying almost exclusively on LOTL techniques. Their campaign has been enabled by privately-owned SOHO routers, infected with the ‘KV Botnet’ malware.  Volt Typhoon uses PowerShell, a command and scripting interpreter, to: Discover remote systems [T1059.001, T1018]. Identify associated user and computer account names using the command  Get-EventLog security –instanceid 4624 [T1033]. Enumerate event logs to search for successful logons using wevtutil.exe and the command Get-EventLog Security [T1654]. Volt Typhoon consistently obtains valid credentials by extracting the Active Directory database file NTDS.dit.[6]  To do so, Volt Typhoon has been observed to: Execute the Windows-native vsssadmin command to create a volume shadow copy [T1006]. Use Windows Management Instrumentation Console (WMIC) commands [T1047] to execute ntdsutil.exe to copy NTDS.dit and the SYSTEM registry from the volume shadow copy. Move laterally to the Microsoft Active Directory Domain Services (AD DS) domain controller via an interactive RDP session using a compromised user account with domain administrator privileges [T1021.001]. Other LOTL techniques that Volt Typhoon has been observed to use includes: Accessing hashed credentials from the Local Security Authority SubSystem Service (LSASS) process memory space [T1003.001]. Using ntdsutil.exe to create installation media from Microsoft AD DS domain controllers, either remote or locally, which contain username and password hashes [T1003.003]. Using PowerShell, WMIC, and the ping command, to facilitate system discovery [T1018]. Using the built-in netsh portproxy command to create proxies on compromised systems to facilitate access [T1090]. While Volt Typhoon uses LOTL techniques to make detection more difficult, the behaviors that the malware exhibits would be considered abnormal compared to business-as-usual activity and could be used to create detection use cases. For more information, consider visiting MITRE ATT&CK®’s Volt Typhoon page and the MITRE ATT&CK framework.