28min

Logging Best Practices

Summary

This is a guide to help engineers and engineering managers understand logging best practices. The focus is on security, but you may have compliance needs that may exceed or not require everything in this document. In that case, YMMV.

The best way to look at logging, is to understand everything around this question:

  • What information do I need to investigate an information security incident?
    • Who did something?
    • What did they do?
    • When did they do it?
    • How did they do it?

This translates to:

  • Authentication logs
  • Authorization logs (sudo, etc)
  • Timestamp accuracy

Question

Log Type

Examples

Who did something?

Authentication Logs

Okta, Google Workspace, AWS Cloudtrail, /var/log/auth,

What did they do?

Authorization Logs

sudo

When did they do it?

n/a

Logs are all timestamps in the same timezone (preferably) and sync with a reliable time server

How did they do it?

Key Access Logs (Indirect auth)

1Password Vault Access Logs, Shared account logs with IP Address, User-agent, and metadata

Logging Issues

It's important to note that logging can become very unwieldy very quickly.

Some of the issues that have or will come up are:

  • Logging too much information
  • Other groups asking for access to logs
  • Logging will become too expensive
    • (Paying $/GB is so 2015)
  • Unclear retention requirements
  • Too much noise
  • Additional sources are not added regularly
  • Too many people have access to sensitive information
  • Devs enable debug and chaos ensues
    • Passwords logged in cleartext!
    • Quota limits are hit

To address some of these, here are some guidelines:

Logging Best Practices

Centralized Logging

In all cases, Engineering and Security need to collaborate on a logging solution that works for most teams (80/20).

We recommend your logs go to a central location where various teams (Security, Engineering, Data Analysts) can pull logs into their own respective tools. This allows the most flexibility and can be efficient when you have a larger and growing company (500+ employees).

You can also all decide to use one tool as well. It often comes down to a choice of cost and preferences.

Retention

12-18 months is the generally accepted logging retention for most logs. If you are subject to HIPAA it may be up to 7 years.

Realtime vs Cold Storage

Keep in mind, you only need 90 days of logs for realtime access. The rest of the logs can be stored cheaply in something like Glacier. If you are paying for 6 months or more of realtime access it's probably a waste of money or a sign that you don't have a good logging tooling strategy.

RBAC and ABAC Access to Logs

RBAC -Role Based Access Controls

ABAC - Attribute Based Access Controls

Access to your logs should be limited on a need to know basis.

Log sources should limited based on role and/or attribute of the logs being accessed.



Authentication Logs

Authentication logs capture detailed information about a user or service authentication action.

This includes:

  • Access Granted or Denied requests
  • Valid password, but invalid MFA
  • Username
  • IP Address
  • User-Agent
  • Any other additional metadata



AWS Logging

AWS Cloudtrail

AWS Guardduty

Where possible enable AWS Guardduty. Guardduty uses additional log sources such as VPC Flow Logs to detect malicious behaviour. Capture these logs to your logging tool as well.

Additional Guides

Below are some additional sources around the web that may help you in your logging journey:



Updated 04 Jul 2022
Did this page help you?
Yes
No