Logging Best Practices
This is a guide to help engineers and engineering managers understand logging best practices. The focus is on security, but you may have compliance needs that may exceed or not require everything in this document. In that case, YMMV.
The best way to look at logging, is to understand everything around this question:
- What information do I need to investigate an information security incident?
- Who did something?
- What did they do?
- When did they do it?
- How did they do it?
This translates to:
- Authentication logs
- Authorization logs (sudo, etc)
- Timestamp accuracy
Who did something?
Okta, Google Workspace, AWS Cloudtrail, /var/log/auth,
What did they do?
When did they do it?
Logs are all timestamps in the same timezone (preferably) and sync with a reliable time server
How did they do it?
Key Access Logs (Indirect auth)
1Password Vault Access Logs, Shared account logs with IP Address, User-agent, and metadata
It's important to note that logging can become very unwieldy very quickly.
Some of the issues that have or will come up are:
- Logging too much information
- Other groups asking for access to logs
- Logging will become too expensive
- (Paying $/GB is so 2015)
- Unclear retention requirements
- Too much noise
- Additional sources are not added regularly
- Too many people have access to sensitive information
- Devs enable debug and chaos ensues
- Passwords logged in cleartext!
- Quota limits are hit
To address some of these, here are some guidelines:
In all cases, Engineering and Security need to collaborate on a logging solution that works for most teams (80/20).
We recommend your logs go to a central location where various teams (Security, Engineering, Data Analysts) can pull logs into their own respective tools. This allows the most flexibility and can be efficient when you have a larger and growing company (500+ employees).
You can also all decide to use one tool as well. It often comes down to a choice of cost and preferences.
Here is a guide on choosing a Cloud Friendly SIEM: https://www.cloudsecuritylabs.io/insights/what-to-look-for-in-a-cloud-friendly-siem
12-18 months is the generally accepted logging retention for most logs. If you are subject to HIPAA it may be up to 7 years.
Realtime vs Cold Storage
Keep in mind, you only need 90 days of logs for realtime access. The rest of the logs can be stored cheaply in something like Glacier. If you are paying for 6 months or more of realtime access it's probably a waste of money or a sign that you don't have a good logging tooling strategy.
RBAC -Role Based Access Controls
ABAC - Attribute Based Access Controls
Access to your logs should be limited on a need to know basis.
Log sources should limited based on role and/or attribute of the logs being accessed.
Authentication logs capture detailed information about a user or service authentication action.
- Access Granted or Denied requests
- Valid password, but invalid MFA
- IP Address
- Any other additional metadata
See this for details on configuring AWS Cloudtrail: https://docs.cloudsecuritylabs.io/aws-security-best-practices#zo-aws-cloudtrail-configuration
Where possible enable AWS Guardduty. Guardduty uses additional log sources such as VPC Flow Logs to detect malicious behaviour. Capture these logs to your logging tool as well.
Below are some additional sources around the web that may help you in your logging journey: