Secure Your Logs to the Root

A presentation at DevOpsDays Boston in September 2019 in Boston, MA, USA by Quintessence Anx

Slide 1

Slide 1

SECURE YOUR LOGS DOWN TO THE ROOT QuintessenceAnx

Slide 2

Slide 2

@QuintessenceAnx /@AppDynamics Before I Get Started

Slide 3

Slide 3

@QuintessenceAnx /@AppDynamics There will be some text heavy slides. !”

Slide 4

Slide 4

@QuintessenceAnx /@AppDynamics There is a link to my slides & resources at the end.

Slide 5

Slide 5

@QuintessenceAnx /@AppDynamics Let’s Dive In.

Slide 6

Slide 6

@QuintessenceAnx /@AppDynamics Quick Overview of Terms and Concepts* *Not an exhaustive list.

Slide 7

Slide 7

@QuintessenceAnx /@AppDynamics Hash: obscuring data (one-way)

Slide 8

Slide 8

@QuintessenceAnx /@AppDynamics Pinch of salt #

Slide 9

Slide 9

@QuintessenceAnx /@AppDynamics Encrypt: obscuring data (reversibly)

Slide 10

Slide 10

@QuintessenceAnx /@AppDynamics Try to avoid bloating the term “security”

Slide 11

Slide 11

@QuintessenceAnx /@AppDynamics Different Security Objectives* Confidentiality Integrity Availability Authentication Authorization Non-repudiation *Also not an exhaustive list.

Slide 12

Slide 12

@QuintessenceAnx /@AppDynamics Always be aware of your objective(s).

Slide 13

Slide 13

@QuintessenceAnx /@AppDynamics Oh, and what do I not mean by security?

Slide 14

Slide 14

@QuintessenceAnx /@AppDynamics No. Security Through Obscurity Do not do this.

Slide 15

Slide 15

@QuintessenceAnx /@AppDynamics ‘cause consequences

Slide 16

Slide 16

@QuintessenceAnx /@AppDynamics e.g. “They don’t know where ${X} is, right?” Who needs consistent naming conventions anyway?

Slide 17

Slide 17

@QuintessenceAnx /@AppDynamics

Slide 18

Slide 18

@QuintessenceAnx /@AppDynamics e.g. “Key management is hard, let’s share.” This isn’t your housemate.

Slide 19

Slide 19

@QuintessenceAnx /@AppDynamics

Slide 20

Slide 20

@QuintessenceAnx /@AppDynamics There are more, but I think you grok me. ☺

Slide 21

Slide 21

@QuintessenceAnx /@AppDynamics The main event: how does this apply to logs? %

Slide 22

Slide 22

@QuintessenceAnx /@AppDynamics Log Lifecycle Create Store Ship Consume Convert Destroy

Slide 23

Slide 23

@QuintessenceAnx /@AppDynamics Create Create Store Ship Consume Convert Destroy

Slide 24

Slide 24

@QuintessenceAnx /@AppDynamics Do not write sensitive data to your logs

Slide 25

Slide 25

@QuintessenceAnx /@AppDynamics Do not. write. sensitive data. to your logs.

Slide 26

Slide 26

@QuintessenceAnx /@AppDynamics Sensitive data, e.g.: Personally identifying information (PII) SSNs are high cardinality, right? Credentials, including passwords and keys e.g. ever version control your dotfiles? Keystrokes Matching results by either percent (e.g. X% match on FaceID or fingerprint) or pass/fail Financial or health data Internal endpoints and/or IP addresses Database queries The list goes on.

Slide 27

Slide 27

@QuintessenceAnx /@AppDynamics Essentially, log only what you need.

Slide 28

Slide 28

@QuintessenceAnx /@AppDynamics “What if I really need that sensitive data”, you ask?

Slide 29

Slide 29

@QuintessenceAnx /@AppDynamics Food for thought, this is CWE-532. So it comes up.

Slide 30

Slide 30

@QuintessenceAnx /@AppDynamics Don’t ship it - log around it, e.g.: Use a token that references the data Use a salted or low-sodium hash Encrypt the log and/or your data Redact data as needed Remember to adhere to any regulatory compliance requirements e.g. PCI, HIPAA

Slide 31

Slide 31

@QuintessenceAnx /@AppDynamics Now what to do with these logs? ☺

Slide 32

Slide 32

@QuintessenceAnx /@AppDynamics Store Create Store Ship Consume Convert Destroy

Slide 33

Slide 33

@QuintessenceAnx /@AppDynamics Batten Down the Hatches Limit access to the log files Limit access to the storage volume(s) they reside on Log files should be append only Encrypt where possible Take a look at forward secure sealing (FSS) if you’re encrypting your logs i.e. how to prevent past manipulation with current keys Rotate your log files regularly

Slide 34

Slide 34

@QuintessenceAnx /@AppDynamics Ship Create Store Ship Consume Convert Destroy

Slide 35

Slide 35

@QuintessenceAnx /@AppDynamics Actually shipping it this time If you are using a 3rd party / SaaS solution: Make sure your provider supports shippers that allow you to ship securely, e.g. over TLS / SSL via rsyslog. If using an on prem solution: Secure your network Ship encrypted Limit key access to central log server

Slide 36

Slide 36

@QuintessenceAnx /@AppDynamics Consume & Convert Create Store Ship Consume Convert Destroy

Slide 37

Slide 37

@QuintessenceAnx /@AppDynamics Safe Data Use For a SaaS solution: ensure they provide access control For an on prem solution: ensure you have access control Also: limit access to the log server itself Limit / deny malformed or malicious queries e.g. Elastic has a handy 2014 blog post (back in its youth) that explains a few ways to crash the then-current version of Elasticsearch (to help you start thinking about this topic).

Slide 38

Slide 38

@QuintessenceAnx /@AppDynamics Destroy Create Store Ship Consume Convert Destroy

Slide 39

Slide 39

@QuintessenceAnx /@AppDynamics Secure Destruction This also comes up often (CWE-117) Ensure that locally and remotely (if using a SaaS) that data is destroy according to relevant industry standards e.g. CESG CPA, Crypt Erase, NIST This may mean anything from wiping data to shredding physical storage, depending on your industry. Do you need to delete or wipe? Know the difference. Use the difference.

Slide 40

Slide 40

@QuintessenceAnx /@AppDynamics Closing Tips

Slide 41

Slide 41

@QuintessenceAnx /@AppDynamics Tip # 1: Know your data &

Slide 42

Slide 42

@QuintessenceAnx /@AppDynamics Tip # 2: Know your infrastructure ‘

Slide 43

Slide 43

@QuintessenceAnx /@AppDynamics Tip # 3: Know your risks (

Slide 44

Slide 44

@QuintessenceAnx /@AppDynamics Tip # 4: Don’t apply what doesn’t apply %

Slide 45

Slide 45

@QuintessenceAnx /@AppDynamics Tip # 5: Trust, but verify )

Slide 46

Slide 46

@QuintessenceAnx /@AppDynamics Tip # 6: Use your metrics *

Slide 47

Slide 47

@QuintessenceAnx /@AppDynamics Tip # 7: Protect & utilize your audit trail +

Slide 48

Slide 48

@QuintessenceAnx /@AppDynamics Tip # 8: Use well designed alerts judiciously ,

Slide 49

Slide 49

@QuintessenceAnx /@AppDynamics Tip # 9: Don’t be a target - find help as needed

Slide 50

Slide 50

@QuintessenceAnx /@AppDynamics Tip # 10: Prevention is the difference between This Is a Problem and This Is a Disaster. % .

Slide 51

Slide 51

@QuintessenceAnx /@AppDynamics Slides, References, & Reading Available on Notist https://noti.st/quintessence

Slide 52

Slide 52

Thank you! QuintessenceAnx Technical Evangelist / @