Can you keep a secret?

A presentation at PyGotham in October 2019 in New York, NY, USA by Aaron Bassett

Slide 1

Slide 1

Can you keep a secret? @aaronbassett

Slide 2

Slide 2

Aaron Bassett Developer Advocate

Slide 3

Slide 3

Slide 4

Slide 4

import nexmo client = nexmo.Client( key=”a925db1ar392”, secret=”01nd637fn29oe31mc721” ) client.send_message({ “from”: “Python”, “to”: “447700900981”, “text”: “Hello world” })

Slide 5

Slide 5

import nexmo client = nexmo.Client( key=”a925db1ar392”, secret=”01nd637fn29oe31mc721” ) client.send_message({ “from”: “Python”, “to”: “447700900981”, “text”: “Hello world” })

Slide 6

Slide 6

import nexmo NEXMO_KEY = “a925db1ar392” NEXMO_SECRET = “01nd637fn29oe31mc721” MY_NUMBER = “447700900981” client = nexmo.Client(key=NEXMO_KEY, secret=NEXMO_SECRET) client.send_message({ “from”: “Python”, “to”: MY_NUMBER, “text”: “Hello world” })

Slide 7

Slide 7

import nexmo NEXMO_KEY = “a925db1ar392” NEXMO_SECRET = “01nd637fn29oe31mc721” MY_NUMBER = “447700900981” client = nexmo.Client(key=NEXMO_KEY, secret=NEXMO_SECRET) client.send_message({ “from”: “Python”, “to”: MY_NUMBER, “text”: “Hello world” })

Slide 8

Slide 8

import nexmo NEXMO_KEY = “a925db1ar392” NEXMO_SECRET = “01nd637fn29oe31mc721” MY_NUMBER = “447700900981” client = nexmo.Client(key=NEXMO_KEY, secret=NEXMO_SECRET) client.send_message({ “from”: “Python”, “to”: MY_NUMBER, “text”: “Hello world” })

Slide 9

Slide 9

git add . git commit -m “wip” git push

Slide 10

Slide 10

“the median time to discovery was 20 seconds, with times ranging from half a second to over 4 minutes” – How Bad Can It Git? Characterizing Secret Leakage in Public GitHub Repositories

Slide 11

Slide 11

Not safe, BUT VERY easy

Slide 12

Slide 12

Functionality Security Usability

Slide 13

Slide 13

Low Friction

Slide 14

Slide 14

Easy to implement

Slide 15

Slide 15

12 Factor Apps

Slide 16

Slide 16

I. Codebase II. Dependencies III. Config IV. Backing services V. Build, release, run VI. Processes VII. Port binding VIII. Concurrency IX. Disposability X. Dev/prod parity XI. Logs XII. Admin processes

Slide 17

Slide 17

III. Config

Slide 18

Slide 18

“A litmus test for whether an app has all config correctly factored out of the code is whether the codebase could be made open source at any moment, without compromising any credentials.” –The twelve-factor app

Slide 19

Slide 19

environment variables

Slide 20

Slide 20

import os import nexmo client = nexmo.Client( key=os.environ[“NEXMO_KEY”], secret=os.environ[“NEXMO_SECRET”] ) client.send_message({ “from”: “Python”, “to”: os.environ[“MY_NUMBER”], “text”: “Hello world” })

Slide 21

Slide 21

import os import nexmo client = nexmo.Client( key=os.environ[“NEXMO_KEY”], secret=os.environ[“NEXMO_SECRET”] ) client.send_message({ “from”: “Python”, “to”: os.environ[“MY_NUMBER”], “text”: “Hello world” })

Slide 22

Slide 22

import os import nexmo client = nexmo.Client( key=os.environ.get(“NEXMO_KEY”), secret=os.environ.get(“NEXMO_SECRET”) ) client.send_message({ “from”: “Python”, “to”: os.environ.get(“MY_NUMBER”), “text”: “Hello world” })

Slide 23

Slide 23

import os import nexmo client = nexmo.Client( key=os.environ.get(“NEXMO_KEY”), secret=os.environ.get(“NEXMO_SECRET”) ) client.send_message({ “from”: “Python”, “to”: os.environ.get(“MY_NUMBER”), “text”: “Hello world” })

Slide 24

Slide 24

import os import nexmo client = nexmo.Client( key=os.getenv(“NEXMO_KEY”), secret=os.getenv(“NEXMO_SECRET”), ) client.send_message( { “from”: “Python”, “to”: os.getenv(“MY_NUMBER”), “text”: “Hello world”, } )

Slide 25

Slide 25

import os import nexmo client = nexmo.Client( key=os.getenv(“NEXMO_KEY”), secret=os.getenv(“NEXMO_SECRET”), ) client.send_message( { “from”: “Python”, “to”: os.getenv(“MY_NUMBER”), “text”: “Hello world”, } )

Slide 26

Slide 26

Creating environment variables

Slide 27

Slide 27

This file must be used with “source bin/activate” from bash # you cannot run it directly deactivate () { $$…

}

Unset variables unset NEXMO_KEY unset NEXMO_SECRET unset MY_NUMBER $$… export NEXMO_KEY=”a925db1ar392” export NEXMO_SECRET=”01nd637fn29oe31mc721” export MY_NUMBER=”447700900981”

Slide 28

Slide 28

This file must be used with “source bin/activate” from bash # you cannot run it directly deactivate () { $$…

}

Unset variables unset NEXMO_KEY unset NEXMO_SECRET unset MY_NUMBER $$… export NEXMO_KEY=”a925db1ar392” export NEXMO_SECRET=”01nd637fn29oe31mc721” export MY_NUMBER=”447700900981”

Slide 29

Slide 29

This file must be used with “source bin/activate” from bash # you cannot run it directly deactivate () { $$…

}

Unset variables unset NEXMO_KEY unset NEXMO_SECRET unset MY_NUMBER $$… export NEXMO_KEY=”a925db1ar392” export NEXMO_SECRET=”01nd637fn29oe31mc721” export MY_NUMBER=”447700900981”

Slide 30

Slide 30

“direnv is an extension for your shell. It augments existing shells with a new feature that can load and unload environment variables depending on the current directory.” – direnv.net

Slide 31

Slide 31

$ echo export NEXMO_KEY=a925db1ar392 > .envrc .envrc is not allowed $ direnv allow . direnv: reloading direnv: loading .envrc direnv export: +NEXMO_KEY $ cd $.. direnv: unloading

Slide 32

Slide 32

$ echo export NEXMO_KEY=a925db1ar392 > .envrc .envrc is not allowed $ direnv allow . direnv: reloading direnv: loading .envrc direnv export: +NEXMO_KEY $ cd $.. direnv: unloading

Slide 33

Slide 33

$ echo export NEXMO_KEY=a925db1ar392 > .envrc .envrc is not allowed $ direnv allow . direnv: reloading direnv: loading .envrc direnv export: +NEXMO_KEY $ cd $.. direnv: unloading

Slide 34

Slide 34

$ echo export NEXMO_KEY=a925db1ar392 > .envrc .envrc is not allowed $ direnv allow . direnv: reloading direnv: loading .envrc direnv export: +NEXMO_KEY $ cd $.. direnv: unloading

Slide 35

Slide 35

$ echo export NEXMO_KEY=a925db1ar392 > .envrc .envrc is not allowed $ direnv allow . direnv: reloading direnv: loading .envrc direnv export: +NEXMO_KEY $ cd $.. direnv: unloading

Slide 36

Slide 36

$ echo export NEXMO_KEY=a925db1ar392 > .envrc .envrc is not allowed $ direnv allow . direnv: reloading direnv: loading .envrc direnv export: +NEXMO_KEY $ cd $.. direnv: unloading

Slide 37

Slide 37

Ignore .envrc echo .envrc > ~/.gitignore git config $—global core.excludesfile ~/.gitignore

Slide 38

Slide 38

Example .envrc grep -ohr “^export .*=” .envrc > .envrc.example

Slide 39

Slide 39

Sharing values & Files

Slide 40

Slide 40

import os import nexmo client = nexmo.Client( application_id=os.getenv(“NEXMO_APPLICATION_ID”), private_key=os.getenv(“NEXMO_PRIVATE_KEY”, “./private.key”), ) response = client.create_call( { “to”: [{“type”: “phone”, “number”: os.getenv(“TO_NUMBER”)}], “from”: {“type”: “phone”, “number”: os.getenv(“NEXMO_NUMBER”)}, “answer_url”: [f”{os.getenv(‘VAPI_URL’)}/answer”], } )

Slide 41

Slide 41

import os import nexmo client = nexmo.Client( application_id=os.getenv(“NEXMO_APPLICATION_ID”), private_key=os.getenv(“NEXMO_PRIVATE_KEY”, “./private.key”), ) response = client.create_call( { “to”: [{“type”: “phone”, “number”: os.getenv(“TO_NUMBER”)}], “from”: {“type”: “phone”, “number”: os.getenv(“NEXMO_NUMBER”)}, “answer_url”: [f”{os.getenv(‘VAPI_URL’)}/answer”], } )

Slide 42

Slide 42

import os import nexmo client = nexmo.Client( application_id=os.getenv(“NEXMO_APPLICATION_ID”), private_key=os.getenv(“NEXMO_PRIVATE_KEY”, “./private.key”), ) response = client.create_call( { “to”: [{“type”: “phone”, “number”: os.getenv(“TO_NUMBER”)}], “from”: {“type”: “phone”, “number”: os.getenv(“NEXMO_NUMBER”)}, “answer_url”: [f”{os.getenv(‘VAPI_URL’)}/answer”], } )

Slide 43

Slide 43

git-secret.io

Slide 44

Slide 44

“git-secret encrypts files and stores them inside the git repository, so you will have all the changes for every commit.” –git-secret.io

Slide 45

Slide 45

Safe & Easy

Slide 46

Slide 46

Safe & Easy-ish

Slide 47

Slide 47

PGP

Slide 48

Slide 48

$ git secret init git-secret: init created: ‘/myproject/.gitsecret/’

Slide 49

Slide 49

$ git secret tell me@aaronbassett.com gpg: marginals needed: 3 completes needed: 1 trust model: pgp gpg: depth: 0 valid: 1 signed: 0 trust: 0-, 0q, 0n, 0m, 0f, 1u gpg: next trustdb check due at 2021-09-30 gpg: keybox ‘/myproject/.gitsecret/keys/pubring.kbx’ created gpg: /myproject/.gitsecret/keys/trustdb.gpg: trustdb created git-secret: done. me@aaronbassett.com added as user(s) who know the secret.

Slide 50

Slide 50

$ git secret add private.key git-secret: these files are not in .gitignore: private.key git-secret: auto adding them to .gitignore git-secret: 1 item(s) added.

Slide 51

Slide 51

$ cat .gitignore .gitsecret/keys/random_seed !*.secret private.key

Slide 52

Slide 52

$ ls .git .gitignore .gitsecret private.key

Slide 53

Slide 53

$ git secret hide git-secret: done. 1 of 1 files are hidden.

Slide 54

Slide 54

$ ls .git .gitignore .gitsecret private.key private.key.secret

Slide 55

Slide 55

$ git secret reveal File ‘/myproject/private.key’ exists. Overwrite? (y/N) y git-secret: done. 1 of 1 files are revealed.

Slide 56

Slide 56

$ git secret whoknows me@aaronbassett.com $ git secret list private.key

Slide 57

Slide 57

$ git secret killperson me@aaronbassett.com git-secret: removed keys. git-secret: now [me@aaronbassett.com] do not have an access to the repository. git-secret: make sure to hide the existing secrets again. $ git secret reveal git-secret: abort: no public keys for users found. run ‘git secret tell email@address.

Slide 58

Slide 58

Git Secrets

Slide 59

Slide 59

“git-secrets scans commits, commit messages, and —no-ff merges to prevent adding secrets into your git repositories. If a commit, commit message, or any commit in a —no-ff merge history matches one of your configured prohibited regular expression patterns, then the commit is rejected.” – awslabs/git-secrets

Slide 60

Slide 60

$ git secrets $—register-aws $—global OK $ git secrets $—install ~/.git-templates/git-secrets ✓ Installed commit-msg hook to /Users/aaronbassett/.git-templates/gitsecrets/hooks/commit-msg ✓ Installed pre-commit hook to /Users/aaronbassett/.git-templates/gitsecrets/hooks/pre-commit ✓ Installed prepare-commit-msg hook to /Users/aaronbassett/.git-templates/ git-secrets/hooks/prepare-commit-msg $ git config $—global init.templateDir ~/.git-templates/git-secrets

Slide 61

Slide 61

$ git secrets $—register-aws $—global OK $ git secrets $—install ~/.git-templates/git-secrets ✓ Installed commit-msg hook to /Users/aaronbassett/.git-templates/gitsecrets/hooks/commit-msg ✓ Installed pre-commit hook to /Users/aaronbassett/.git-templates/gitsecrets/hooks/pre-commit ✓ Installed prepare-commit-msg hook to /Users/aaronbassett/.git-templates/ git-secrets/hooks/prepare-commit-msg $ git config $—global init.templateDir ~/.git-templates/git-secrets

Slide 62

Slide 62

$ git secrets $—register-aws $—global OK $ git secrets $—install ~/.git-templates/git-secrets ✓ Installed commit-msg hook to /Users/aaronbassett/.git-templates/gitsecrets/hooks/commit-msg ✓ Installed pre-commit hook to /Users/aaronbassett/.git-templates/gitsecrets/hooks/pre-commit ✓ Installed prepare-commit-msg hook to /Users/aaronbassett/.git-templates/ git-secrets/hooks/prepare-commit-msg $ git config $—global init.templateDir ~/.git-templates/git-secrets

Slide 63

Slide 63

$ git secrets $—register-aws $—global OK $ git secrets $—install ~/.git-templates/git-secrets ✓ Installed commit-msg hook to /Users/aaronbassett/.git-templates/gitsecrets/hooks/commit-msg ✓ Installed pre-commit hook to /Users/aaronbassett/.git-templates/gitsecrets/hooks/pre-commit ✓ Installed prepare-commit-msg hook to /Users/aaronbassett/.git-templates/ git-secrets/hooks/prepare-commit-msg $ git config $—global init.templateDir ~/.git-templates/git-secrets

Slide 64

Slide 64

providers

Slide 65

Slide 65

^[5KL][1-9A-HJ-NP-Za-km-z]{50,51}$ (xox[p|b|o|a]-[0-9]{12}-[0-9]{12}-[0-9]{12}-[a-z0-9]{32}) https:$//hooks.slack.com/services/T[a-zA-Z0-9_]{8}/B[a-zA-Z0-9_]{8}/[a-zA-Z0-9_]{24} EAACEdEose0cBA[0-9A-Za-z]+ [t|T][w|W][i|I][t|T][t|T][e|E][r|R].[1-9][0-9]+-[0-9a-zA-Z]{40} [t|T][w|W][i|I][t|T][t|T][e|E][r|R].[‘|”][0-9a-zA-Z]{35,44}[‘|”] AIza[0-9A-Za-z*-]{35} [0-9]+-[0-9A-Za-z]{32}*.apps*.googleusercontent*.com ya29*.[0-9A-Za-z*-_]+ [h|H][e|E][r|R][o|O][k|K][u|U].*[0-9A-F]{8}-[0-9A-F]{4}-[0-9A-F]{4}-[0-9A-F]{4}-[0-9A-F]{12} (——-(BEGIN|END) PRIVATE KEY——-) (——-(BEGIN|END) RSA PRIVATE KEY——-)

Slide 66

Slide 66

$ git secrets $—add-provider $— cat /secret/file/patterns

Slide 67

Slide 67

(——-(BEGIN|END) RSA PRIVATE KEY——-)

Slide 68

Slide 68

ya29*.[0-9A-Za-z*-_]+

Slide 69

Slide 69

EAACEdEose0cBA[0-9A-Za-z]+

Slide 70

Slide 70

$ cd /secret/file/patterns $ ls crypto keys vendors

Slide 71

Slide 71

Slack (xox[p|b|o|a]-[0-9]{12}-[0-9]{12}-[0-9]{12}-[a-z0-9]{32}) https:$//hooks.slack.com/services/T[a-zA-Z0-9_]{8}/B[a-zA-Z0-9_]{8}/[a-zA-Z0-9_]{24} # Facebook EAACEdEose0cBA[0-9A-Za-z]+ # Twitter [t|T][w|W][i|I][t|T][t|T][e|E][r|R].[1-9][0-9]+-[0-9a-zA-Z]{40} [t|T][w|W][i|I][t|T][t|T][e|E][r|R].[‘|”][0-9a-zA-Z]{35,44}[‘|”] # Google AIza[0-9A-Za-z*-]{35} [0-9]+-[0-9A-Za-z]{32}*.apps*.googleusercontent*.com ya29*.[0-9A-Za-z*-_]+ # Heroku [h|H][e|E][r|R][o|O][k|K][u|U].*[0-9A-F]{8}-[0-9A-F]{4}-[0-9A-F]{4}-[0-9A-F]{4}-[0-9A-F]{12}

Slide 72

Slide 72

git secrets $—add-provider $— egrep -rhv “(^#|^$)” /secret/file/patterns

Slide 73

Slide 73

^[5KL][1-9A-HJ-NP-Za-km-z]{50,51}$ (xox[p|b|o|a]-[0-9]{12}-[0-9]{12}-[0-9]{12}-[a-z0-9]{32}) https:$//hooks.slack.com/services/T[a-zA-Z0-9_]{8}/B[a-zA-Z0-9_]{8}/[a-zA-Z0-9_]{24} EAACEdEose0cBA[0-9A-Za-z]+ [t|T][w|W][i|I][t|T][t|T][e|E][r|R].[1-9][0-9]+-[0-9a-zA-Z]{40} [t|T][w|W][i|I][t|T][t|T][e|E][r|R].[‘|”][0-9a-zA-Z]{35,44}[‘|”] AIza[0-9A-Za-z*-]{35} [0-9]+-[0-9A-Za-z]{32}*.apps*.googleusercontent*.com ya29*.[0-9A-Za-z*-_]+ [h|H][e|E][r|R][o|O][k|K][u|U].*[0-9A-F]{8}-[0-9A-F]{4}-[0-9A-F]{4}-[0-9A-F]{4}-[0-9A-F]{12} (——-(BEGIN|END) PRIVATE KEY——-) (——-(BEGIN|END) RSA PRIVATE KEY——-)

Slide 74

Slide 74

$ git add ‘private.key’ $ git commit -m “Adding some files I shouldn’t” private.key:1:——-BEGIN PRIVATE KEY——[ERROR] Matched one or more prohibited patterns Possible mitigations: - Mark false positives as allowed using: git config $—add secrets.allowed $$… - Mark false positives as allowed by adding regular expressions to .gitallowed at repository’s root directory - List your configured patterns: git config $—get-all secrets.patterns - List your configured allowed patterns: git config $—get-all secrets.allowed - List your configured allowed patterns in .gitallowed at repository’s root directory - Use $—no-verify if this is a one-time false positive

Slide 75

Slide 75

GITLEAKS

Slide 76

Slide 76

“Audit git repos for secrets. Gitleaks provides a way for you to find unencrypted secrets and other unwanted data types in git source code repositories” – zricethezav/gitleaks

Slide 77

Slide 77

I. A GIT Repo II. Github user III. Github organization IV. Github PR V. GitLab user VI. GitLab group

Slide 78

Slide 78

{ } “line”: “——-BEGIN PRIVATE KEY——-“, “commit”: “37f19780583f12fd2fb687f2d7d3840880e79c76”, “offender”: “——-BEGIN PRIVATE KEY——-“, “rule”: “PKCS8”, “info”: “——-BEGIN PRIVATE KEY——- regex match”, “commitMsg”: “wip backup “, “author”: “Joe Bloggs”, “email”: “jbloggs@example.com”, “file”: “app/private.key”, “repo”: “my-awesome-app”, “date”: “2019-09-14T12:26:10+08:00”, “tags”: “key, PKCS8”, “severity”: “”

Slide 79

Slide 79

gitleaks —github-org=MyOrg > /dev/null if [ $? -eq 0 ]; then echo “No Leaks” else nexmo sms 447700900235 Git is leaking $—confirm fi

Slide 80

Slide 80

gitleaks —github-org=MyOrg > /dev/null if [ $? -eq 0 ]; then echo “No Leaks” else nexmo sms 447700900235 Git is leaking $—confirm fi

Slide 81

Slide 81

gitleaks —github-org=MyOrg > /dev/null if [ $? -eq 0 ]; then echo “No Leaks” else nexmo sms 447700900235 Git is leaking $—confirm fi

Slide 82

Slide 82

gitleaks —github-org=MyOrg > /dev/null if [ $? -eq 0 ]; then echo “No Leaks” else nexmo sms 447700900235 Git is leaking $—confirm fi

Slide 83

Slide 83

  1. Keep secrets and code separate

Slide 84

Slide 84

  1. If you need to share secrets encrypt them first # PGP is a PITA, use tools to make it easier #

Slide 85

Slide 85

  1. Automate, Automate, Automate

Slide 86

Slide 86

  1. Late is better than never

Slide 87

Slide 87

@aaronbassett noti.st/aaronbassett