Securing Distributed Software

A presentation at PHPSW Security in September 2016 in Bristol, UK by Drew McLellan

Slide 1

SECURING DISTRIBUTED SOFTWARE DREW MCLELLAN, PHPSW SEPTEMBER 2016

Slide 2

DEALING WITH BUGS ON A PROJECT WITH RESIDENT DEVS IS EASY

Slide 3

FOR AN UPDATE TO A DEPENDANCY > Read about an update > Download and test the patch > Push it to production

Slide 4

FOR YOUR OWN CODE > Identify the problem > Fix the problem > Test the patch > Push it to production

Slide 5

DEALING WITH BUGS AS A LIBRARY MAINTAINER IS EASY

Slide 6

BUGS IN LIBRARIES > Identify the problem > Fix the problem > Test the patch > Publish a new release > Announce that users should update

Slide 7

BUT WHAT HAPPENS FOR THINGS THAT FALL IN BETWEEN?

Slide 8

PERCH IS A SELF-HOSTED CONTENT MANAGEMENT SYSTEM

Slide 9

PERCH > PHP + MySQL > (GD, Imagick etc) > The WordPress Stack

Slide 10

AN ANTIDOTE TO WORDPRESS

Slide 11

PERCH > No reliance on dozens of third party plugins > Stable and reliable, set and forget > Update when you work on the site, not in between > Excellent security track record

Slide 12

PROJECT-WORK FRIENDLY

Slide 13

EXCELLENT SECURITY TRACK RECORD

Slide 14

FROM MAY 2009 TO NOVEMBER 2015: NO EXPLOITABLE SECURITY ISSUES

Slide 15

WE GOT AN EMAIL… From a user saying they could hack our online demo server and download a list of all our customers and would we pay them for this information?

Slide 16

WE KNEW TWO THINGS

Slide 17

They couldn’t access our customer data, at most they could get the names and email addresses of 100 or so current demo users (it’s all sandboxed) 2. We don’t negotiate with terrorists

Slide 18

THEY FAILED TO REALISE WE COULD JUST LOOK AT OUR LOGS

Slide 19

FROM THE LOGS > I could see they’d uploaded a PHP webshell > I could see how they’d uploaded it, which lead me directly to the bug > I could also see that even though they’d uploaded it, our server wouldn’t run it

Slide 20

WEBSHELL ATTACKS

Slide 21

WEBSHELL I wasn’t familiar with these Single page PHP script that provides a web console for executing shell commands and causing havoc

Slide 22

TWO THINGS NEED TO HAPPEN FOR THE ATTACK TO WORK 1. The attacker needs to be able to get the script onto the server 2. The script needs to be able to execute via a standard HTTP request

Slide 23

IT WAS POSSIBLE TO GET THE FILE ONTO THE SERVER > Our demo server is well configured, so it failed at the second part > the script couldn’t run

Slide 24

PERCH HAS PLUGINS FOR USING RICH EDITORS ON TEXTAREAS

Slide 25

F'KING WYSIWYG > These can be plain text, Markdown, Textile, or HTML > As long as the editor enhances an HTML textarea input, it’ll work

Slide 26

YOU NEED TO WRITE TWO THINGS 1. A bit of JavaScript to initialise the editor 2. If the editor has a file upload button, a quick PHP script to handle the file upload via our API

Slide 27

UH OH. FILE UPLOADS.

Slide 28

FILE UPLOADS The default editor we include does Markdown and is called MarkItUp. It has a very quick, simple image and file upload script which grabs the file uses the Perch API to add it as content

Slide 29

THE PERCH API HAS TWO POSSIBLE ENTRY POINTS

Slide 30

ONE DESIGNED FOR ON-PAGE USE AS WEB PAGES ARE RENDERED > light, fast, does as little work as possible > ‘runtime’

Slide 31

ONE DESIGNED FOR CONTROL PANEL OPERATIONS > more complex, richer, does far more to assist but uses more resources > ‘admin’ > checks authentication > checks authentication > checks authentication

Slide 32

CHECKS AUTHENTICATION

Slide 33

THE MARKITUP UPLOAD SCRIPT WAS USING THE RUNTIME ENTRY POINT

Slide 34

should have been using the admin entry point > files could be uploaded without first checking that the user was authenticated

Slide 35

ANYTHING COULD POST A FILE TO THE SCRIPT AND IT WOULD UPLOAD IT… … AND HELPFULLY GET A RESPONSE WITH THE URL TO THE UPLOADED FILE. JEEZ.

Slide 36

BUT WE’RE NOT COMPLETELY DAFT

Slide 37

All uploads are processed through a central clearing house that does basics like checking mime types and file extensions You can’t upload a file to Perch if it has a file extension with ‘php’ in it (.php, .php3, .php5). They get saved as .php.txt

Slide 38

THIS HELPS TO ADDRESS THE SECOND PART OF THE ISSUE THEY CAN UPLOAD THE FILE, BUT IT WON’T EXECUTE BECAUSE WE’VE DEFUSED THE FILE EXTENSION

Slide 39

LOOKING AT MY LOG FILES I COULD SEE THIS IN ACTION webshell.php > webshell.php.txt webshell.php5 > webshell.php5.txt

Slide 40

THE DAMAGE WAS:

Anyone can upload files to your website (bad!) 2. But they could upload and execute scripts (phew!)

Slide 41

LOOKING BACK THROUGH SOURCE CONTROL, THIS PROBLEM HAD EXISTED FOR 6 YEARS

Slide 42

Apart from this one guy, no one seemed to have found it > I’d missed it in dozens of code reviews

Slide 43

SO... > We figured the best course of action was to fix it swiftly and move on without drawing specific attention to it. > Entry in the change log said: Fixes bugs in MarkItUp editor

Slide 44

TIME PASSES

Slide 45

A FEW WEEKS LATER WE GOT AN EMAIL FROM SOMEONE CLAIMING THEIR PERCH SITE HAD BEEN HACKED

Slide 46

We get these from time to time - it’s always something else on the server, or the server itself

Slide 47

Except then we got another with the same symptoms

Slide 48

and another, but with more information. A webshell had been uploaded to the Perch resources folder with a .phtml extension.

Slide 49

.phtml

Slide 50

THIS WAS A NEW ONE ON ME I HAD NO IDEA IT WAS USED FOR PHP

Slide 51

.phtml WAS THE FILE EXTENSION FOR PHP 2

Slide 52

.phtml FILES COULD BE UPLOADED WITHOUT BEING DEFUSED which was no big deal because we don’t have our web servers configured to run random file extensions we’ve never heard of. So no problem. … but wait.

Slide 53

SOME OF OUR CUSTOMERS USE REALLY TERRIBLE HOSTING

Slide 54

WHICH IS BADLY CONFIGURED AND MIGHT BE EXPECTING PHP 2 FILES TO BE UPLOADED CRAP

Slide 55

THIS PUT US IN AN ODD SITUATION

Slide 56

a bug in the software was being exploited > but we’d already fixed it weeks ago > but customers weren’t updating their software so they were still vulnerable

Slide 57

THE VENN DIAGRAM OF CUSTOMERS ON BAD HOSTING AND CUSTOMERS UNLIKELY TO APPLY SOFTWARE UPDATES IS ALMOST A PERFECT CIRCLE

Slide 58

EVEN IF WE MADE A LOT OF NOISE ABOUT THIS, WE POSSIBLY WOULDN’T EVEN REACH THE PEOPLE AFFECTED, AND THEY MIGHT NOT BE IN A POSITION TO DO ANYTHING

Slide 59

WE HAD CUSTOMERS ON VERY OLD VERSIONS OF THE SOFTWARE SOMETIMES UP TO 4 OR 5 YEARS OLD. UPDATING TO THE CURRENT VERSION WOULD BE A CHARGEABLE PROJECT THAT WOULD REQUIRE PLANNING AND SCHEDULING.

Slide 60

SO WHAT DO YOU DO?

Slide 61

YOU MAKE 79 INDIVIDUAL PATCH FILES

Slide 62

We produced a drop-in replacement file for 79 previous releases of Perch Updating a site was a case of finding your version, downloading the patch, uploading it to your site Guaranteed to change nothing else but patching the bug

Slide 63

This was important as it needed to be safe to do to a live site If anything needed testing, it would need time scheduling and it wouldn’t get done

Slide 64

We emailed customers and stressed that there was an important security update that needed applying straight away

Slide 65

DOWNSIDES > Generating 79 patches is laborious > We have no way to easily see if a site is patched > That one file doesn’t affect the part of the app that reports its version number > (We could tackle this various ways if it became and issue)

Slide 66

UPSIDES > A easy fix gave customers confidence that we were on top of the problem > A quick and safe-to-apply update meant that customers actually updated > …Or enough updated that it wasn’t worthwhile attackers to continue to waste energy on

Slide 67

WHAT WE LEARNED

Slide 68

TAKE MINOR SECURITY BUGS SERIOUSLY AS THEY CAN OPEN THE DOOR TO MAJOR ONES

Slide 69

MAKE IT REALLY EASY IF YOU NEED PEOPLE TO UPDATE, ELSE THEY’LL PUT IT OFF

Slide 70

FINDING SERIOUS BUGS IN YOUR OWN CODE MAKES YOU FEEL REALLY BAD BUT THAT’S OK, YOU SHOULD FEEL BAD.

Slide 71

THANKS! JOIND.IN/TALK/8062B @DREWM