Jonathan Relf Devel-Ops - Bringing Development Practices to Operations Tasks

27th June 2019 Jonathan Relf DEVEL-OPS

Commify is the team behind a global portfolio of business messaging brands. We work with more than 45,000 companies, helping them transform their mobile communications with their customers and staff.

We provide SMS, voice, web, IP/OTT, email and intelligent multichannel messaging services both on a self-serve basis (through an online platform or API), and as tailored solutions to more complex needs. www.commify.com

BRINGING DEVELOPMENT PRACTICES TO OPERATIONS TASKS DEVEL-OPS

THIS TALK IS NOT: VS Devs Ops

THIS TALK IS MORE:

@jbjon

“THE DEFINITION OF INSANITY IS REPEATING THE SAME MISTAKES OVER AND OVER AGAIN AND EXPECTING DIFFERENT RESULTS” Unknown @jbjon

“THE DEFINITION OF INSANITY IN I.T. OPERATIONS IS MANUALLY REPEATING A TASK OVER AND OVER AGAIN AND EXPECTING THE SAME RESULTS” Jonathan Relf @jbjon

THE IRON AGE OF I.T.

THE CLOUD AGE OF I.T.

“UNRELIABLE SOFTWARE DEPENDING ON RELIABLE HARDWARE TO RELIABLE SOFTWARE RUNNING ON UNRELIABLE HARDWARE” Infrastructure As Code @jbjon

2017

2018

https://landscape.cncf.io/

THOUGHTWORKS RADAR @jbjon https://www.thoughtworks.com/radar/tools

https://blogs.technet.microsoft.com/xdot509/2013/07/24/getting-started-with-windows-azure-part-2-what-are-cloud-services/

WHY BOTHER DEFINING YOUR OWN INFRASTRUCTURE IN CODE? @jbjon

REGULATORY REQUIREMENTS

3RD PARTY SOLUTION UNCERTAINTY

HARDEN APPLICATION HOSTING

AVOID VENDOR LOCK-IN

MAINTAINING SERVERS OVER TIME OWN SOFTWARE CONFIGURATION UPDATES 3RD PARTY SOFTWARE CONFIGURATION UPDATES OPERATING SYSTEM FEATURES CONFIGURATION HYPERVISOR CONFIGURATION UPDATES PHYSICAL SERVER CONFIGURATION UPDATES UPDATES

THE FOUR STAGES OF INFRASTRUCTURE PROVISIONING CONFIGURING MAINTENANCE TERMINATION @jbjon

PROVISIONING ▸ Since virtualisation, one of the easiest of the phases ▸ Traditionally based off ISO images ▸ Out of date, unpatched images @jbjon

CONFIGURATION ▸ Can be where most configuration drift is added if not done through automation ▸ It may involve ‘tinkering’ to get a server working ▸ Can end up with ‘Snowflake servers’ ▸ Configuration Management software like Puppet & Chef @jbjon

MAINTENANCE ▸ Updates or upgrades of software components ▸ From security patches to in-place upgrades of O.S. ▸ Ordering of patches may affect outcome ▸ Scripting can help ensure consistency @jbjon

TERMINATION ▸ Fear of shutting off servers ▸ Treating servers like “pets, not cattle” ▸ Anti-pattern: Celebrating up-time ▸ Plan for the fact an instance could disappear @jbjon

GOALS OF INFRASTRUCTURE AS CODE @jbjon

IT INFRASTRUCTURE SUPPORTS & ENABLES CHANGE Kief Morris @jbjon

CHANGES TO THE SYSTEM ARE ROUTINE WITHOUT DRAMA OR STRESS Kief Morris @jbjon

IT STAFF SPEND THEIR TIME ON VALUABLE THINGS… NOT REPETITIVE TASKS Kief Morris @jbjon

USERS ARE ABLE TO DEFINE, PROVISION, AND MANAGE THE RESOURCES THEY NEED WITHOUT I.T. STAFF TO DO IT FOR THEM Kief Morris @jbjon

DEFINITION TOOLS ▸ Good Infrastructure As Code tools ▸ have scriptable interfaces ▸ can be run unattended ▸ can be tailored through config ▸ allow tasks to be defined in code ▸ the definition files become ‘living documentation’ @jbjon

VERSION CONTROL ▸ Natural part of development workflow ▸ branching, rollbacks, ownership ▸ Single point of truth ▸ Living documentation @jbjon

CONTINUOUS INTEGRATION ▸ Early feedback about potentially breaking changes ▸ Changes tested in isolation from production ▸ Can apply to infrastructure, server, and configuration changes ▸ “Does this produce the instance I expect?” ▸ “Does this instance have all the features I expect?” ▸ “Is this instance configured for its role correctly?” @jbjon

BUILD PIPELINES ▸ Avoid ‘automation fear’ ▸ Build regularly as well as on changes ▸ Pipelines maintaining templates ensures up-to-date images ▸ Reduces manual knowledge fading ▸ Services used to build can be provisioned temporarily ▸ Packer supports building machine images @jbjon

AUTOMATED TESTING ▸ One of the best practices to borrow from Development ▸ Not relying on ‘Green builds’ ▸ Frameworks like ServerSpec (http://serverspec.org) @jbjon

CONTINUOUS DELIVERY ▸ Not ‘Continuous Deployment’ ▸ Being able to update Test / Lab environments regularly ▸ Risk increases with ‘Time Since Last Success’ @jbjon

CODE ANALYSIS ▸ Emerging area drawing from Dev practices ▸ Config management declarative languages have ‘lint’ tools @jbjon

“A NETFLIX TEAM KNEW THAT A PERCENTAGE OF AWS INSTANCES, WHEN PROVISIONED, WILL PERFORM MUCH WORSE THAN THE AVERAGE INSTANCE SO THEY WROTE THEIR PROVISIONING SCRIPTS TO IMMEDIATELY TEST THE PERFORMANCE OF EACH NEW INSTANCE. IF IT DOESN’T MEET THEIR STANDARDS, THE SCRIPT DESTROYS THE INSTANCE AND TRIES AGAIN WITH A NEW INSTANCE.” Kief Morris @jbjon

WAYS TO START INTRODUCING THIS ▸ Start small ▸ Script everything ▸ Automate the process ▸ Run it regularly ▸ Test the changes in a safe environment ▸ Monitor all the things @jbjon

QUESTIONS?

ABOUT ME ▸ Jonathan Relf ▸ Solutions Architect @ Commify ▸ about.me/jbjon @jbjon