How DevOps changed everything Karthik Gaekwad @iteration1 Wednesday, August 7, 13 #AgileDevOps

Howdy! • I’m Karthik Gaekwad • Senior Web Engineer • Mentor Graphics Embedded • From Austin, TX • We have great BBQ, chill people and brutal summers! @iteration1 Wednesday, August 7, 13 Agile 2013 #AgileDevOps

My Background • National Instruments (NI) • Ecommerce Team (Agile) • R&D Cloud Team (DevOps) • Mentor Graphics Embedded • Cloud Team (DevOps) • Organizer: CloudAustin, DevOpsDays Austin, TheAgileAdmin @iteration1 Wednesday, August 7, 13 Agile 2013 #AgileDevOps

Audience Survey @iteration1 Wednesday, August 7, 13 Agile 2013 #AgileDevOps

Let’s not waste time! The most important DevOps things I’ve learned over the last 4 years @iteration1 Wednesday, August 7, 13 Agile 2013 #AgileDevOps

First, we have to talk about agile… @iteration1 Wednesday, August 7, 13 Agile 2013 #AgileDevOps

@iteration1 Wednesday, August 7, 13 Agile 2013 #AgileDevOps

There was a waterfall Requirements Design Implement, Test Deploy Maintain @iteration1 Wednesday, August 7, 13 Agile 2013 #AgileDevOps

Roles Requirements Product Manager, Business Analyst Design Business Analyst, Web Designer, Architect Implement Programmer Analyst, Web Developer Deploy Programmer Analyst, System Administrator Maintain System Administrator @iteration1 Wednesday, August 7, 13 Agile 2013 #AgileDevOps

Waterfall.. S @iteration1 Wednesday, August 7, 13 Agile 2013 #AgileDevOps

Communication “But I told you this already” @iteration1 Wednesday, August 7, 13 Agile 2013 #AgileDevOps

Accountability “That is not my job!” @iteration1 Wednesday, August 7, 13 Agile 2013 #AgileDevOps

Time to deliver “That is not a priority” @iteration1 Wednesday, August 7, 13 Agile 2013 #AgileDevOps

@iteration1 Wednesday, August 7, 13 Agile 2013 #AgileDevOps

**About the time I started… @iteration1 Wednesday, August 7, 13 Agile 2013 #AgileDevOps

New Product: Partslist Collaborate and tag parts to build large systems @iteration1 Wednesday, August 7, 13 Agile 2013 #AgileDevOps

Transitioned to SCRUM 2 Week Iterations Targeted Feature sets Demo at end of iteration @iteration1 Wednesday, August 7, 13 Agile 2013 #AgileDevOps

The Product Team > Traditional Organization @iteration1 Wednesday, August 7, 13 Agile 2013 #AgileDevOps

Results Working together, building things that were wanted @iteration1 Wednesday, August 7, 13 Agile 2013 #AgileDevOps

Results Between Business and IT @iteration1 Wednesday, August 7, 13 Agile 2013 #AgileDevOps

Results Better products being delivered, at a faster pace. Methodology expanded to all other web teams… @iteration1 Wednesday, August 7, 13 Agile 2013 #AgileDevOps

Results I love my job! #ForReal @iteration1 Wednesday, August 7, 13 Agile 2013 #AgileDevOps

Agile: Retrospective • Scrum brought team together. • Building products on time, which were actually needed. • Superior collaboration between business and IT. • Organization shift -> Scrum @iteration1 Wednesday, August 7, 13 Agile 2013 #AgileDevOps

Ops conversations “Agile is cool! but it’s hard for ops to be agile because it takes months to buy and provision servers.” @iteration1 Wednesday, August 7, 13 Agile 2013 #AgileDevOps

@iteration1 Wednesday, August 7, 13 Agile 2013 #AgileDevOps

Ops conversations Deployer Dude “All day everyday” @iteration1 Wednesday, August 7, 13 Agile 2013 #AgileDevOps

But in the community… @iteration1 Wednesday, August 7, 13 Agile 2013 #AgileDevOps

But in the community… @iteration1 Wednesday, August 7, 13 Agile 2013 #AgileDevOps

“Go build us cloud products “We’re making cloud now” products now” Sincerely, Management @iteration1 Wednesday, August 7, 13 Agile 2013 #AgileDevOps

“Cloud is “We’re making cloud AWESOME!” products now” @iteration1 Wednesday, August 7, 13 Agile 2013 #AgileDevOps

“Cloud is “We’reOSSM” making cloud -Dave Nielson products now” -NIST @iteration1 Wednesday, August 7, 13 Agile 2013 #AgileDevOps

“Cloud is AWESOME OSSM” On-demand “We’re making cloud Self-service products now” Scalable Measurable @iteration1 Wednesday, August 7, 13 Agile 2013 #AgileDevOps

“Give me API or give me death” @littleidea @iteration1 Wednesday, August 7, 13 Agile 2013 #AgileDevOps

“Agile DevOps “We’re making cloud Cloud Team” products now” @iteration1 Wednesday, August 7, 13 Agile 2013 #AgileDevOps

DevOps “A cultural and professional movement” Adam Jacobs @iteration1 Wednesday, August 7, 13 Agile 2013 #AgileDevOps

DevOps “System administrators participating in an agile development process alongside developers and using many of the same agile techniques for their systems work.” The Agile Admin Blog @iteration1 Wednesday, August 7, 13 Agile 2013 #AgileDevOps

“Get rid of the devs; get rid of the ops; let’s just find people who can solve problems” Alison Chaiken Embedded Linux, Thought Leader @iteration1 Wednesday, August 7, 13 Agile 2013 #AgileDevOps

Pillars of DevOps CULTURE AUTOMATION MEASUREMENT By: John Willis SHARING Damon Edwards @iteration1 Wednesday, August 7, 13 Agile 2013 #AgileDevOps

DevOps 101 Ideas and Stories Free!! No charge!! @iteration1 Wednesday, August 7, 13 Agile 2013 #AgileDevOps

“Agile DevOps “We’re making cloud Cloud Team” products now” Buzzword friendly! @iteration1 Wednesday, August 7, 13 Agile 2013 #AgileDevOps

We had a goal: “Build a cloud authentication prototype” Wednesday, August 7, 13

R E U LT U C Our 1st sprint(s) sucked! @iteration1 Wednesday, August 7, 13 Agile 2013 #AgileDevOps

R E U LT U C Devs: “REST Services!” But we suffered from “Wall of confusionitis” Ops: “System Stability!” @iteration1 Wednesday, August 7, 13 Agile 2013 #AgileDevOps

R E U LT U C Not seeing eye to eye #Frustrating #Angry @iteration1 Wednesday, August 7, 13 Agile 2013 #AgileDevOps

R E U LT U C @iteration1 Wednesday, August 7, 13 Agile 2013 #AgileDevOps

R E U LT U C Really, this is “Platform Architecture”, but I’m too lazy to redo the image @iteration1 Wednesday, August 7, 13 Agile 2013 #AgileDevOps

R E U LT U C Devs and ops understand good architecture. #GoodPlaceToStart @iteration1 Wednesday, August 7, 13 Agile 2013 #AgileDevOps

R E U LT U C Think platform first, then app later. #PaaS #REST @iteration1 Wednesday, August 7, 13 Agile 2013 #AgileDevOps

IO N A U T O M A T 1st win: Architecture • Everyone understood architecture • Helped everyone speak same language • Reduced friction • Called it “The System Model” @iteration1 Wednesday, August 7, 13 Agile 2013 #AgileDevOps

IO N M A T T O A U 1st win: Architecture • “Model Driven Automation” • Championed by ops • Everyone needs to understand the model • Figure out how to deploy the model @iteration1 Wednesday, August 7, 13 Agile 2013 #AgileDevOps

IO N M A T T O A U • 1st win: Architecture Programmable Infrastructure Environment (PIE) • Based on version controlled architecture diagram • • • • Provisioned cloud boxes Ran deploys Set standard naming for boxes On demand deploy for apps @iteration1 Wednesday, August 7, 13 Agile 2013 #AgileDevOps

IO N M A T T O A U Outcomes • Ops learns Java! • Ops learns version control! • Developers get comfortable with production environments! • …Now we’re speaking many common languages! @iteration1 Wednesday, August 7, 13 Agile 2013 #AgileDevOps

A U T O M A T IO N Are y’all crazy??? No Chef? No Puppet? @iteration1 Wednesday, August 7, 13 Agile 2013 #AgileDevOps

A U T O M A T IO N Everybody be cool • We believe the cleanest abstraction layer is starting with the entire system you’re trying to model • In 2009, chef and puppet were younger • We were scared of Ruby • We needed windows cloud support • Plus dawg, cloud was a wild wild world then! @iteration1 Wednesday, August 7, 13 Agile 2013 #AgileDevOps

IO N M A T T O A U Please automate!! @iteration1 Wednesday, August 7, 13 Agile 2013 #AgileDevOps

IO N M A T T O A U Do I pick Chef or Puppet?? “whatever makes you happy” @iteration1 Wednesday, August 7, 13 Agile 2013 #AgileDevOps

IN G R SH A For chef/puppet, you’ll need to code. #NotJust4Devs #NotScary @iteration1 Wednesday, August 7, 13 Agile 2013 #AgileDevOps

IN G R SH A Stuck? Just ask your DevOps dev guy! #NotJust4Devs @iteration1 Wednesday, August 7, 13 Agile 2013 #AgileDevOps

IN G R SH A Don’t hide. Share the knowledge @iteration1 Wednesday, August 7, 13 Agile 2013 #AgileDevOps

IN G R SH A Wednesday, August 7, 13 App configs

IN G R SH A App configs Easy Change Management Let your automation control these kind of files Wednesday, August 7, 13

PS D EV O Tools Philosophy • Needed to work for cloud • Facilitate collaboration • Charts and graphs • Everyone wants to use it @iteration1 Wednesday, August 7, 13 Agile 2013 #AgileDevOps

E R SU EA M Tools Philosophy • Monitoring = Cloudkick • Monitor servers • Monitor code (custom synthetic checks) We use #datadog now @iteration1 Wednesday, August 7, 13 Agile 2013 #AgileDevOps

E R SU EA M Tools Philosophy Teach your devs how to monitor their code, and they will love the tool Example: #NewRelic @iteration1 Wednesday, August 7, 13 Agile 2013 #AgileDevOps

M EA SU R E Tools Philosophy If it measures business metrics, management will love you too! #statsd #dogstatsd HINT: Management pays you. Make them happy @iteration1 Wednesday, August 7, 13 Agile 2013 #AgileDevOps

E R SU EA M Tools Philosophy WASTE OF TIME: SSH’ing into a box to read log files @iteration1 Wednesday, August 7, 13 Agile 2013 #AgileDevOps

M EA SU R E Tools Philosophy Get a logging solution. Plethora of choices! #Logstash #Graylog #Splunk #Sumologic Which one? CloudAustin July Meetup @iteration1 Wednesday, August 7, 13 Agile 2013 #AgileDevOps

IN G R SH A Tools Philosophy You’ll need more tools Find them or write them! You’re devops now! @iteration1 Wednesday, August 7, 13 Agile 2013 #AgileDevOps

E R SU EA M Tools Philosophy DevOps victory: “People” use tools that solve their core problems @iteration1 Wednesday, August 7, 13 Agile 2013 #AgileDevOps

DevOps 201 MOAR Ideas and Stories For ops who can code and dev not scared of linux @iteration1 Wednesday, August 7, 13 Agile 2013 #AgileDevOps

R E U LT U C Is different for every team. What works for your team, works for your team @iteration1 Wednesday, August 7, 13 Agile 2013 #AgileDevOps

R E U LT U C This worked for us… #AlwaysBeInclusive @iteration1 Wednesday, August 7, 13 Agile 2013 #AgileDevOps

R E U LT U C Stuff Breaks Like…all the time @iteration1 Wednesday, August 7, 13 Agile 2013 #AgileDevOps

R E U LT U C @iteration1 Wednesday, August 7, 13 Agile 2013 #AgileDevOps

R E U LT U C #DontDoThis @iteration1 Wednesday, August 7, 13 Agile 2013 #AgileDevOps

R E U LT U C Be Cool. Fix the issue at hand If you panic, everyone else does too. @iteration1 Wednesday, August 7, 13 Agile 2013 #AgileDevOps

C U LT U R E Post Mortem Do a Post mortem @iteration1 Wednesday, August 7, 13 Agile 2013 #AgileDevOps

R E U LT C U Post Mortem It’s okay to admit mistakes You’re not Batman… @iteration1 Wednesday, August 7, 13 Agile 2013 #AgileDevOps

R E U LT C U Post Mortem Why x5 Add more tests Make sure it doesn’t happen again @iteration1 Wednesday, August 7, 13 Agile 2013 #AgileDevOps

R E U LT C U RELEASE OFTEN Release with 4 changes is less risky than Release with 40,000 changes @iteration1 Wednesday, August 7, 13 Agile 2013 #AgileDevOps

PS D EV O Wednesday, August 7, 13 Testing

D EV O PS Test Everything Test the code Test the infrastructure It’s all the same now #infrastructureAsCode @iteration1 Wednesday, August 7, 13 Agile 2013 #AgileDevOps

PS D EV O Free Time? Write more tests! @iteration1 Wednesday, August 7, 13 Agile 2013 #AgileDevOps

PS D EV O CONTINUOUSLY @iteration1 Wednesday, August 7, 13 Agile 2013 #AgileDevOps

PS D EV O CONTINUOUSLY @iteration1 Wednesday, August 7, 13 Agile 2013 #AgileDevOps

PS D EV O Manual Changes To Production @iteration1 Wednesday, August 7, 13 Agile 2013 #AgileDevOps

PS D EV O Manual Changes Don’t do this. Do this: 1. Make manual changes in dev 2. Add to your chef/puppet script 3. Release to test/prod @iteration1 Wednesday, August 7, 13 Agile 2013 #AgileDevOps

DevOps 301 MOAR MOAR Tips For when your team grows @iteration1 Wednesday, August 7, 13 Agile 2013 #AgileDevOps

Security in the loop Add security audits as a part of your sprint. We do an audit periodically @Gauntlt @RuggedDevOps @iteration1 Wednesday, August 7, 13 Agile 2013 #AgileDevOps

Security in the loop Hire an infosec pro to your devops team @iteration1 Wednesday, August 7, 13 Agile 2013 #AgileDevOps

Security in the loop No budget? Try a cross functional working group @iteration1 Wednesday, August 7, 13 Agile 2013 #AgileDevOps

Expanding Teams? • Rotate times for meetings (status/planning) • Scrum standups were recorded and dropbox(ed) • “OK to call” philosophy • Campfire chat (saves history) • Ownership of new features @iteration1 Wednesday, August 7, 13 Agile 2013 #AgileDevOps

Expanding Teams? • No secret: Face to face is always better • Prioritize travels for visits to teammates @iteration1 Wednesday, August 7, 13 Agile 2013 #AgileDevOps

Everyone asking for DevOps? The embedded DevOps model works* @iteration1 Wednesday, August 7, 13 Agile 2013 #AgileDevOps

Everyone asking for DevOps? Make sure your team culture is strong enough so you don’t lose person permanently :) @iteration1 Wednesday, August 7, 13 Agile 2013 #AgileDevOps

Start Small… Don’t say “WE’RE 100% DEVOPS TODAY” Wednesday, August 7, 13

DevOps 401 MOAR MOAR MOAR Ideas For next time. I don’t have any. Tweet #agiledevops for your best ideas! @iteration1 Wednesday, August 7, 13 Agile 2013 #AgileDevOps

STILL STUCK? Just ask! We’re devops, and this is real (even though we can’t define it) Twitter: #devopsama http://devopsweekly.com/ Go to a devopsdays https://groups.google.com/forum/#!forum/devops @iteration1 Wednesday, August 7, 13 Agile 2013 #AgileDevOps

DevOps: Retrospective @iteration1 Wednesday, August 7, 13 • DevOps and CAMS bring team together. • Building stable products on time, which were actually needed. • Superior collaboration between devs and ops • Organization shift -> DevOps Agile 2013 #AgileDevOps

DevOps: Retrospective • DevOps and CAMS bring team together. • Superior collaboration between business and IT. Building stable products on time, which were actually needed. • Superior collaboration between devs and ops Organization shift -> Scrum • Organization shift -> DevOps • Scrum brought team together. • Building products on time, which were actually needed. • • @iteration1 Wednesday, August 7, 13 Agile 2013 #AgileDevOps

Thank you! Let’s work together, and solve the problems that our business wants us to! #DevOps @iteration1 Wednesday, August 7, 13 Agile 2013 #AgileDevOps