Update Strategies for the Edge There’s a better way. @jfrog

Kat Cosgrove IoT Engineer Developer Advocate Twitter: @Dixie3Flatline Email: katc@jfrog.com jfrog.com/shownotes @jfrog

How large is the Edge? @jfrog

20,400,000,000 That’s a lot of devices. @jfrog

Updates Today They don’t update; device is effectively single-use OR It’s time-consuming, complicated, or requires physical access @jfrog

Why change? @jfrog

It’s beyond inconvenient Edge computing is massive and growing

  • Consumer - Industrial - Medical Slow OTA updates are annoying Wired updates are expensive and more annoying @jfrog

It’s dangerous Unpatched bugs can be a huge vulnerability

Expose private data Harnessed for a botnet Used for cryptocurrency mining Safety implications for medical @jfrog

What’s slowing us down? @jfrog

Not building for it. Many devices are not made to be updated.

  • Designed to run one version until the end - “Update strategy” here is flashing the device - Bugs are inevitable @jfrog

Between 1 and 25 Number of bugs per 1000 LOC @jfrog

Connectivity Concerns We can’t rely on the device’s network

  • Networks may be unstable - Bandwidth may be low - Network probably isn’t secure @jfrog

Hardware Variations - It’s 20.4 billion devices - Lots of specialized hardware - Variations in memory, storage space, architecture How do we design something that handles so much variety? @jfrog

Think future-forward. Updates are your friend. Embrace updates, not security nightmares. @jfrog

Get better with age. Your product should not be getting worse from the moment it ships. @jfrog

Build robust. Brittle software means a brittle device, and that doesn’t inspire trust. @jfrog

Modern DevOps tools. Your developers will thank you and things will run more smoothly. @jfrog

The Proof of Concept @jfrog

@jfrog | Copyright © 2019 JFrog. All Rights Reserved

Cars Now - Majority not designed for OTA updates OTA updates are still slow and inconvenient Little standardization Significant portion of recalls are due to software @jfrog | Copyright © 2019 JFrog. All Rights Reserved

Cars as Edge Devices - Presented a range of solvable pain points in one device - Tangible example for end users and manufacturers - Device in question meant speed, reliability, and safety were equally important @jfrog | Copyright © 2019 JFrog. All Rights Reserved

Workflows and Tools @jfrog | Copyright © 2019 JFrog. All Rights Reserved

Two Distinct Workflows Software Updates Firmware Updates

  • Without flashing firmware - No interruption of user - Takes only seconds - Relies on K3S and Helm @jfrog |
  • More difficult update - Takes only minutes - Rollback if there is a failure - Relies on Mender, Yocto, and Artifactory Copyright © 2019 JFrog. All Rights Reserved

Software Workflow @jfrog | Copyright © 2019 JFrog. All Rights Reserved

PIPELINES VCS & CI Code & Build CD ARTIFACTORY XRAY Schedule Containers K3S + Helm ACCESS @jfrog | Copyright © 2019 JFrog. All Rights Reserved Deploy to production (car) MISSION CONTROL

@jfrog | Copyright © 2019 JFrog. All Rights Reserved

@jfrog | Copyright © 2019 JFrog. All Rights Reserved

JFrog Xray - Vulnerability scanning tool All major package types supported Continuously scans your artifacts Risk Based Security’s VulnDB @jfrog | Copyright © 2019 JFrog. All Rights Reserved

Kubernetes, but 5 less @jfrog | Copyright © 2019 JFrog. All Rights Reserved

K3S - Lightweight Kubernetes, designed for Edge devices - Uses only 512mb of RAM - 40mb binary - Very minimal OS requirements @jfrog | Copyright © 2019 JFrog. All Rights Reserved

A package manager for Kubernetes @jfrog | Copyright © 2019 JFrog. All Rights Reserved

Helm “Charts” describe complex applications

Easily repeatable installation Final authority on application Easy to version Supports rollbacks @jfrog | Copyright © 2019 JFrog. All Rights Reserved

Helm Charts @jfrog | Copyright © 2019 JFrog. All Rights Reserved

The Result - Software Application updates are quick and efficient

Average of 35 seconds from dev to car No interruption for the user Can happen while device is in use Could happen silently, depends on device purpose @jfrog | Copyright © 2019 JFrog. All Rights Reserved

Firmware Workflow @jfrog | Copyright © 2019 JFrog. All Rights Reserved

PIPELINES VCS & CI CODE & BUILD XRAY ARTIFACTORY EMBEDDED OS ACCESS @jfrog | Copyright © 2019 JFrog. All Rights Reserved Deploy to production (car) MISSION CONTROL

OTA updates for embedded Linux devices @jfrog | Copyright © 2019 JFrog. All Rights Reserved

Mender Overview Ticks several of the boxes we’re looking for:

Updates are signed and verified Supports automatic rollbacks Several distinct installation strategies Dual A/B strategy @jfrog | Copyright © 2019 JFrog. All Rights Reserved

Mender - A/B Two partitions are on the device

  • Bootloader aware of “active” - Update streams to “inactive” - Automatically revert to previous partition on failure Update A User Space A Update B Kernel Initramfs A Now let’s handle the size of our builds. @jfrog | User Space B Copyright © 2019 JFrog. All Rights Reserved Kernel Initramfs B Bootloader

Custom Linux distributions for any hardware architecture @jfrog | Copyright © 2019 JFrog. All Rights Reserved

Yocto Overview - Eliminates OS bloat Drastically reduces resources required BitBake recipes and layers define your build Layers for common configurations are provided Custom layers to isolate applications or behaviors @jfrog | Copyright © 2019 JFrog. All Rights Reserved

Yocto Layers @jfrog | Copyright © 2019 JFrog. All Rights Reserved

Yocto and Artifactory - After first build, we can make things much faster Yocto cache allows for incremental updates Build cache can be stored in Artifactory Reduces time required to build by up to 50% @jfrog | Copyright © 2019 JFrog. All Rights Reserved

The Result - Firmware - Cuts the total time after first build to 5-10 minutes Build is as small as possible Updates are signed and secure Automatic rollbacks in case of failure Success! @jfrog | Copyright © 2019 JFrog. All Rights Reserved

Other Tools @jfrog | Copyright © 2019 JFrog. All Rights Reserved

OSTree Git for operating systems @jfrog | Copyright © 2019 JFrog. All Rights Reserved

OSTree - Versions updates of Linux operating systems Git-like system with branching Tracks file system trees Allows for updates and rollbacks Exists as a meta-layer for Yocto @jfrog | Copyright © 2019 JFrog. All Rights Reserved

Testing framework for operating systems on embedded devices @jfrog | Copyright © 2019 JFrog. All Rights Reserved

LAVA - Linaro Automation and Validation Architecture CI system for deploying an OS to device for testing Can deploy to physical or virtual hardware Boot testing, bootloader testing, or system testing Results tracked over time @jfrog | Copyright © 2019 JFrog. All Rights Reserved

LAVA - Designed for validation during development For example, whether the kernel compiles and boots Templates for more than 100 boards built in Custom devices types can be added @jfrog | Copyright © 2019 JFrog. All Rights Reserved

LAVA Tests @jfrog | Copyright © 2019 JFrog. All Rights Reserved

Wrapping Up @jfrog | Copyright © 2019 JFrog. All Rights Reserved

Edge and IoT updates are broken This is a security problem that must be addressed Modern DevOps tools are here to help @jfrog | Copyright © 2019 JFrog. All Rights Reserved

THANKS! @Dixie3Flatline katc@jfrog.com jfrog.com/shownotes @jfrog | Copyright © 2019 JFrog. All Rights Reserved

@jfrog | Copyright © 2019 JFrog. All Rights Reserved