Let’s take this offline

Matt Hobbs Head of Frontend, Lead Developer Government Digital Service @TheRealNooshu Intro slide

Let’s take this offline Morning all. So in this talk I’m going to give you a little information and a few demos on how we could potentially start leveraging the “offline first” methodology at GDS.

I recently gave a talk to the the cross-government Design community about Progressive Web Apps (PWAs). I’ve spoken about these before. PWA is a marketing term for a set of technologies that I find really interesting. Now I’m not going to delve into the details of PWAs, but I will just re-cap on the three core technologies required for a PWA. To be classed as a PWA, a website must use these three technologies:

1 - Web App Manifest GDS The first technology is the Web App Manifest. It is a short, simple file that describes the application, its icons, its background colors, and other design details.

GDS Here’s an example of a manifest file. The file holds some data that tells the browser things like: ● ● ● ● The name of the PWA The short name for the PWA (one that is displayed on the homescreen) The icons to use for the device home screen The background colour to use for the splash screen ○ A splash screen is the first screen that appears when you touch the icon. Essentially a loading screen. ● A theme colour for the theming the native controls (scroll bar, browser UI) ● How the application is displayed (with or without the browser UI for example)

2 - HTTPS GDS Second is HTTPS. PWAs will only work on a secure connection. This is because the technology involved has the ability to modify browser requests on the fly. This would be a serious security issue if on an unsecured connection.

3 - Service Worker GDS And lastly, and most importantly for this presentation is the Service Worker. So what is a service worker? The official definition is: “It’s a client side proxy written in JavaScript” That’s not very clear, so I’ll expand on it more in the following slides.

GDS I think of a Service Worker is a little gatekeeper that sits between the network and your web browser. It examines the traffic coming in and out (requests) and decides what to do with it. The service worker can modify requests, or let them pass as normal if no changes are required. What a service worker allows us to do is to store certain files in cache for an unlimited period of time. For example a user visiting GOV.UK for the first time could cache all the CSS, JavaScript and images when they first visit the page. The next time they visit GOV.UK, the browser only needs to download the HTML. This is a great example of using a service worker to improve performance for users. But we can take it further.

Offline Documentation For the rest of the talk I will focus on using service workers in our documentation. As an example of the power of service workers I’ve created three demos that will allow users to access documentation when offline.

I’ve used the GOV.UK Design System as a base to work from for this offline documentation. The reason for this choice is being a Frontend developer, I’m more at home with the build and workflow used on the DS. But all of the functionality shown in the demo’s can be replicated on any documentation or service if we like.

Demo 1: Passive In the first version of the demo I will show you a version where the user doesn’t need to do anything for the offline functionality. It all happens automatically. I’ve called this “Passive offline”.

http://bit.ly/gukds-passive GDS To try out the demo you can go to this URL in any modern browser. What you will see is a standard webpage. Nothing will be different, and it will look like nothing has happened. But if you enable offline mode, or turn off your connection, you will see a little red notice in the bottom right hand corner signifying that the browser recognises it is now disconnected from the network.

Offline detected GDS Due to the passive nature of the offline strategy used, a user will still be able to access all of the documentation. Even the search functionality will still work (since the search index is powered by a simple JSON file). [describe the GIF] So what has happened under the hood to make this happen?

Workbox Precaching GDS At the end of the build process an abstraction library called Workbox.js has been used to generate a large array of files that match a specific pattern (e.g. HTML, CSS, JavaScript, Images). All of the files for the whole design system have been listed and precached ahead of time. When a user comes to request that file, the service worker already has it sitting in the cache and it will serve it from the local file system. What you may also notice in the image is that each of the files is given a unique revision hash. Workbox generates these based on the file metadata, URL and content. So if a file changes by a single byte, it will have a whole new revision number. Workbox uses these revision numbers to check see which files have changed, and then updates them accordingly. So a user will never see an outdated version of a file, they will see the last version that was available when they visited the website.

Download size GDS Now you may be thinking: “If you’ve just downloaded the whole website how much data has that used?” That’s a very important consideration to be made, especially on mobile devices and users on expensive data plans. Amazingly, the 289 requests only came to a total of 2MB! That includes all HTML, CSS, JavaScript, Images and fonts. It’s a real testament to the excellent work the Design System team have done over the past year. When I first prototyped this in April 2018 the initial download was a whopping 23MB! That’s a 91% improvement in 12 months.

GDS To put that 2MB in perspective, the HTTPArchive track the state of the web by collecting metrics about millions of websites across the internet every day. According to their latest stats the median desktop page weight is 1.8MB, and for mobile it is almost 1.7MB. That’s for a single page! One of our users can get the whole of the Design System to view offline in 2MB. That’s pretty impressive! Now that all these assets are cached the performance of the website will be near instant for users. In many cases the browser doesn’t even need to make a HTTP request to the network even when online. It’s served directly from the users hard drive. A real web performance win!

Demo 2: Semi-passive Demo 2 again uses the Design System but the offline-caching works differently. I’ve called this “semi-passive” because naming things is hard and I couldn’t think of anything better.

http://bit.ly/gukds-spassive GDS Here’s another URL you can try in the browser. This demo introduces the concept of a separate offline page. The offline page is the fallback that is displayed if the network connection fails.

GDS You can use this offline page for anything really. For example, the Guardian have gamified their offline page. A user on an unreliable network (e.g. commuting on the tube / train, or in a rural area) will receive a random interactive crossword they can complete while they wait for a connection to re-establish.

GDS Trivago do something similar only with an interactive maze. A real bonus for them is they found 67% of users continued to browse the site once the connection was re-established. Gamification is great but not really relevant to our users.

Visited pages GDS So what we can actually do is show our users a list of the last X number of pages they have viewed. That could be as simple as the last 10, or the last 100 pages, it is entirely up to us to decide. [describe the GIF] So what is happening at the service worker level to make this work?

For the pages we are using something called the Stale while revalidate strategy. What this does is allow us to serve a resource as fast as possible from the cache, while the network is used to look for an updated resource. In the diagram arrows 2 and 4 show the requests out to cache and network at the same time. Whichever one completes first is passed back to the page (always the cache). If it is that the cache has an outdated version compared to the network, this version will be replaced in the cache. That way the user gets an immediate response, and assets that are always up-to-date. A network first strategy could also be used in this situation. This is where the SW will try the network first, but if the network isn’t available then the cache will be used as a fallback.

GDS The service worker code for this setup is actually quite straightforward (it’s even simpler using Workbox). There’s some JavaScript used on the offline page to loop through the service worker cache and pull page meta-data from localStorage to show the list in the page. The good thing about this setup is we aren’t imposing additional data requirements on the user. The page will be stored only when they visit the page. The browser storage cleanup is automated. If the user has visited 21 pages but we are only storing the last 20, the oldest is automatically removed from the cache. And again the performance benefit is huge. For users visiting the same page more than once, a high percentage (probably 100%) of the page assets will already be cached, meaning an almost instant page load.

Demo 3: Active And the final demo, version 3 builds on some of the work from demo 2, but with a major functionality change. In this version we give the user the option to save / unsave pages for reading later.

http://bit.ly/gukds-active GDS And here’s the final URL for you to try. This demo still has a separate offline page, but pages are only saved if a user specifically asks to save the page. Hopefully you can read the link, if not it says “Save for reading offline”.

GDS Here’s an example in the wild being used on the Financial Times website. It allows a user to save articles for offline reading and store them in “myFT”. I believe other news sites are also starting to experiment with this functionality too.

Save & delete offline pages GDS In this demo we give each user the option to save the pages that would like to refer to later. A user gets instant feedback that the page has been saved. They can remove the page from the cache if they no longer want it stored offline too. They are in complete control. [describe the GIF] So what is happening at the service worker level to make this work?

For this functionality we are using a cache-first strategy. Once a user decides to save the page for offline, the service worker will look in its cache for the page (arrow 2). If the page isn’t found in the cache, the network will be used (3). You may be wondering what happens if the page is updated on the network when a user has saved it for offline. It has been built in such a way that if there’s a new version of the page on the network it will automatically be updated in the cache. So a user will never see an outdated version of the page.

iframes saved GDS In this example we see a page with lots of iframes, each containing important guidance that’s very relevant to the page content. With a little bit of JavaScript magic it’s possible to save the iframes in the page too, so they are also available for offline reading. You could do this with any media on a page, be that images, audio or even videos. Since the user is only storing the pages they want, the resource footprint on the device is very small. There’s the same web performance benefit for users who revisit the same page. Loading will be almost instant.

Progressive Enhancement GDS The great thing about these demos and this technology in general is it can be applied using the progressive enhancement methodology. Offline functionality will only work if the user has a browser that supports the technology. Any older browser that doesn’t support it will see no difference, they will just get a standard webpage.

GDS That being said, the support for this technology is really good. 88% of users in the UK use a browser that supports service workers according to the latest statistics on CanIUse.

The future So what can we do with this technology at GDS? Well as you’ve seen it is possible to add some offline capability for our documentation. But there’s no reason this can’t be rolled out to all our services if there’s a user need for it.

I personally could see this being really useful for guidance pages on GOV.UK. “I don’t have time to read all this guidance now, but on my journey home commuting would be perfect”. The internet connection isn’t reliable, but we can save for offline later. That could be the perfect chance to do something useful with this commuting time. We already have a “Print entire guide” link available to users. There could be a “Save for reading offline” link that sits right next to it. It doesn’t have to be across the whole of GOV.UK either, it could be specifically tailored to certain sections.

The same applies for the Service Manual. Apparently one of the most requested features is to have the guidance in the form of a PDF for reading offline. Introducing a service worker and adding the ability to save the guidance for offline reading should be fairly straight-forward. There are no complex assets like audio, video or iframes to deal with. Just a set of static pages.

Feedback I’d love to get some feedback on what people think of this idea, or where else it could be used. So feel free to hunt me down for a chat if you’d like to discuss it further.

Demos ● “Passive” - http://bit.ly/gukds-passive ● “Semi-passive” - http://bit.ly/gukds-spassive ● “Active” - http://bit.ly/gukds-active GDS Also, any feedback on the demos I’ve thrown together. The code isn’t production ready by any means, but they are working prototypes. Here are the URL’s again if you’re interested in trying them out.

Thanks! Matt Hobbs @TheRealNooshu Thanks for listening. If anyone would like to find out more please get in touch. I’m more than happy to try and answer any questions you have.