Offline First Apps with PouchDB and CouchDB

OfflineFirst Apps With PouchDB and CouchDB Lorna Mitchell, IBM

OfflineFirst Apps Offline is not an error condition “Offline capability is a key characteristic of modern Progressive Web Applications” - http://offlinefirst.org @lornajane

Being Offline Could include: • being in a place without internet infrastructure • being without a data package on your phone • being on the tube or a plane @lornajane

Being Offline Could include: • being in a place without internet infrastructure • being without a data package on your phone • being on the tube or a plane • getting the train from Huddersfield to … really anywhere @lornajane

OfflineFirst OfflineFirst means failing gracefully You’ll also hear PWA which is a Progressive Web App, covering more than just network failures @lornajane

Achieving OfflineFirst: Code Service Worker caches on first page load @lornajane

Achieving OfflineFirst: Data Client-side app and storage, background sync @lornajane

PouchDB and CouchDB • https://pouchdb.com/ • A database that your client-side javascript can use • Can also sync to CouchDB (-compatible) databases • https://couchdb.apache.org • A NoSQL document database • Best replication on the planet (probably) • HTTP API = good support in all languages @lornajane

Example App: Shopping List • Client-side JavaScript with PouchDB • Works locally • If connected, syncs to Cloudant/CouchDB • Code here: https://github.com/lornajane/robust-shopping-list @lornajane

PouchDB in Action In index.html: <script src=”/js/pouchdb-6.1.2.min.js”></script> <script src=”/js/shopping.js”></script> shopping.js is where my client-side JavaScript lives @lornajane

PouchDB in Action 1 var db = new PouchDB(‘shopping’); 2 var remoteDB = new PouchDB(‘http://localhost:5984/shopping’); 3 window.onload = function() { 4 db.sync(remoteDB, { live: true, retry: true } 5 ).on(‘change’, function (change) { 6 return getItemList().then(function (contents) { 7 document.getElementById(‘itemList’).innerHTML = con 8 }) 9 }).on(‘active’, function (info) { 10 return getItemList().then(function (contents) { 11 document.getElementById(‘itemList’).innerHTML = con 12 }); 13 }); @lornajane

NoSQL Document Database @lornajane

NoSQL Document Database @lornajane

NoSQL Document Database @lornajane

Document Databases Store collections of schemaless documents @lornajane

Document Databases Choose a document database if: • the records you store don’t have the same structure as one another • you need to change data structures without downtime • you like high availability @lornajane

Data Design for PouchDB • data structure: include nested data/array, omit empty fields • identifiers: pick a meaningful ID where appropriate • beware updating/appending data: these cause conflicts @lornajane

CouchDB Cluster Of Unreliable Commodity Hardware • HTTP API • JSON data format • Performant views use JavaScript and MapReduce • Ad-hoc queries with a JSON structure using Mango @lornajane

Curl and Not-Curl • love curl? (https://curl.haxx.se/) • try jq (https://stedolan.github.io/jq/) • hate curl? Try one of these • http-console https://github.com/cloudhead/http-console • Postman https://www.getpostman.com/ • for more, try this HTTP Tools post (and comments): http://lornajane.net/posts/2017/http-tools-roundup @lornajane

Fauxton Friendly web interface @lornajane

Fauxton @lornajane

Fauxton @lornajane

CouchDB: Lovely Doc DB I could stop here: • JSON format • HTTP interface and nice web UI • Scales well • Modern, performant document database @lornajane

Changes Feed A feed containing all database changes. GET /_changes @lornajane

Replication @lornajane

Replication • Replication can be in either direction - or both • Can be one-off, or continuous • Other CouchDB-compatible storage also exists • e.g. PouchDB, a JavaScript implementation @lornajane

Conflicts Change docs in both places, replicate again: 87bf-bluemix.cloudant.com:443/shopping> GET /hat?conflicts=true { _id: ‘123’, _rev: ‘4-ecbc38075f9a8535c123e523519613b9’, item: ‘cheese’, _conflicts: [ ‘3-0bb689d59034fb769d99dcf697ae2de7’ ] } CouchDB will always choose the same “winning” doc @lornajane

Conflicts Fetch the “losing” doc(s) with ?rev= parameter 87bf-bluemix.cloudant.com:443/shopping> GET /123?rev=3-0bb689d5903 { _id: 123, _rev: ‘3-0bb689d59034fb769d99dcf697ae2de7’, item: ‘cheddar cheese’ } CouchDB doesn’t store old revisions forever @lornajane

Mango @lornajane

Mango: CouchDB Queries Mango is a mongo-like query language, useful for ad-hoc querying It is a JSON structure containing: • Selector: the criteria to match records on • Fields: which fields to return • Sort: what order you’d like that in (use with Skip) • Limit: how many records (default = 25) @lornajane

Mango: Example Query Use a query like this with the _find endpoint { “selector”: { “Year”: {“$eq”: “2012”} }, “fields”: [“Quarter”, “Product line”], “limit”: 5 } @lornajane

Mango: Example Query $ curl -X POST -H Content-Type:application/json \ http://localhost:5984/products/_find —data @mango.json {“warning”:”no “docs”:[ {“Quarter”:”Q1 {“Quarter”:”Q1 {“Quarter”:”Q1 {“Quarter”:”Q1 {“Quarter”:”Q1 ]} matching index found, create an index to optimize q 2012”,”Product 2012”,”Product 2012”,”Product 2012”,”Product 2012”,”Product line”:”Mountaineering line”:”Mountaineering line”:”Mountaineering line”:”Mountaineering line”:”Mountaineering Equipment”}, Equipment”}, Equipment”}, Equipment”}, Equipment”} @lornajane

Mango: Indexes Describe the index in JSON, then use the _index endpoint { “index”: { “fields”: [“Year”] }, “name”: “Year” } @lornajane

Mango: Indexes $ curl -X POST -H Content-Type:application/json \ http://localhost:5984/products/_index —data @index.json { “result”: “created”, “id”: “_design/e9b54f2ac34b8823ccbe8aaf6f406d464f50f521”, “name”: “Year” } Check which indexes are used by putting _explain where the _find normally goes! @lornajane

Views @lornajane

Views • Written in Javascript • Use MapReduce • The map results are stored • Can be used either for filtering, or for aggregation @lornajane

MapReduce Primer: Map • Examine each document, “emit” 0+ keys/value pairs • Scales well because each document is independent • To filter a collection of documents, use map step only @lornajane

MapReduce Primer: Map @lornajane

MapReduce Primer: Map @lornajane

MapReduce Primer: Map @lornajane

MapReduce Primer: Map @lornajane

MapReduce Primer: Reduce @lornajane

MapReduce Primer: Reduce • “Reduce” values in batches with the same key • CouchDB has useful built in functions for most things • Use reduce step when you want aggregate data • (SQL equivalent: a query with GROUP BY) @lornajane

Views Example @lornajane

OfflineFirst Apps With PouchDB and CouchDB @lornajane

Example Apps Ready-made shopping list examples are available: • VanillaJS and PouchDB (a more detailed example) • Polymer and PouchDB • React and PouchDB • Vue.js and PouchDB • React Native and PouchDB https://github.com/ibm-watson-data-lab/shopping-list @lornajane

Resources • https://lornajane.net • https://github.com/lornajane/robust-shopping-list • https://github.com/ibm-watson-data-lab/shopping-list • https://offlinefirst.org • http://hood.ie/ @lornajane