A presentation at Front End North in in Sheffield, UK by Lorna Jane Mitchell
OfflineFirst Apps With PouchDB and CouchDB Lorna Mitchell, IBM
OfflineFirst Apps Offline is not an error condition “Offline capability is a key characteristic of modern Progressive Web Applications” - http://offlinefirst.org @lornajane
Being Offline Could include: • being in a place without internet infrastructure • being without a data package on your phone • being on the tube or a plane @lornajane
Being Offline Could include: • being in a place without internet infrastructure • being without a data package on your phone • being on the tube or a plane • getting the train from Huddersfield to … really anywhere @lornajane
OfflineFirst OfflineFirst means failing gracefully You’ll also hear PWA which is a Progressive Web App, covering more than just network failures @lornajane
Achieving OfflineFirst: Code Service Worker caches on first page load @lornajane
Achieving OfflineFirst: Data Client-side app and storage, background sync @lornajane
PouchDB and CouchDB • https://pouchdb.com/ • A database that your client-side javascript can use • Can also sync to CouchDB (-compatible) databases • https://couchdb.apache.org • A NoSQL document database • Best replication on the planet (probably) • HTTP API = good support in all languages @lornajane
Example App: Shopping List • Client-side JavaScript with PouchDB • Works locally • If connected, syncs to Cloudant/CouchDB • Code here: https://github.com/lornajane/robust-shopping-list @lornajane
PouchDB in Action In index.html: <script src=”/js/pouchdb-6.1.2.min.js”></script> <script src=”/js/shopping.js”></script> shopping.js is where my client-side JavaScript lives @lornajane
PouchDB in Action 1 var db = new PouchDB(‘shopping’); 2 var remoteDB = new PouchDB(‘http://localhost:5984/shopping’); 3 window.onload = function() { 4 db.sync(remoteDB, { live: true, retry: true } 5 ).on(‘change’, function (change) { 6 return getItemList().then(function (contents) { 7 document.getElementById(‘itemList’).innerHTML = con 8 }) 9 }).on(‘active’, function (info) { 10 return getItemList().then(function (contents) { 11 document.getElementById(‘itemList’).innerHTML = con 12 }); 13 }); @lornajane
NoSQL Document Database @lornajane
NoSQL Document Database @lornajane
NoSQL Document Database @lornajane
Document Databases Store collections of schemaless documents @lornajane
Document Databases Choose a document database if: • the records you store don’t have the same structure as one another • you need to change data structures without downtime • you like high availability @lornajane
Data Design for PouchDB • data structure: include nested data/array, omit empty fields • identifiers: pick a meaningful ID where appropriate • beware updating/appending data: these cause conflicts @lornajane
CouchDB Cluster Of Unreliable Commodity Hardware • HTTP API • JSON data format • Performant views use JavaScript and MapReduce • Ad-hoc queries with a JSON structure using Mango @lornajane
Curl and Not-Curl • love curl? (https://curl.haxx.se/) • try jq (https://stedolan.github.io/jq/) • hate curl? Try one of these • http-console https://github.com/cloudhead/http-console • Postman https://www.getpostman.com/ • for more, try this HTTP Tools post (and comments): http://lornajane.net/posts/2017/http-tools-roundup @lornajane
Fauxton Friendly web interface @lornajane
Fauxton @lornajane
Fauxton @lornajane
CouchDB: Lovely Doc DB I could stop here: • JSON format • HTTP interface and nice web UI • Scales well • Modern, performant document database @lornajane
Changes Feed A feed containing all database changes. GET /_changes @lornajane
Replication @lornajane
Replication • Replication can be in either direction - or both • Can be one-off, or continuous • Other CouchDB-compatible storage also exists • e.g. PouchDB, a JavaScript implementation @lornajane
Conflicts Change docs in both places, replicate again: 87bf-bluemix.cloudant.com:443/shopping> GET /hat?conflicts=true { _id: ‘123’, _rev: ‘4-ecbc38075f9a8535c123e523519613b9’, item: ‘cheese’, _conflicts: [ ‘3-0bb689d59034fb769d99dcf697ae2de7’ ] } CouchDB will always choose the same “winning” doc @lornajane
Conflicts Fetch the “losing” doc(s) with ?rev= parameter 87bf-bluemix.cloudant.com:443/shopping> GET /123?rev=3-0bb689d5903 { _id: 123, _rev: ‘3-0bb689d59034fb769d99dcf697ae2de7’, item: ‘cheddar cheese’ } CouchDB doesn’t store old revisions forever @lornajane
Mango @lornajane
Mango: CouchDB Queries Mango is a mongo-like query language, useful for ad-hoc querying It is a JSON structure containing: • Selector: the criteria to match records on • Fields: which fields to return • Sort: what order you’d like that in (use with Skip) • Limit: how many records (default = 25) @lornajane
Mango: Example Query Use a query like this with the _find endpoint { “selector”: { “Year”: {“$eq”: “2012”} }, “fields”: [“Quarter”, “Product line”], “limit”: 5 } @lornajane
Mango: Example Query $ curl -X POST -H Content-Type:application/json \ http://localhost:5984/products/_find —data @mango.json {“warning”:”no “docs”:[ {“Quarter”:”Q1 {“Quarter”:”Q1 {“Quarter”:”Q1 {“Quarter”:”Q1 {“Quarter”:”Q1 ]} matching index found, create an index to optimize q 2012”,”Product 2012”,”Product 2012”,”Product 2012”,”Product 2012”,”Product line”:”Mountaineering line”:”Mountaineering line”:”Mountaineering line”:”Mountaineering line”:”Mountaineering Equipment”}, Equipment”}, Equipment”}, Equipment”}, Equipment”} @lornajane
Mango: Indexes Describe the index in JSON, then use the _index endpoint { “index”: { “fields”: [“Year”] }, “name”: “Year” } @lornajane
Mango: Indexes $ curl -X POST -H Content-Type:application/json \ http://localhost:5984/products/_index —data @index.json { “result”: “created”, “id”: “_design/e9b54f2ac34b8823ccbe8aaf6f406d464f50f521”, “name”: “Year” } Check which indexes are used by putting _explain where the _find normally goes! @lornajane
Views @lornajane
Views • Written in Javascript • Use MapReduce • The map results are stored • Can be used either for filtering, or for aggregation @lornajane
MapReduce Primer: Map • Examine each document, “emit” 0+ keys/value pairs • Scales well because each document is independent • To filter a collection of documents, use map step only @lornajane
MapReduce Primer: Map @lornajane
MapReduce Primer: Map @lornajane
MapReduce Primer: Map @lornajane
MapReduce Primer: Map @lornajane
MapReduce Primer: Reduce @lornajane
MapReduce Primer: Reduce • “Reduce” values in batches with the same key • CouchDB has useful built in functions for most things • Use reduce step when you want aggregate data • (SQL equivalent: a query with GROUP BY) @lornajane
Views Example @lornajane
OfflineFirst Apps With PouchDB and CouchDB @lornajane
Example Apps Ready-made shopping list examples are available: • VanillaJS and PouchDB (a more detailed example) • Polymer and PouchDB • React and PouchDB • Vue.js and PouchDB • React Native and PouchDB https://github.com/ibm-watson-data-lab/shopping-list @lornajane
Resources • https://lornajane.net • https://github.com/lornajane/robust-shopping-list • https://github.com/ibm-watson-data-lab/shopping-list • https://offlinefirst.org • http://hood.ie/ @lornajane
This talk was for the excellent Frontend North conference in Sheffield
