Optimizing Data Retrieval with Python Celery

A presentation at Pycon in April 2021 in by Jenna Conn

Slide 1

Slide 1

Optimizing Data Retrieval with Python Celery Jenna Conn & Hannah Cline

Slide 2

Slide 2

Current Outline Introduction The Problem Possible Solutions Our Solution Takeaways

Slide 3

Slide 3

Jenna Conn & Hannah Cline

Slide 4

Slide 4

Background When do you use data? Choosing what to eat Online shopping Budget Apps Data Driven Decision Making – See 10 Tips For Your Business Success. datapine. (2020, August 17). https://www.datapine.com/blog/data-driven-decision-making-in-businesses/.

Slide 5

Slide 5

Problem REST API

Slide 6

Slide 6

Dashboards Staff, F. (2017, March 14). Fitbit Dashboard Updated with Weekly Activity and More. Fitbit Blog. https://blog.fitbit.com/fitbit-dashboard-updated-with-weekly-activity-and-more/.

Slide 7

Slide 7

Solutions ● ● ● ● Optimize DB queries Batch queries together Make more specialized API’s (microservices) Create a task queue Adams, S. (1970, November 7). Dilbert Comic Strip on November 03, 2007. Dilbert. https://dilbert.com/strip/2007-11-03.

Slide 8

Slide 8

Task Queues ● ● ● ● ● Distribute tasks across threads and/or machines Each thread carries out one function of code UI can add a task to the queue and the queue return an async Id back UI can then poll to see if results are ready Types ○ ○ ○ RabbitMQ Beanstalk Celery

Slide 9

Slide 9

Python Celery Simple task queue that requires no configuration files Why developers like Celery. StackShare. (n.d.). https://stackshare.io/celery. Can manage tasks across a single machine or multiple machines. Easy to integrate with web frameworks such as Django and Flask Has an active community

Slide 10

Slide 10

REST API ASYNC ID Message Broker

Slide 11

Slide 11

Message Broker Celery Worker CeleryCelery Worker Workers

Slide 12

Slide 12

“Is it ready?” ASYNC ID REST API PENDING, SUCCESS, FAILURE Message Broker Celery Celery Worker Celery Worker Workers

Slide 13

Slide 13

Video

Slide 14

Slide 14

Dashboard Staff, F. (2017, March 14). Fitbit Dashboard Updated with Weekly Activity and More. Fitbit Blog. https://blog.fitbit.com/fitbit-dashboard-updated-with-weekly-activity-and-more/.

Slide 15

Slide 15

Extras Lots of additional configuration is available Schedule tasks for different times Celery workers can be spread across multiple servers to enable scaling with minimal setup Can cache results in Redis to further increase speeds when duplicate queries are requested