Optimizing Data Retrieval with Python Celery Jenna Conn & Hannah Cline

Current Outline Introduction The Problem Possible Solutions Our Solution Takeaways

Jenna Conn & Hannah Cline

Background When do you use data? Choosing what to eat Online shopping Budget Apps Data Driven Decision Making – See 10 Tips For Your Business Success. datapine. (2020, August 17). https://www.datapine.com/blog/data-driven-decision-making-in-businesses/.

Problem REST API

Dashboards Staff, F. (2017, March 14). Fitbit Dashboard Updated with Weekly Activity and More. Fitbit Blog. https://blog.fitbit.com/fitbit-dashboard-updated-with-weekly-activity-and-more/.

Solutions ● ● ● ● Optimize DB queries Batch queries together Make more specialized API’s (microservices) Create a task queue Adams, S. (1970, November 7). Dilbert Comic Strip on November 03, 2007. Dilbert. https://dilbert.com/strip/2007-11-03.

Task Queues ● ● ● ● ● Distribute tasks across threads and/or machines Each thread carries out one function of code UI can add a task to the queue and the queue return an async Id back UI can then poll to see if results are ready Types ○ ○ ○ RabbitMQ Beanstalk Celery

Python Celery Simple task queue that requires no configuration files Why developers like Celery. StackShare. (n.d.). https://stackshare.io/celery. Can manage tasks across a single machine or multiple machines. Easy to integrate with web frameworks such as Django and Flask Has an active community

REST API ASYNC ID Message Broker

Message Broker Celery Worker CeleryCelery Worker Workers

“Is it ready?” ASYNC ID REST API PENDING, SUCCESS, FAILURE Message Broker Celery Celery Worker Celery Worker Workers

Video

Dashboard Staff, F. (2017, March 14). Fitbit Dashboard Updated with Weekly Activity and More. Fitbit Blog. https://blog.fitbit.com/fitbit-dashboard-updated-with-weekly-activity-and-more/.

Extras Lots of additional configuration is available Schedule tasks for different times Celery workers can be spread across multiple servers to enable scaling with minimal setup Can cache results in Redis to further increase speeds when duplicate queries are requested