Optimizing Data Retrieval with Python Celery Jenna Conn & Hannah Cline
A presentation at Pycon in April 2021 in by Jenna Conn
Optimizing Data Retrieval with Python Celery Jenna Conn & Hannah Cline
Current Outline Introduction The Problem Possible Solutions Our Solution Takeaways
Jenna Conn & Hannah Cline
Background When do you use data? Choosing what to eat Online shopping Budget Apps Data Driven Decision Making – See 10 Tips For Your Business Success. datapine. (2020, August 17). https://www.datapine.com/blog/data-driven-decision-making-in-businesses/.
Problem REST API
Dashboards Staff, F. (2017, March 14). Fitbit Dashboard Updated with Weekly Activity and More. Fitbit Blog. https://blog.fitbit.com/fitbit-dashboard-updated-with-weekly-activity-and-more/.
Solutions ● ● ● ● Optimize DB queries Batch queries together Make more specialized API’s (microservices) Create a task queue Adams, S. (1970, November 7). Dilbert Comic Strip on November 03, 2007. Dilbert. https://dilbert.com/strip/2007-11-03.
Task Queues ● ● ● ● ● Distribute tasks across threads and/or machines Each thread carries out one function of code UI can add a task to the queue and the queue return an async Id back UI can then poll to see if results are ready Types ○ ○ ○ RabbitMQ Beanstalk Celery
Python Celery Simple task queue that requires no configuration files Why developers like Celery. StackShare. (n.d.). https://stackshare.io/celery. Can manage tasks across a single machine or multiple machines. Easy to integrate with web frameworks such as Django and Flask Has an active community
REST API ASYNC ID Message Broker
Message Broker Celery Worker CeleryCelery Worker Workers
“Is it ready?” ASYNC ID REST API PENDING, SUCCESS, FAILURE Message Broker Celery Celery Worker Celery Worker Workers
Video
Dashboard Staff, F. (2017, March 14). Fitbit Dashboard Updated with Weekly Activity and More. Fitbit Blog. https://blog.fitbit.com/fitbit-dashboard-updated-with-weekly-activity-and-more/.
Extras Lots of additional configuration is available Schedule tasks for different times Celery workers can be spread across multiple servers to enable scaling with minimal setup Can cache results in Redis to further increase speeds when duplicate queries are requested
