From Hype to Help: Intelligent GenAI Adoption

From Hype to Help: Intelligent GenAI Adoption Sasha Czarkowski (Rosenbaum) @DivineOps COPYRIGHT 2024. ALL RIGHTS RESERVED.

From Hype to Help: Intelligent GenAI Adoption Sasha Czarkowski @DivineOps COPYRIGHT 2024. ALL RIGHTS RESERVED.

Sasha Czarkowski (Rosenbaum) @DivineOps Dev Ops Product Sales Management

Ergonautic Stop Chasing Buzzwords Start Working Better @ergonauticly COPYRIGHT 2024. ALL RIGHTS RESERVED. www.ergonautic.ly

Ergonautic Stop Chasing Buzzwords Start Working Better @ergonauticly COPYRIGHT 2024. ALL RIGHTS RESERVED. www.ergonautic.ly

How about you? Sasha Czarkowski COPYRIGHT 2024. ALL RIGHTS RESERVED. Ergonautic

Transformers are the most Transformative Technology Of the last 2 decades Sasha Czarkowski COPYRIGHT 2024. ALL RIGHTS RESERVED. www.ergonautic.ly

Have you ever asked an LLM to generate an image? Sasha Czarkowski COPYRIGHT 2024. ALL RIGHTS RESERVED. Ergonautic

2 hours later… Sasha Czarkowski COPYRIGHT 2024. ALL RIGHTS RESERVED. Ergonautic

Intermittent reinforcement Sasha Czarkowski COPYRIGHT 2024. ALL RIGHTS RESERVED. Ergonautic

We work harder for intermittent rewards Sasha Czarkowski COPYRIGHT 2024. ALL RIGHTS RESERVED. Ergonautic

GenAI is an accidental experiment in Intermittent reinforcement Sasha Czarkowski COPYRIGHT 2024. ALL RIGHTS RESERVED. Ergonautic

Your “feels” might be wrong: “While Copilot did not necessarily improve the task completion time or success rate, most participants preferred to use Copilot in daily programming tasks” Source: Expectation vs. Experience: Evaluating the Usability of Code Generation Tools Powered by Large Language Models. Sasha Czarkowski COPYRIGHT 2024. ALL RIGHTS RESERVED. Ergonautic

“What I’m telling you is, productivity is meaningless unless you know what your goal is” Sasha Czarkowski COPYRIGHT 2024. ALL RIGHTS RESERVED. Ergonautic

Your choice of productivity metrics will determine your outcomes Sasha Czarkowski COPYRIGHT 2024. ALL RIGHTS RESERVED. Ergonautic

Source: https://www.klarna.com/international/press/klarna-ai-assistant-handles-two-thirds-of-customer-service-chats-in-its-first-month/

Source: https://www.theregister.com/2024/10/16/ibm_insurance_industry_bosses_keen/

Feb 2023 Source: https://github.blog/news-insights/research/research-quantifying-github-copilots-impact-on-developer-productivity-and-happiness/

Apr 2024 Source: https://resources.uplevelteam.com/gen-ai-for-coding

Source: https://www.upwork.com/research/ai-enhanced-work-models

We have been here before Sasha Czarkowski COPYRIGHT 2024. ALL RIGHTS RESERVED. Ergonautic

More Robots @ergonauticly COPYRIGHT 2024. ALL RIGHTS RESERVED. www.ergonautic.ly

Tesla Robot Revolution ● Tesla’s 2017 Model 3 assembly line built with > 1000 robots ● Musk said the line will produce 20,000 cars a month by EOY ● Tesla made only 2,425 in the 4th quarter of 2017 Sasha Czarkowski COPYRIGHT 2024. ALL RIGHTS RESERVED. Ergonautic

But wait, there is more Sasha Czarkowski COPYRIGHT 2024. ALL RIGHTS RESERVED. Ergonautic

GM Robot Revolution ● In 1980s, GM spends > $45 Billion on industrial robots ● CEO Roger Smith lauded as a “visionary 21st century futurist” ● By 1990, GM earnings per US car sale fell from $588 to $47 Sasha Czarkowski COPYRIGHT 2024. ALL RIGHTS RESERVED. Ergonautic

Ergonautic

But wait, there is more Sasha Czarkowski COPYRIGHT 2024. ALL RIGHTS RESERVED. Ergonautic

The origins of the term “Socio-Technical”: studies by Eric Trist in 1950 Britain of mechanization of Coal Mining Sasha Czarkowski COPYRIGHT 2024 ALL RIGHTS RESERVED. Ergonautic

“Shortwall” Sasha Czarkowski vs COPYRIGHT 2024. ALL RIGHTS RESERVED. “Longwall” www.ergonautic.ly

vs Video: Safety in Longwall Mining - https://youtu.be/85gg_kymDxg

Mostly Manual Giant Machine vs Sasha Czarkowski COPYRIGHT 2024. ALL RIGHTS RESERVED. www.ergonautic.ly

The expected productivity of the new mechanized approach did not develop Sasha Czarkowski COPYRIGHT 2024 ALL RIGHTS RESERVED. Ergonautic

Ergonautic

Timeless Ergonautic

All this effort gets shut down In favor of top-down bureaucracy With “it must have only worked there” Sasha Czarkowski COPYRIGHT 2024 ALL RIGHTS RESERVED. Ergonautic

The Irony @ergonauticly COPYRIGHT 2024. ALL RIGHTS RESERVED. www.ergonautic.ly

February 17 2024 Sasha Czarkowski COPYRIGHT 2024. ALL RIGHTS RESERVED. www.ergonautic.ly

Aggregated decades of Human Factors research, and new studies of GenAI automation usage in different domains Sasha Czarkowski COPYRIGHT 2024. ALL RIGHTS RESERVED. Ergonautic

Source: Ironies of Generative AI: Understanding and mitigating productivity loss in human-AI interactions Sasha Czarkowski COPYRIGHT 2024. ALL RIGHTS RESERVED. Ergonautic

Production to Evaluation • Output overload • Complacency • Reduced situational awareness Sasha Czarkowski COPYRIGHT 2024. ALL RIGHTS RESERVED. Ergonautic

Workflow Restructuring New lengthy tasks competing with workflow: •Prompting •Output Adaptation Sasha Czarkowski COPYRIGHT 2024. ALL RIGHTS RESERVED. Ergonautic

Workflow Restructuring Errors due to: • Loss of task sequence • Loss of progress tracking Sasha Czarkowski COPYRIGHT 2024. ALL RIGHTS RESERVED. Ergonautic

Workflow Restructuring Loss of feedback: Loss of rich social commentary on coding suggestions Sasha Czarkowski COPYRIGHT 2024. ALL RIGHTS RESERVED. Ergonautic

Task Interruptions Cognitive costs Broken Flow Sasha Czarkowski COPYRIGHT 2024. ALL RIGHTS RESERVED. Ergonautic

Task Complexity Polarization Automation can make easy tasks easier and hard tasks even harder Sasha Czarkowski COPYRIGHT 2024. ALL RIGHTS RESERVED. Ergonautic

Inevitably, the friction with the machine is getting blamed on human error Sasha Czarkowski COPYRIGHT 2024. ALL RIGHTS RESERVED. Ergonautic

We have this incredible tool Sasha Czarkowski COPYRIGHT 2024. ALL RIGHTS RESERVED. Ergonautic

But things are not as simple as they seem Sasha Czarkowski COPYRIGHT 2024. ALL RIGHTS RESERVED. Ergonautic

The Transformation @ergonauticly COPYRIGHT 2024. ALL RIGHTS RESERVED. www.ergonautic.ly

GitLab Duo Generate: ● Documentation ● Test Cases ● Release Notes ● Code Reviews ● Incident Summaries ● External Codebase Analysis Source: https://about.gitlab.com/blog/2024/05/20/developinggitlab-duo-how-we-are-dogfooding-our-ai-features/ Sasha Czarkowski COPYRIGHT 2024. ALL RIGHTS RESERVED. Ergonautic

xkcd.com/1425/

Sasha Czarkowski COPYRIGHT 2024. ALL RIGHTS RESERVED. www.ergonautic.ly

COPYRIGHT 2024. ALL RIGHTS RESERVED.

Ergonautic Is AI winter coming?

Serverless GenAI? Cloud IoT Continuous Delivery Big Data DevOps Lean Agile Microservices

Workflow Design @ergonauticly COPYRIGHT 2024. ALL RIGHTS RESERVED. www.ergonautic.ly

Person Sasha Czarkowski Product www.ergonautic.ly

Personal [work] flow @ergonauticly COPYRIGHT 2024. ALL RIGHTS RESERVED. www.ergonautic.ly

“The Death of the Junior Professional” Sasha Czarkowski COPYRIGHT 2024. ALL RIGHTS RESERVED. Ergonautic

What skills do “seniors” have? Sasha Czarkowski COPYRIGHT 2024. ALL RIGHTS RESERVED. Ergonautic

Recognizing what “good” looks like Sasha Czarkowski COPYRIGHT 2024. ALL RIGHTS RESERVED. Ergonautic

Sasha Czarkowski COPYRIGHT 2024. ALL RIGHTS RESERVED. Ergonautic

Desired State Current State Sasha Czarkowski COPYRIGHT 2024. ALL RIGHTS RESERVED. www.ergonautic.ly

Sasha Czarkowski COPYRIGHT 2024. ALL RIGHTS RESERVED. www.ergonautic.ly

Models the System Control Loop Controller Decision Information Sensor Actuator Acts on the System Measures the System Action Sasha Czarkowski COPYRIGHT 2024. ALL RIGHTS RESERVED. www.ergonautic.ly

Improving Productivity: Controller

Better Information - Better Decisions - Better Actions Decision Information Actuator Sensor Action Sasha Czarkowski COPYRIGHT 2024. ALL RIGHTS RESERVED. Ergonautic

It is control loops all the way down Sasha Czarkowski COPYRIGHT 2024. ALL RIGHTS RESERVED. Ergonautic

Creator to Curator Sasha Czarkowski COPYRIGHT 2024. ALL RIGHTS RESERVED. Ergonautic

Controller You are the loop! Decision Information Actuator Sensor Action Sasha Czarkowski COPYRIGHT 2024. ALL RIGHTS RESERVED. Ergonautic

Task Decomposition @ergonauticly COPYRIGHT 2024. ALL RIGHTS RESERVED. www.ergonautic.ly

July 23 2024 Sasha Czarkowski COPYRIGHT 2024. ALL RIGHTS RESERVED. www.ergonautic.ly

Humans are better at understanding complex tasks after they are decomposed into simpler subtasks Sasha Czarkowski COPYRIGHT 2024. ALL RIGHTS RESERVED. Ergonautic

Human-Designed Problem GenAI Proposal GenAI Decomposition Human Repair Unit Test Evaluation Initial Experts Coding Problem GenAI Proposed Solution Heuristic Vanilla Solution Evaluation “NonExperts” Assistive COPYRIGHT 2024. ALL RIGHTS RESERVED. Ergonautic

Initial Decomposition Model M solving problem P, no explicit decomposition Sasha Czarkowski COPYRIGHT 2024 ALL RIGHTS RESERVED. Ergonautic

Heuristic Decomposition Cyclomatic complexity: decomposed if-statements and for/while loops into separate functions Sasha Czarkowski COPYRIGHT 2024 ALL RIGHTS RESERVED. Ergonautic

Vanilla Decomposition Model M decomposing problem P using basic demonstrations Sasha Czarkowski COPYRIGHT 2024 ALL RIGHTS RESERVED. Ergonautic

Assistive Value (AssistV) Measures the feasibility and speed of humans’ ability to repair the decomposed solution Sasha Czarkowski COPYRIGHT 2024 ALL RIGHTS RESERVED. Ergonautic

Assistive Decomposition High Assistive Value decomposition created through an iterative process involving human and LLM decomposition refinement Sasha Czarkowski COPYRIGHT 2024 ALL RIGHTS RESERVED. Ergonautic

Human-Designed Problem GenAI Proposal GenAI Decomposition Human Repair Unit Test Evaluation Initial Experts Coding Problem GenAI Proposed Solution Heuristic Vanilla Solution Evaluation “NonExperts” Assistive COPYRIGHT 2024. ALL RIGHTS RESERVED. Ergonautic

Source: Learning Task Decomposition to Assist Humans in Competitive Programming Sasha Czarkowski COPYRIGHT 2024. ALL RIGHTS RESERVED. Ergonautic

Assistive Decomposition Effects ● ● ● ● Accuracy improved by ~30% for both groups Had more benefits on hard problems Improved LLM ability to supervise itself Was preferred by human participants Sasha Czarkowski COPYRIGHT 2024. ALL RIGHTS RESERVED. Ergonautic

Controller Experts, “non-experts”, and LLMs are better at “repairing” the well-decomposed LLM proposal Decision Information Actuator Sensor Action Sasha Czarkowski COPYRIGHT 2024. ALL RIGHTS RESERVED. Ergonautic

Productivity @ergonauticly COPYRIGHT 2024. ALL RIGHTS RESERVED. www.ergonautic.ly

Generative AI enables us to create MORE - More code - More text - More media Sasha Czarkowski COPYRIGHT 2024 ALL RIGHTS RESERVED. Ergonautic

If you ask for more [X] You will get more [X] Sasha Czarkowski COPYRIGHT 2024. ALL RIGHTS RESERVED. Ergonautic

“What I’m telling you is, productivity is meaningless unless you know what your goal is” Sasha Czarkowski COPYRIGHT 2024. ALL RIGHTS RESERVED. Ergonautic

Source: https://www.software.com/reports/code-time-report

https://xkcd.com/303/ COPILOTING

Better Metrics @ergonauticly COPYRIGHT 2024. ALL RIGHTS RESERVED. www.ergonautic.ly

COPYRIGHT 2023. ALL RIGHTS RESERVED.

Process Quality Creativity Reuse Speed

Mission / Market Fit Sasha Czarkowski COPYRIGHT 2024. ALL RIGHTS RESERVED. Ergonautic

Market Fit - Absolute Revenue - Relative Revenue - Cash Flow Sasha Czarkowski COPYRIGHT 2024 ALL RIGHTS RESERVED. Ergonautic

Market Fit - Customer Lifetime Value - Customer Acquisition Cost - Churn Sasha Czarkowski COPYRIGHT 2024 ALL RIGHTS RESERVED. Ergonautic

Operability Sasha Czarkowski COPYRIGHT 2024. ALL RIGHTS RESERVED. Ergonautic

Operability - Production Readiness - Reliability - Observability - Security Sasha Czarkowski COPYRIGHT 2024 ALL RIGHTS RESERVED. Ergonautic

Improving Productivity: Controller

Better Information - Better Decisions - Better Actions Decision Information Actuator Sensor Action Sasha Czarkowski COPYRIGHT 2024. ALL RIGHTS RESERVED. Ergonautic

What’s the action? What’s the action?

Product workflow @ergonauticly COPYRIGHT 2024. ALL RIGHTS RESERVED. www.ergonautic.ly

New Paradigms @ergonauticly COPYRIGHT 2024. ALL RIGHTS RESERVED. www.ergonautic.ly

Non-Deterministic Software Sasha Czarkowski COPYRIGHT 2024. ALL RIGHTS RESERVED. Ergonautic

“Hallucinations” are a Feature Sasha Czarkowski COPYRIGHT 2024. ALL RIGHTS RESERVED. Ergonautic

Catastrophic Mistakes Sasha Czarkowski COPYRIGHT 2024. ALL RIGHTS RESERVED. Ergonautic

Conversational recruiting platform using AI to automate administrative tasks and improve the hiring process

New attack vectors @ergonauticly COPYRIGHT 2024. ALL RIGHTS RESERVED. www.ergonautic.ly

https://xkcd.com/327/

New style of user interactions @ergonauticly COPYRIGHT 2024. ALL RIGHTS RESERVED. www.ergonautic.ly

Non-keyword search Sasha Czarkowski COPYRIGHT 2024. ALL RIGHTS RESERVED. Ergonautic

If you remember nothing else: Sasha Czarkowski COPYRIGHT 2024. ALL RIGHTS RESERVED. Ergonautic

All systems are socio-technical Sasha Czarkowski COPYRIGHT 2024. ALL RIGHTS RESERVED. Ergonautic

Improving Productivity: Controller

Better Information - Better Decisions - Better Actions Decision Information Actuator Sensor Action Sasha Czarkowski COPYRIGHT 2024. ALL RIGHTS RESERVED. Ergonautic

Thank you! Sasha Czarkowski (Rosenbaum) COPYRIGHT 2024. ALL RIGHTS RESERVED.

From Hype to Help: Intelligent GenAI Adoption Sasha Czarkowski (Rosenbaum) @DivineOps COPYRIGHT 2024. ALL RIGHTS RESERVED.