Open-Source AI Toolkit for Developers

A presentation at Qinshift Tech Talk in January 2025 in by Petyo Dimitrov

Slide 1

Slide 1

Open-Source AI Toolkit for Developers Petyo Dimitrov

Slide 2

Slide 2

About me 17 years in Software Engineering Senior Software Architect in Musala Soft Head of Data & AI Service Offer in Qinshift

Slide 3

Slide 3

Background 2023 2024 Niki Uzunov 3

Slide 4

Slide 4

Agenda 01 Commercial leaders 02 Development use cases & issues 03 Open-source alternatives 04 Trade-offs 05 Next steps 4

Slide 5

Slide 5

This will age like fine… 5

Slide 6

Slide 6

Which AI dev tools do you use? bit.ly/DevAITools 6

Slide 7

Slide 7

Commercial leaders

Slide 8

Slide 8

ChatGPT 8 GitHub Copilot

Slide 9

Slide 9

Development use cases

Slide 10

Slide 10

Autocomplete 10

Slide 11

Slide 11

Explain 11

Slide 12

Slide 12

Improve 12

Slide 13

Slide 13

Generate tests… 13

Slide 14

Slide 14

…and test data 14

Slide 15

Slide 15

Document 15

Slide 16

Slide 16

Expected impact ~25% (std. error ~10%)* The Effects of Generative AI on High Skilled Work: Evidence from Three Field Experiments with Software Developers (Sep 2024) 16

Slide 17

Slide 17

Issues

Slide 18

Slide 18

Sensitive data 18

Slide 19

Slide 19

Lack of control 19

Slide 20

Slide 20

Open-source alternatives Ollama Continue.dev Open WebUI

Slide 21

Slide 21

Ollama

Slide 22

Slide 22

Docker for LLMs

Slide 23

Slide 23

Specifics Supported on Linux, MacOS & Windows Works with GPU and CPU Loads and unloads models dynamically Related to llama.cpp, llamafile, vLLM, etc. 23

Slide 24

Slide 24

Resource requirements Parameters Via CPU (RAM) Via GPU (VRAM) GPU card 3B 8GB 4-6GB 7B 16GB 6GB 13B 32GB 10-12GB RTX 3060/3080 20B 64GB 16GB RTX 3090, A100 65B+ 128GB 40GB dual RTX 3090, A100 RTX 2060

  • Apple M1 chips with 16GB RAM handle up to ~13B parameters

Slide 25

Slide 25

LLM models Autocomplete: Chat: • codestral:22b • Llama 3.1 405B • llama3:8b • DeepSeek Coder 2 16B • deepseek-coder:6.7b • Llama 3.1 8B • starcoder2:3B • deepseek-coder:1.3b https://evalplus.github.io/leaderboard.html https://aider.chat/docs/leaderboards/#llmcode-editing-skill-by-model-release-date

Slide 26

Slide 26

Continue.dev

Slide 27

Slide 27

27

Slide 28

Slide 28

Specifics Supported for VS Code and IntelliJ Works with local and cloud LLMs Features: autocomplete, chat, edit, shortcuts, indexing workspace 28

Slide 29

Slide 29

29

Slide 30

Slide 30

Open WebUI

Slide 31

Slide 31

Specifics ChatGPT-like UI Runs via Docker Supports chat, multi-modality, RAG 31

Slide 32

Slide 32

32

Slide 33

Slide 33

Developer toolkit

Slide 34

Slide 34

Trade-offs Requires GPU resource Harder to scale (cost) Requires more expertise & training Behind state-of-the-art models 34

Slide 35

Slide 35

Honorable mentions Cursor 35 Aider

Slide 36

Slide 36

Next steps

Slide 37

Slide 37

Next steps Experiment with any of these* Run locally on CPU/GPU Use Groq / MistralAI Use a VM 37

Slide 38

Slide 38

Questions? Tools used by TechTalk’s audience 38

Slide 39

Slide 39

Thanks! Petyo Dimitrov Senior Software Architect, Musala Soft Head of Data & AI, Qinshift petyo.dimitrov@qinshift.com