Open-Source AI Toolkit for Developers Petyo Dimitrov

About me 17 years in Software Engineering Senior Software Architect in Musala Soft Head of Data & AI Service Offer in Qinshift

Background 2023 2024 Niki Uzunov 3

Agenda 01 Commercial leaders 02 Development use cases & issues 03 Open-source alternatives 04 Trade-offs 05 Next steps 4

This will age like fine… 5

Which AI dev tools do you use? bit.ly/DevAITools 6

Commercial leaders

ChatGPT 8 GitHub Copilot

Development use cases

Autocomplete 10

Explain 11

Improve 12

Generate tests… 13

…and test data 14

Document 15

Expected impact ~25% (std. error ~10%)* The Effects of Generative AI on High Skilled Work: Evidence from Three Field Experiments with Software Developers (Sep 2024) 16

Issues

Sensitive data 18

Lack of control 19

Open-source alternatives Ollama Continue.dev Open WebUI

Ollama

Docker for LLMs

Specifics Supported on Linux, MacOS & Windows Works with GPU and CPU Loads and unloads models dynamically Related to llama.cpp, llamafile, vLLM, etc. 23

Resource requirements Parameters Via CPU (RAM) Via GPU (VRAM) GPU card 3B 8GB 4-6GB 7B 16GB 6GB 13B 32GB 10-12GB RTX 3060/3080 20B 64GB 16GB RTX 3090, A100 65B+ 128GB 40GB dual RTX 3090, A100 RTX 2060

  • Apple M1 chips with 16GB RAM handle up to ~13B parameters

LLM models Autocomplete: Chat: • codestral:22b • Llama 3.1 405B • llama3:8b • DeepSeek Coder 2 16B • deepseek-coder:6.7b • Llama 3.1 8B • starcoder2:3B • deepseek-coder:1.3b https://evalplus.github.io/leaderboard.html https://aider.chat/docs/leaderboards/#llmcode-editing-skill-by-model-release-date

Continue.dev

27

Specifics Supported for VS Code and IntelliJ Works with local and cloud LLMs Features: autocomplete, chat, edit, shortcuts, indexing workspace 28

29

Open WebUI

Specifics ChatGPT-like UI Runs via Docker Supports chat, multi-modality, RAG 31

32

Developer toolkit

Trade-offs Requires GPU resource Harder to scale (cost) Requires more expertise & training Behind state-of-the-art models 34

Honorable mentions Cursor 35 Aider

Next steps

Next steps Experiment with any of these* Run locally on CPU/GPU Use Groq / MistralAI Use a VM 37

Questions? Tools used by TechTalk’s audience 38

Thanks! Petyo Dimitrov Senior Software Architect, Musala Soft Head of Data & AI, Qinshift petyo.dimitrov@qinshift.com