A presentation at Qinshift Tech Talk by Petyo Dimitrov
Open-Source AI Toolkit for Developers Petyo Dimitrov
About me 17 years in Software Engineering Senior Software Architect in Musala Soft Head of Data & AI Service Offer in Qinshift
Background 2023 2024 Niki Uzunov 3
Agenda 01 Commercial leaders 02 Development use cases & issues 03 Open-source alternatives 04 Trade-offs 05 Next steps 4
This will age like fine… 5
Which AI dev tools do you use? bit.ly/DevAITools 6
Commercial leaders
ChatGPT 8 GitHub Copilot
Development use cases
Autocomplete 10
Explain 11
Improve 12
Generate tests… 13
…and test data 14
Document 15
Expected impact ~25% (std. error ~10%)* The Effects of Generative AI on High Skilled Work: Evidence from Three Field Experiments with Software Developers (Sep 2024) 16
Issues
Sensitive data 18
Lack of control 19
Open-source alternatives Ollama Continue.dev Open WebUI
Ollama
Docker for LLMs
Specifics Supported on Linux, MacOS & Windows Works with GPU and CPU Loads and unloads models dynamically Related to llama.cpp, llamafile, vLLM, etc. 23
Resource requirements Parameters Via CPU (RAM) Via GPU (VRAM) GPU card 3B 8GB 4-6GB 7B 16GB 6GB 13B 32GB 10-12GB RTX 3060/3080 20B 64GB 16GB RTX 3090, A100 65B+ 128GB 40GB dual RTX 3090, A100 RTX 2060
LLM models Autocomplete: Chat: • codestral:22b • Llama 3.1 405B • llama3:8b • DeepSeek Coder 2 16B • deepseek-coder:6.7b • Llama 3.1 8B • starcoder2:3B • deepseek-coder:1.3b https://evalplus.github.io/leaderboard.html https://aider.chat/docs/leaderboards/#llmcode-editing-skill-by-model-release-date
Continue.dev
27
Specifics Supported for VS Code and IntelliJ Works with local and cloud LLMs Features: autocomplete, chat, edit, shortcuts, indexing workspace 28
29
Open WebUI
Specifics ChatGPT-like UI Runs via Docker Supports chat, multi-modality, RAG 31
32
Developer toolkit
Trade-offs Requires GPU resource Harder to scale (cost) Requires more expertise & training Behind state-of-the-art models 34
Honorable mentions Cursor 35 Aider
Next steps
Next steps Experiment with any of these* Run locally on CPU/GPU Use Groq / MistralAI Use a VM 37
Questions? Tools used by TechTalk’s audience 38
Thanks! Petyo Dimitrov Senior Software Architect, Musala Soft Head of Data & AI, Qinshift petyo.dimitrov@qinshift.com
Recent research suggests that AI-assisted coding can boost developer productivity by ~20%. Do you need to use a commercial product like GitHub Copilot or ChatGPT to achieve this?
Learn how you can extend your IDE or offer a chat-like interface via existing open-source solutions. Experiment with LLMs locally via Ollama and discover which are the most promising models. Compare this setup to the market leaders and understand the tradeoffs.
Key Takeaways: Productivity gains with popular AI-assisted tools; Overview of open-source AI-assisted tools; Comparison of open source vs. commercial AI models; Benefits and drawbacks.
for free. You
can too.