Episode 185 - Cloud vs Local LLMs: A Developer's Dilemma

In this episode of Two Voice Devs, Allen Firstenberg and Roger Kibbe explore the rising trend of local LLMs, smaller language models designed to run on personal devices instead of relying on cloud-based APIs. They discuss the advantages and disadvantages of this approach, focusing on data privacy, control, cost efficiency, and the unique opportunities it presents for developers. They also delve into the importance of fine-tuning these smaller models for specific tasks, enabling them to excel in areas like legal contract analysis and mobile app development.

The conversation dives into various popular local LLM models, including:

Mistral: Roger's favorite, lauded for its capabilities and ability to run efficiently on smaller machines.
Phi-2: A tiny model from Microsoft ideal for on-device applications.
Llama: Meta's influential model, with Llama 2 currently leading the pack and Llama 3 anticipated to be comparable to ChatGPT 4.
Gemma: Google's new open-source model with potential, but still under evaluation.

Learn more:

Ollama: https://ollama.com/
Ollama source: https://github.com/ollama/ollama
LM Studio: https://lmstudio.ai/

Timestamps:

00:00:00: Introduction and welcome back to Roger Kibbe.

00:01:31: Roger discusses his career path and his passion for voice and AI.

00:06:33: The discussion turns to the larger vs. smaller LLMs.

00:13:52: Understanding key terminology like quantization and fine-tuning.

00:20:58: Roger shares his favorite local LLM models.

00:25:14: Discussing the strengths and weaknesses of smaller models like Gemma.

00:30:32: Exploring the benefits and challenges of running LLMs locally.

00:39:15: The value of local LLMs for developers and individual learning.

00:40:29: The impact of local LLMs on mobile devices and app development.

00:49:27: Closing thoughts and call for audience feedback.

Join Allen and Roger as they explore the exciting potential of local LLMs and how they might revolutionize the development landscape!