101: Replicating SQLite using Litestream with Ben Johnson

Ben Johnson explains his project Litestream.io, an OpenSource tool that replicates SQLite databases to remote servers and to backup locations like S3 for durability. We talk about how moving data out to the user creates true edge applications. We discuss what types of problems this helps solve, the architectures that become possible, and how a globally distributed Phoenix application could use this. He shares how Fly.io acquired the project and brought him on full-time to continue his work on it. Fascinating discussion that challenges many of the assumptions about how we've been building “web” systems for years.
Show Notes online - http://podcast.thinkingelixir.com/101 (http://podcast.thinkingelixir.com/101)
Elixir Community News
- https://www.erlang.org/blog/my-otp-25-highlights/ (https://www.erlang.org/blog/my-otp-25-highlights/) – OTP 25 was officially released
- https://www.erlang.org/eeps/eep-0049 (https://www.erlang.org/eeps/eep-0049) – EEP document discussing the “maybe” expression
- https://www.erlang.org/doc/reference_manual/expressions.html#maybe (https://www.erlang.org/doc/reference_manual/expressions.html#maybe) – Official docs for “maybe” feature
- https://www.erlang.org/blog/faster-rand/ (https://www.erlang.org/blog/faster-rand/) – A new fast Pseudo Random Generator
- https://jobs.ericsson.com/job/Stockholm-Open-SourceErlang-Developer-Stoc/746811902/ (https://jobs.ericsson.com/job/Stockholm-Open-SourceErlang-Developer-Stoc/746811902/) – Ericsson, the company behind Erlang, is hiring for an OpenSource developer to join the Erlang team.
- https://www.bbc.com/news/business-61562651 (https://www.bbc.com/news/business-61562651) – Klarna layoffs
- https://hexdocs.pm/elixir/main/PartitionSupervisor.html (https://hexdocs.pm/elixir/main/PartitionSupervisor.html) – PartitionManager feature coming in Elixir v1.14 has docs online so you can learn about it early.
- https://twitter.com/DNAutics/status/1528434291872505856 (https://twitter.com/DNAutics/status/1528434291872505856) – Isaac Yonemoto made the JavaScript engine in Zig accessible through Zigler and therefore Elixir
- https://podcast.thinkingelixir.com/98 (https://podcast.thinkingelixir.com/98) – Follow-up from episode 98 with Dominic Letz about writing an Elixir application that runs on mobile devices and can be installed from the iOS AppStore.
- https://ionicframework.com/ (https://ionicframework.com/)
- https://twitter.com/ElixirConf/status/1526654041626923008 (https://twitter.com/ElixirConf/status/1526654041626923008) – ElixirConf 2022 in Colorado - CFP is open
- https://2022.elixirconf.com/speaker-cfp (https://2022.elixirconf.com/speaker-cfp) – Where to submit your talk proposal
- https://www.elixirconf.eu/ (https://www.elixirconf.eu/) – ElixirConf EU in London on June 9-10
- https://elixirconf.com/events (https://elixirconf.com/events) – ElixirConf US in Colorado on August 30-Sep2
Do you have some Elixir news to share? Tell us at @ThinkingElixir (https://twitter.com/ThinkingElixir) or email at [email protected] (mailto:[email protected])
Discussion Resources
- https://litestream.io/ (https://litestream.io/)
- https://fly.io/blog/all-in-on-sqlite-litestream/ (https://fly.io/blog/all-in-on-sqlite-litestream/)
- https://github.com/benbjohnson/litestream (https://github.com/benbjohnson/litestream) – Project on Github
- https://sqlite.org/index.html (https://sqlite.org/index.html)
- https://sqlite.org/whentouse.html (https://sqlite.org/whentouse.html) – SQLite guide on "when to use" it
- https://github.com/elixir-sqlite/ecto_sqlite3 (https://github.com/elixir-sqlite/ecto_sqlite3)
- https://github.com/boltdb/bolt (https://github.com/boltdb/bolt)
- https://en.wikipedia.org/wiki/Write-ahead_logging (https://en.wikipedia.org/wiki/Write-ahead_logging)
- https://raft.github.io/ (https://raft.github.io/)
- https://en.wikipedia.org/wiki/Conflict-freereplicateddata_type (https://en.wikipedia.org/wiki/Conflict-free_replicated_data_type)
- https://litestream.io/tips/ (https://litestream.io/tips/) – Page covers tips
- https://www.kernel.org/doc/html/latest/filesystems/fuse.html (https://www.kernel.org/doc/html/latest/filesystems/fuse.html) – Understanding what a FUSE filesystem is
Guest Information
- https://twitter.com/benbjohnson (https://twitter.com/benbjohnson) – on Twitter
- https://github.com/benbjohnson/ (https://github.com/benbjohnson/) – on Github
- https://github.com/benbjohnson/litestream (https://github.com/benbjohnson/litestream) – Project on Github
Find us online
- Message the show - @ThinkingElixir (https://twitter.com/ThinkingElixir)
- Email the show - [email protected] (mailto:[email protected])
- Mark Ericksen - @brainlid (https://twitter.com/brainlid)
- David Bernheisel - @bernheisel (https://twitter.com/bernheisel)
- Cade Ward - @cadebward (https://twitter.com/cadebward)

Ben Johnson explains his project Litestream.io, an OpenSource tool that replicates SQLite databases to remote servers and to backup locations like S3 for durability. We talk about how moving data out to the user creates true edge applications. We discuss what types of problems this helps solve, the architectures that become possible, and how a globally distributed Phoenix application could use this. He shares how Fly.io acquired the project and brought him on full-time to continue his work on it. Fascinating discussion that challenges many of the assumptions about how we've been building “web” systems for years.

Show Notes online - http://podcast.thinkingelixir.com/101

Elixir Community News

https://www.erlang.org/blog/my-otp-25-highlights/ – OTP 25 was officially released
https://www.erlang.org/eeps/eep-0049 – EEP document discussing the “maybe” expression
https://www.erlang.org/doc/reference_manual/expressions.html#maybe – Official docs for “maybe” feature
https://www.erlang.org/blog/faster-rand/ – A new fast Pseudo Random Generator
https://jobs.ericsson.com/job/Stockholm-Open-SourceErlang-Developer-Stoc/746811902/ – Ericsson, the company behind Erlang, is hiring for an OpenSource developer to join the Erlang team.
https://www.bbc.com/news/business-61562651 – Klarna layoffs
https://hexdocs.pm/elixir/main/PartitionSupervisor.html – PartitionManager feature coming in Elixir v1.14 has docs online so you can learn about it early.
https://twitter.com/DNAutics/status/1528434291872505856 – Isaac Yonemoto made the JavaScript engine in Zig accessible through Zigler and therefore Elixir
https://podcast.thinkingelixir.com/98 – Follow-up from episode 98 with Dominic Letz about writing an Elixir application that runs on mobile devices and can be installed from the iOS AppStore.
https://ionicframework.com/
https://twitter.com/ElixirConf/status/1526654041626923008 – ElixirConf 2022 in Colorado - CFP is open
https://2022.elixirconf.com/speaker-cfp – Where to submit your talk proposal
https://www.elixirconf.eu/ – ElixirConf EU in London on June 9-10
https://elixirconf.com/events – ElixirConf US in Colorado on August 30-Sep2

Do you have some Elixir news to share? Tell us at @ThinkingElixir or email at [email protected]

Discussion Resources

https://litestream.io/
https://fly.io/blog/all-in-on-sqlite-litestream/
https://github.com/benbjohnson/litestream – Project on Github
https://sqlite.org/index.html
https://sqlite.org/whentouse.html – SQLite guide on "when to use" it
https://github.com/elixir-sqlite/ecto_sqlite3
https://github.com/boltdb/bolt
https://en.wikipedia.org/wiki/Write-ahead_logging
https://raft.github.io/
https://en.wikipedia.org/wiki/Conflict-free_replicated_data_type
https://litestream.io/tips/ – Page covers tips
https://www.kernel.org/doc/html/latest/filesystems/fuse.html – Understanding what a FUSE filesystem is

Guest Information

https://twitter.com/benbjohnson – on Twitter
https://github.com/benbjohnson/ – on Github
https://github.com/benbjohnson/litestream – Project on Github

Find us online

Message the show - @ThinkingElixir
Email the show - [email protected]
Mark Ericksen - @brainlid
David Bernheisel - @bernheisel
Cade Ward - @cadebward

Sponsored By: