Dropbox Gives You Secure Access to All of Your Files

Running in Production

English - May 10, 2021 04:00 - 1 hour - 63.9 MB - ★★★★★ - 8 ratings
Technology News Tech News deployment elixir flask hosting node rubyonrails phoenix production python techstacks Homepage Download Apple Podcasts Google Podcasts Overcast Castro Pocket Casts RSS feed

Previous Episode: Mito Is a JupyterLab Extension to Make Python Data Analysis Easy

Next Episode: SongRender Lets You Create Audio Visualizer Videos from Audio Clips

Utsav Shah goes over building Dropbox with Pylons, Python, Rust and Go. It's mostly hosted on their own datacenters across 1000+ servers.

In this episode of Running in Production, Utsav Shah goes over building
Dropbox with Pylons, Python, Rust and Go. It’s mostly hosted on their own
data centers across 1,000+ servers. It’s been available since 2008. They have
700+ million users and handle 100k+ requests per second.

Utsav talks about working with hundreds of engineers on a multi-million line
Python based monolithic app, handling payments without Stripe, storing exabytes
of files, using Rust in the desktop client, having remote dev boxes, leveraging
open source tools and tons more.

Topics Include

1:32 – A couple of hundred of developers committing to a monolithic app
2:30 – Motivation for choosing Pylons and Python
4:04 – Their service architecture can be described as a solar system
6:03 – Building their own NoSQL database called Edgestore before MongoDB existed
8:41 – Pylons is a micro-framework for Python
10:29 – Developing a custom payment handling system before Stripe existed
13:43 – How Dropbox stores your files on disk and who can access those files
17:32 – Back in 2014 Dropbox moved away from S3 to their own infrastructure
18:50 – Dealing with exabytes of storage
20:44 – A “multi mono repo” set up with millions of lines of Python
21:57 – The desktop client is a combination of Python and Rust
23:34 – Updating the desktop client code base from Python 2 to Python 3
26:09 – Pytest is being used for tests and Black for code formatting
29:04 – A developer can spin up their own Dropbox stack on a hosted dev server
32:34 – The web UI is mostly server rendered templates and a mix of React
36:46 – There’s a lot going on with MySQL, plus memcached and nginx / Envoy
39:06 – Using open source libraries created by Facebook and YouTube
41:39 – The open source version of nginx is being used a bunch
43:33 – Most things are hosted on their own data centers running Ubuntu
45:32 – There’s dedicated teams that focus on the infrastructure
46:59 – How code gets from a developer’s dev environment to passing in CI
53:46 – Once everything passes, it’s rolled out internally and then incrementally to users
55:13 – Feature flags are being used with a home grown solution
56:06 – Dealing with secrets using something they’ve developed in house
58:01 – Sometimes you end up building out a lot more than your core product
59:03 – Your files are very safely and securely backed up
1:01:16 – Monitoring, alerting, logging and error handling
1:04:09 – Rate limiting is handled at the app level along with memcached
1:05:13 – Best tips? Monoliths aren’t bad if you invest in them
1:06:30 – Think deeply about what you’re developing and focus on the architecture
1:08:55 – Utsav has a podcast at https://www.softwareatscale.dev, check it out!

Links

📄 References

⚙️ Tech Stack

pylons →

python →

golang →

rust →

react →

cassandra →

envoy →

memcached →

mysql →

nginx →

phabricator →

ses →

ubuntu →

webpack →

🛠 Libraries Used

https://pythonpaste.readthedocs.io/en/latest/
https://github.com/python/mypy
https://github.com/pytest-dev/pytest
https://github.com/psf/black
https://github.com/bazelbuild/bazel

Support the Show

This episode does not have a sponsor and this podcast is a labor of love. If
you want to support the show, the best way to do it is to purchase one of my
courses or suggest one to a friend.

Dive into Docker is a video course that takes you from not knowing what Docker is
to being able to confidently use Docker and Docker Compose for your own apps.
Long gone are the days of "but it works on my machine!". A bunch of follow
along labs are included.

Build a SAAS App with Flask is a video course where we build a real
world SAAS app that accepts payments, has a custom admin, includes high test
coverage and goes over how to implement and apply 50+ common web app features.
There's over 20+ hours of video.

Twitter Mentions

@utsav_sha