Storage Unpacked 253 – Building Virtual SANs in the Cloud (Revisited)

Storage Unpacked Podcast

English - November 03, 2023 12:00 - 51 minutes - 70.4 MB - ★★★★★ - 21 ratings
Technology Homepage Download Apple Podcasts Google Podcasts Overcast Castro Pocket Casts RSS feed

Previous Episode: Storage Unpacked 252 – A Vision of Storage Future with Coz from Pure Storage

Next Episode: Storage Unpacked 254 – Announcing VSP One and Hitachi Vantara Reorganisation with Gary Lyng

In this episode, Chris chats with Abel Gordon, Chief System Architect at Lightbits Labs, discussing the challenges and benefits of building a virtual storage area network (SAN) on public cloud infrastructure. Lightbits originally developed the NVMe/TCP protocol and uses this feature to build virtual SANs using public cloud instances. This is a topic we first looked at in episode #210, so it’s good to get a practitioner’s experience.

Modern public cloud now features fast networking, low-latency NVMe and high-performance virtual and physical instances. Unfortunately, NVMe devices are ephemeral and any provisioned storage in the cloud is charged at full capacity. For users of on-premises SANs, the lack of thin provisioning may be an unwelcome surprise.

Why build a virtual SAN, other than to save storage costs? There’s a lot more involved, including delivering resiliency, scalability, targeted performance and capacity. Abel discusses the benefits, then goes on to enumerate the challenges involved when building on vendor-owned infrastructure. Finally, the discussion moves on to how Lightbits’ software is deployed and operated, including the managed application capability in Microsoft Azure.

For more information on Lightbits Labs, visit the company website at https://www.lightbitslabs.com/

As Abel, suggests you can contact him on LinkedIn or email him directly at [email protected].

Elapsed Time: 00:51:15

Timeline

00:00:00 – Intros
00:02:07 – Why build a virtual SAN in the public cloud
00:04:30 – SANs balance out and fully exploit available performance resources
00:06:36 – Public cloud charges for performance and capacity
00:08:12 – On-premises SANs offered full flexibility to manage all metrics
00:09:25 – Cloud autoscaling combined with software gives much more flexible storage
00:13:27 – The on-demand nature of cloud works well for scaling SANs
00:14:40 – New cloud features – NVMe, fast networking and NVMe/TCP have enabled solutions
00:17:19 – What is NVMe/TCP?
00:20:50 – What challenges are there in delivering a SAN on public cloud instances?
00:24:03 – Cloud providers optimise for their system, not for your application
00:25:02 – What operating system issues exist when building a virtual SAN?
00:28:27 – Userspace operation requires a different programming strategy
00:31:00 – NUMA awareness is essential, even in the public cloud
00:32:46 – Each new instance type requires retesting and validation
00:35:53 – What is the Lightbits solution and how is it deployed?
00:37:00 – NVMe cloud drives are ephemeral
00:41:35 – Snapshots work differently in the public cloud
00:44:12 – Is Lightbits dedicated or HCI?
00:46:32 – How is the solution consumed?
00:47:29 – Azure offers management application capability
00:50:36 – Wrap up

Related Podcasts & Blogs

Is the Public Cloud Becoming More Reliable?
Zesty Optimises AWS EC2 EBS Storage
Storage QoS In The Cloud
#97 – Building Storage Using NVMe/TCP with Kam Eshghi from Lightbits Labs
#121 – NVMe 1.4 Deep Dive Part II with J Metz
#210 – Building SANs in the Cloud