Using third-party services with the major cloud providers
Looking for a cloud provider that will maintain the low-cost advantage of working with third-party services? They don’t advertise this, but the major Infrastructure-as-a-service (IaaS) providers charge you exorbitantly to get data out of their networks. Why? You can use them as long as you want, but you can never leave. It’s been referred to as the “Hotel California effect”, and these are called “egress fees”. By the time you realize how much you’re paying for cloud computing at Amazon Web Services (AWS), Microsoft’s Azure, or Google Cloud Platform (GCP), you have all of your data there, and you’re not going to pay the ransom (so you just stick around).
I currently have a need to send data to Wasabi, an independent storage provider, but can’t justify the cost of getting my data over there from the Big 3. Exporting the data alone would cost more than storing my data. So I need a better option.
If you’d like to mix & match services from various providers, it’s not worth it, because the money you’d be saving by going elsewhere for specialized services is lost on egress fees. It’s anti-competitive, and governments are starting to catch on. And while this has been talked about before, it’s not widely recognized as as problem.
Enter the Bandwidth Alliance
One of the big players, Cloudflare, has been working on this issue through its Bandwidth Alliance (see original announcement). It’s a good initiative, but let’s see how much traction it has over time. It looks like the number of members has doubled in 6 years, but some of agreements are simply deals between Cloudflare and the other party, not across the entire alliance. And I haven’t seen more commitments from the companies to drop these fees entirely. Yes, some of them never charged these fees, which is great, but it would be really nice if ones that did stopped doing so.
So what do we do?
Due diligence: Do your research on a cloud provider before you sign up with them initially. What do they charge? Don’t just look at services you want there; look at how much it’s going to cost you to get your data somewhere else. If they don’t list “egress” fees, look for “network” or “transfer out” fees. If they don’t provide it, ask. Don’t sign up for a cloud provider in the first place if you’re not happy with those terms.
If you do get stuck in one, and want to get out, you’d better hope it’s Google Cloud Platform (GCP). They’re the only one that allows you to get out, albeit permanently. For a good analysis of the offering, check out the Register article. At least it’s something.
Holori has done a comparison of these costs, and provides a tool for spend across clouds. They don’t support an exhaustive list of providers, but hopefully they’re working on this.
I looked into Fly.io because they run containers wherever you want, thinking that they’d support this kind of activity. They focus internationally, right? So you’d think they’d support data portability between cloud providers. Well, guess again: “We bill for outbound data transfer from the region a VM is running in, inbound transfer is free.”
Of course there’s Cloudflare; they’re the ones pushing on this the hardest. They have a “serverless” computing offering called Workers. (I’m putting “serverless” in quotation marks because it’s a silly name. It should really be called “server-managed” because there are still servers; you just don’t have to manage them yourself.) I looked into this option, but I need something that’ll respond to possibly long-running SSH connections, and they don’t seem to be designed for that. What I could really use is a similar offering that works with containers, not just code (sorry, but I’m not rewriting SSHD in JavaScript) that can respond to SSH requests. It’s either that or Kubernetes. And while they support services around Kubernetes, they don’t offer it themselves. (Cloudflare: If you’re reading this, please add support for server-managed containers and Kubernetes!)
My last hope was one of the OpenStack providers. If they’re into open infrastructure, they’d certainly be into data portability. No dice. My favourite Canadian OpenStack provider, Vexxhost, charges $0.10/GB (or $100/TB). I looked into the other ones as well, but none of them advertised this feature so I just assumed it wasn’t available. If I were them, I’d be promoting it as a selling point.
Conclusion
In the end, I landed on Scaleway for the project I’m currenlty working on. They’ve never charged for this, and actually promote it in their marketing.
Scaleway believes in unmetered bandwidth and is committed to working with the Bandwidth Alliance on advocating for all-unmetered plans on customer connections (Source).
They’re a good choice in this case because they support the Bandwidth Alliance, offer server-managed containers and Kubernetes (even an option that works with other cloud providers), and have a Terraform provider to automate all infrastructure as code (IaC).
I would have loved working with one of the big three to get deeper experience with them, but the cost, for this project anyway, simply isn’t justifiable.
Update 2024-10-05: I no longer recommend Scaleway because their Kubernetes cluster spin-up time is very long and they’re significantly more expensive than some other good options. See A Comparison of Managed Kubernetes Providers Without Egress Fees for details.