Building a real-time cross-device cloud storage solution

Bridgeway is a clone of Space, a cloud storage solution that lets you upload files to the cloud, and open or edit them on another machine "instantly"₁.

#Background

A couple weeks ago, I came across a post on X about giving your computer infinite storage through this product called SkyGlass₂. The original post can be found here by @byjasonz.

The demo video looked like complete magic to me and I also ran into this problem when I had to transfer a giant folder from macOS to Windows which took a painful amount of time so I challenged myself to rebuild it for all platforms to use it myself and to learn how the tech works.

Some of the requirements I wanted in the recreation were:

• Native Mount: Files should be openable through file explorer.
• Zero Local Allocation: The physical size of files in the file explorer stays at 0. Opening files in an app only streams the parts the app reads.
• Object Storage Backing: Cloud storage is the source of truth, and GETs only happen if an app reads a file. The folders in Bridgeway are built from List and Head requests.
• Cross-Device Over Shared Metadata: If device A uploads a file, device B shows it as available almost instantly through shared metadata.
• Progressive Upload: You can open files while they are still uploading, and priority chunk uploads are supported when a peer tries to read a missing range.

#Architecture

I won't go too in-depth on the technical details since a friend of mine invested in Space, but everything is possible₃ in the Apple (macOS and iOS), Windows, and Android ecosystem but on the high level, it looks something like this:

The one liner of how this works is it separates the file's metadata from its bytes. Consider this example:

When you add a file, Bridgeway sends its metadata: the file name, its path, and its logical size, so other devices see the file before its chunks finish uploading. Then your computer shows the file right away with its real size, but says on disk it's 0. When another app reads a file, it'll read by offset and length (like opening a video or skipping through it). It pretty much means the read engine only fetches the specific byte range needed right then via range requests₄, instead of downloading the whole file. For chunk sizes I chose 8 MiB.

#Why 8 MiB chunks

Choosing a chunk size that is a power of two keeps chunk indexing simple. Since 8 MiB is 2²³ bytes (8,388,608 bytes), mapping any file offset to its chunk index can just be a bit shift (offset >> 23) and getting the offset within that chunk is a bitwise mask (offset & 0x7FFFFF), instead of doing division and modulo math. I don't believe it matters for performance reasons, but it does simplify some of the logic₅.

As for why 8, I found it had the best trade-off for what I wanted, that being network overhead and upload responsiveness. Hypothetically, if you picked like 1 MiB, uploading large files will spam the cloud storage with thousands of separate chunk objects. If you chose something like 64 MiB, committing each chunk takes much longer before another device can start reading it. If you also lose connection, you have to retry that entire 64 MiB piece instead of just an 8 MiB one. Now as for 4 vs 8 vs 16, I found 8 kept the playback the most snappy in my testing. 4 also uses twice the number of upload requests and because I was using R2 in my testing, I didn't want to hammer it since it charges for requests.

#How files show up without downloading

The hardest part about this entire process was finding a way to show a file on your computer without saving it to your hard drive first.

Normal cloud storage options (like Dropbox or Google Drive) don't do this. If you were trying to download a 10 GB file to check a 60-second clip, the machine you're using will download all 10 GB in the background.

I had two plausible ideas that could achieve this behaviour:

1. Empty placeholder files but on open, it would open the original file, not the placeholder taking up 0 bytes but claiming to be the original file size

2. A known file-extension provider for each operating system

My first attempt using the empty placeholder files didn't work. While you can create a file that claims to be 10 GB while using zero storage, clicking play just feeds blank empty data to your video player. A normal placeholder file has no way to pause the player and pull the missing data from the internet.

The file-extension method worked and the results passed my requirements₆:

• On desktops, it lets Bridgeway sit quietly between your file manager and the rest of the computer. Whenever an app opens a file, Bridgeway only grabs that part from the cloud, and reports the file as 0 bytes on disk. Anything cached is in a different folder.
• On mobile, it streams file previews directly inside Bridgeway, and does the same as desktop by fetching only the needed parts.

#Opening files while they upload

If you uploaded a huge 20 GB file, you should not have to wait 30 minutes for the upload to finish just to start watching it.

1. Instant appearance: As machine A starts uploading, it tells the cloud the file name and size so machine B sees the file pop up in Bridgeway "right away".
2. Watching in progress: Uploaded chunks can be played as long as your player has already read whatever container headers it needs to start.
3. Smart priority: If you jump ahead to minute 30 of a video and that part is not uploaded yet, Bridgeway sends a ping to the uploading computer to prioritize that part. Then the uploader moves that piece to the front of its queue and sends it as soon as a slot (in the queue) frees up, while the chunks already in flight finish normally. Buffering can still happen if that piece hasn't reached the cloud yet obviously₇, but it gets prioritized over the rest of the file.

#Keeping things fast

• Local cache: Any part of a file you fetch gets saved to a local folder (or memory on mobile). If you replay that part, it reads from the cache instead of pulling from the cloud again, until older cache entries get cleared.
• Instant folder browsing: Clicking through folders or checking file info reads from a local list on your computer so it doesn't have to wait on network calls just to display folder contents.
• Spotlight search: A keyboard shortcut brings up a search bar to instantly find any file across your currently selected space using its local file index.

#Limitations

There are a couple limitations to this, the first one being your WiFi download and upload speeds. If your download speed is bad, streaming bytes on demand is going to buffer. And if your upload speed is bad, it will take a while for chunks to reach the cloud so another machine can actually read them. But that's out of scope for this project anyway.

Some of the other limitations like the chunk upload queues and metadata syncing also add a bit of delay. For the chunk upload queues, if a device has chunks uploading and an app needs a specific chunk, it has to wait a little bit before the uploading device can send it. The metadata sync also adds delay between machines. If your computer hasn't received the newest file list yet, new files won't pop up right away no matter how fast your internet is. But these two are generally negligible compared to slow WiFi speeds.