Cheat sheet · Data & storage
File uploads
The problem
A video app where phones upload 180 MB files and thousands watch them. Why bytes should not pass through your API, and how presigned URLs, multipart upload, a worker and a CDN share the job.
What you learn
- Why uploads through the app server are slow and fill its disk
- Presigned URLs: the API signs, the phone uploads straight to storage
- Multipart upload: resume a failed part instead of the whole file
- Thumbnails and transcoding in a worker, delivery through a CDN, only metadata in the database
Diagram as code
The lesson's final design as text: paste it into Jira, GitLab, GitHub, Notion or your docs, where Mermaid renders by itself. There are D2 and PlantUML (Confluence, IntelliJ) versions too.
Mermaid: flowchart
flowchart LR
n_phone(["Phone app"])
n_api["API"]
n_bucket[("Object storage")]
n_db[("Database")]
n_queue[["Queue"]]
n_worker("Media worker")
n_viewer(["Viewers"])
n_cdn[/"CDN"\]
n_phone --> n_api
n_phone --> n_bucket
n_api --> n_db
n_bucket --> n_queue
n_queue --> n_worker
n_worker --> n_bucket
n_worker --> n_db
n_viewer --> n_cdn
n_cdn --> n_bucketMermaid: request sequence
sequenceDiagram
participant n_phone as Phone app
participant n_api as API
participant n_disk as API disk
participant n_bucket as Object storage
participant n_db as Database
participant n_queue as Queue
participant n_viewer as Viewers
participant n_cdn as CDN
Note over n_phone,n_api: Every byte goes through the API
n_phone->>n_api: POST /videos HTTP/1.1
n_api-->>n_phone: HTTP/1.1 201 Created
Note over n_api,n_disk: The disk fills up, the other server has no file
n_api-xn_disk: write /var/app/uploads/v_9f3.mp4
Note over n_phone,n_api: The API signs, the phone uploads
n_phone->>n_api: POST /uploads HTTP/1.1
n_api-->>n_phone: HTTP/1.1 200 OK
Note over n_phone,n_bucket: Mobile network: 80% done, start again
n_phone-xn_bucket: PUT /raw/v_9f3.mp4 HTTP/1.1
Note over n_phone,n_bucket: Multipart: the file in 8 MB parts
n_phone->>n_bucket: PUT /raw/v_9f3.mp4?uploadId=up_31a#38;partNumber=17 HTTP/1.1
n_bucket-->>n_phone: HTTP/1.1 200 OK
Note over n_api,n_db: In the database: a row, not the file
n_api->>n_db: INSERT INTO videos
n_db-->>n_api: INSERT 0 1
Note over n_api,n_bucket: Transcoding inside the request
n_api->>n_bucket: api-1: transcode v_9f3 to 720p started
Note over n_bucket,n_queue: Storage event, queue, worker
n_bucket->>n_queue: PUBLISH media.uploaded
Note over n_viewer,n_bucket: A video goes viral
n_viewer->>n_bucket: GET /v/v_9f3/720p.mp4 HTTP/1.1
n_bucket-->>n_viewer: HTTP/1.1 200 OK
Note over n_viewer,n_cdn: A CDN in front of storage
n_viewer->>n_cdn: GET /v/v_9f3/720p.mp4 HTTP/1.1
n_cdn-->>n_viewer: HTTP/1.1 206 Partial ContentD2
direction: right
n_phone: "Phone app" {
shape: rectangle
}
n_api: "API" {
shape: rectangle
}
n_bucket: "Object storage" {
shape: stored_data
}
n_db: "Database" {
shape: cylinder
}
n_queue: "Queue" {
shape: queue
}
n_worker: "Media worker" {
shape: rectangle
style.multiple: true
}
n_viewer: "Viewers" {
shape: rectangle
}
n_cdn: "CDN" {
shape: oval
}
n_phone -> n_api
n_phone -> n_bucket
n_api -> n_db
n_bucket -> n_queue
n_queue -> n_worker
n_worker -> n_bucket
n_worker -> n_db
n_viewer -> n_cdn
n_cdn -> n_bucketPlantUML: request sequence
@startuml
participant "Phone app" as n_phone
participant "API" as n_api
database "API disk" as n_disk
database "Object storage" as n_bucket
database "Database" as n_db
queue "Queue" as n_queue
participant "Viewers" as n_viewer
participant "CDN" as n_cdn
note over n_phone, n_api : Every byte goes through the API
n_phone -> n_api : POST /videos HTTP/1.1
n_api --> n_phone : HTTP/1.1 201 Created
note over n_api, n_disk : The disk fills up, the other server has no file
n_api ->x n_disk : [lost] write /var/app/uploads/v_9f3.mp4
note over n_phone, n_api : The API signs, the phone uploads
n_phone -> n_api : POST /uploads HTTP/1.1
n_api --> n_phone : HTTP/1.1 200 OK
note over n_phone, n_bucket : Mobile network: 80% done, start again
n_phone ->x n_bucket : [lost] PUT /raw/v_9f3.mp4 HTTP/1.1
note over n_phone, n_bucket : Multipart: the file in 8 MB parts
n_phone -> n_bucket : PUT /raw/v_9f3.mp4?uploadId=up_31a&partNumber=17 HTTP/1.1
n_bucket --> n_phone : HTTP/1.1 200 OK
note over n_api, n_db : In the database: a row, not the file
n_api -> n_db : INSERT INTO videos
n_db --> n_api : INSERT 0 1
note over n_api, n_bucket : Transcoding inside the request
n_api -> n_bucket : api-1: transcode v_9f3 to 720p started
note over n_bucket, n_queue : Storage event, queue, worker
n_bucket -> n_queue : PUBLISH media.uploaded
note over n_viewer, n_bucket : A video goes viral
n_viewer -> n_bucket : GET /v/v_9f3/720p.mp4 HTTP/1.1
n_bucket --> n_viewer : HTTP/1.1 200 OK
note over n_viewer, n_cdn : A CDN in front of storage
n_viewer -> n_cdn : GET /v/v_9f3/720p.mp4 HTTP/1.1
n_cdn --> n_viewer : HTTP/1.1 206 Partial Content
@endumlPlantUML: components
@startuml
left to right direction
agent "Phone app" as n_phone
component "API" as n_api
storage "Object storage" as n_bucket
database "Database" as n_db
queue "Queue" as n_queue
collections "Media worker" as n_worker
agent "Viewers" as n_viewer
frame "CDN" as n_cdn
n_phone --> n_api
n_phone --> n_bucket
n_api --> n_db
n_bucket --> n_queue
n_queue --> n_worker
n_worker --> n_bucket
n_worker --> n_db
n_viewer --> n_cdn
n_cdn --> n_bucket
@endumlRecap
Five rules for files.
1) Bytes do not go through the API: it signs a short-lived URL and the client uploads to object storage. 2) Large files go in parts that can be retried one by one. 3) The database keeps metadata and the object key, never the file. 4) Heavy processing runs in a worker triggered by an event. 5) Views go through a CDN with long caching for files that never change.
Numbers: at the start and at the end
| Metric | Start | End |
|---|---|---|
| Upload p95 | 94 s | 22 s |
| Failed uploads | 7% | 0.4% |
| Monthly cost | 1 900 $ | 900 $ |
Steps
- Problem Every byte goes through the API
A user uploads a 180 MB video from a phone.
- Problem The disk fills up, the other server has no file
Videos pile up on the API's own disk until a write fails.
- Fix The API signs, the phone uploads
Files now live in object storage: a separate service built for large objects, with no disk to fill.
- Problem Mobile network: 80% done, start again
One
PUTcarries the whole 180 MB. - Fix Multipart: the file in 8 MB parts
The upload is split into 23 parts of 8 MB.
- Case In the database: a row, not the file
When the upload finishes, the phone tells the API.
- Problem Transcoding inside the request
Viewers need a 720p version and a thumbnail.
- Fix Storage event, queue, worker
Storage now publishes an event when a new object appears.
- Problem A video goes viral
Thousands of viewers open the same video, each fetching it from storage in one region.
- Fix A CDN in front of storage
Viewers now get the video from a CDN: servers in many cities that keep a copy close to the user.
- Recap Recap
Five rules for files.
Terms
- Presigned URL
- A short-lived link signed by the API that lets the client upload a file straight to object storage. Bytes skip the API, which only checks who the user is and how big the file may be.