mufassall

Cheat sheet · Data & storage

File uploads

13 min · Intermediate · mufassall.dtwin.uz/en/l/files/

← Open the lesson

The problem

A video app where phones upload 180 MB files and thousands watch them. Why bytes should not pass through your API, and how presigned URLs, multipart upload, a worker and a CDN share the job.

What you learn

  • Why uploads through the app server are slow and fill its disk
  • Presigned URLs: the API signs, the phone uploads straight to storage
  • Multipart upload: resume a failed part instead of the whole file
  • Thumbnails and transcoding in a worker, delivery through a CDN, only metadata in the database
Final scene
12%20%40%60%

Diagram as code

The lesson's final design as text: paste it into Jira, GitLab, GitHub, Notion or your docs, where Mermaid renders by itself. There are D2 and PlantUML (Confluence, IntelliJ) versions too.

Mermaid: flowchart

flowchart LR
  n_phone(["Phone app"])
  n_api["API"]
  n_bucket[("Object storage")]
  n_db[("Database")]
  n_queue[["Queue"]]
  n_worker("Media worker")
  n_viewer(["Viewers"])
  n_cdn[/"CDN"\]
  n_phone --> n_api
  n_phone --> n_bucket
  n_api --> n_db
  n_bucket --> n_queue
  n_queue --> n_worker
  n_worker --> n_bucket
  n_worker --> n_db
  n_viewer --> n_cdn
  n_cdn --> n_bucket

Mermaid: request sequence

sequenceDiagram
  participant n_phone as Phone app
  participant n_api as API
  participant n_disk as API disk
  participant n_bucket as Object storage
  participant n_db as Database
  participant n_queue as Queue
  participant n_viewer as Viewers
  participant n_cdn as CDN
  Note over n_phone,n_api: Every byte goes through the API
  n_phone->>n_api: POST /videos HTTP/1.1
  n_api-->>n_phone: HTTP/1.1 201 Created
  Note over n_api,n_disk: The disk fills up, the other server has no file
  n_api-xn_disk: write /var/app/uploads/v_9f3.mp4
  Note over n_phone,n_api: The API signs, the phone uploads
  n_phone->>n_api: POST /uploads HTTP/1.1
  n_api-->>n_phone: HTTP/1.1 200 OK
  Note over n_phone,n_bucket: Mobile network: 80% done, start again
  n_phone-xn_bucket: PUT /raw/v_9f3.mp4 HTTP/1.1
  Note over n_phone,n_bucket: Multipart: the file in 8 MB parts
  n_phone->>n_bucket: PUT /raw/v_9f3.mp4?uploadId=up_31a#38;partNumber=17 HTTP/1.1
  n_bucket-->>n_phone: HTTP/1.1 200 OK
  Note over n_api,n_db: In the database: a row, not the file
  n_api->>n_db: INSERT INTO videos
  n_db-->>n_api: INSERT 0 1
  Note over n_api,n_bucket: Transcoding inside the request
  n_api->>n_bucket: api-1: transcode v_9f3 to 720p started
  Note over n_bucket,n_queue: Storage event, queue, worker
  n_bucket->>n_queue: PUBLISH media.uploaded
  Note over n_viewer,n_bucket: A video goes viral
  n_viewer->>n_bucket: GET /v/v_9f3/720p.mp4 HTTP/1.1
  n_bucket-->>n_viewer: HTTP/1.1 200 OK
  Note over n_viewer,n_cdn: A CDN in front of storage
  n_viewer->>n_cdn: GET /v/v_9f3/720p.mp4 HTTP/1.1
  n_cdn-->>n_viewer: HTTP/1.1 206 Partial Content

D2

direction: right

n_phone: "Phone app" {
  shape: rectangle
}
n_api: "API" {
  shape: rectangle
}
n_bucket: "Object storage" {
  shape: stored_data
}
n_db: "Database" {
  shape: cylinder
}
n_queue: "Queue" {
  shape: queue
}
n_worker: "Media worker" {
  shape: rectangle
  style.multiple: true
}
n_viewer: "Viewers" {
  shape: rectangle
}
n_cdn: "CDN" {
  shape: oval
}

n_phone -> n_api
n_phone -> n_bucket
n_api -> n_db
n_bucket -> n_queue
n_queue -> n_worker
n_worker -> n_bucket
n_worker -> n_db
n_viewer -> n_cdn
n_cdn -> n_bucket

PlantUML: request sequence

@startuml
participant "Phone app" as n_phone
participant "API" as n_api
database "API disk" as n_disk
database "Object storage" as n_bucket
database "Database" as n_db
queue "Queue" as n_queue
participant "Viewers" as n_viewer
participant "CDN" as n_cdn
note over n_phone, n_api : Every byte goes through the API
n_phone -> n_api : POST /videos HTTP/1.1
n_api --> n_phone : HTTP/1.1 201 Created
note over n_api, n_disk : The disk fills up, the other server has no file
n_api ->x n_disk : [lost] write /var/app/uploads/v_9f3.mp4
note over n_phone, n_api : The API signs, the phone uploads
n_phone -> n_api : POST /uploads HTTP/1.1
n_api --> n_phone : HTTP/1.1 200 OK
note over n_phone, n_bucket : Mobile network: 80% done, start again
n_phone ->x n_bucket : [lost] PUT /raw/v_9f3.mp4 HTTP/1.1
note over n_phone, n_bucket : Multipart: the file in 8 MB parts
n_phone -> n_bucket : PUT /raw/v_9f3.mp4?uploadId=up_31a&partNumber=17 HTTP/1.1
n_bucket --> n_phone : HTTP/1.1 200 OK
note over n_api, n_db : In the database: a row, not the file
n_api -> n_db : INSERT INTO videos
n_db --> n_api : INSERT 0 1
note over n_api, n_bucket : Transcoding inside the request
n_api -> n_bucket : api-1: transcode v_9f3 to 720p started
note over n_bucket, n_queue : Storage event, queue, worker
n_bucket -> n_queue : PUBLISH media.uploaded
note over n_viewer, n_bucket : A video goes viral
n_viewer -> n_bucket : GET /v/v_9f3/720p.mp4 HTTP/1.1
n_bucket --> n_viewer : HTTP/1.1 200 OK
note over n_viewer, n_cdn : A CDN in front of storage
n_viewer -> n_cdn : GET /v/v_9f3/720p.mp4 HTTP/1.1
n_cdn --> n_viewer : HTTP/1.1 206 Partial Content
@enduml

PlantUML: components

@startuml
left to right direction
agent "Phone app" as n_phone
component "API" as n_api
storage "Object storage" as n_bucket
database "Database" as n_db
queue "Queue" as n_queue
collections "Media worker" as n_worker
agent "Viewers" as n_viewer
frame "CDN" as n_cdn
n_phone --> n_api
n_phone --> n_bucket
n_api --> n_db
n_bucket --> n_queue
n_queue --> n_worker
n_worker --> n_bucket
n_worker --> n_db
n_viewer --> n_cdn
n_cdn --> n_bucket
@enduml

Recap

Five rules for files.

1) Bytes do not go through the API: it signs a short-lived URL and the client uploads to object storage. 2) Large files go in parts that can be retried one by one. 3) The database keeps metadata and the object key, never the file. 4) Heavy processing runs in a worker triggered by an event. 5) Views go through a CDN with long caching for files that never change.

Numbers: at the start and at the end

MetricStartEnd
Upload p9594 s22 s
Failed uploads7%0.4%
Monthly cost1 900 $900 $

Steps

  1. Problem Every byte goes through the API

    A user uploads a 180 MB video from a phone.

  2. Problem The disk fills up, the other server has no file

    Videos pile up on the API's own disk until a write fails.

  3. Fix The API signs, the phone uploads

    Files now live in object storage: a separate service built for large objects, with no disk to fill.

  4. Problem Mobile network: 80% done, start again

    One PUT carries the whole 180 MB.

  5. Fix Multipart: the file in 8 MB parts

    The upload is split into 23 parts of 8 MB.

  6. Case In the database: a row, not the file

    When the upload finishes, the phone tells the API.

  7. Problem Transcoding inside the request

    Viewers need a 720p version and a thumbnail.

  8. Fix Storage event, queue, worker

    Storage now publishes an event when a new object appears.

  9. Problem A video goes viral

    Thousands of viewers open the same video, each fetching it from storage in one region.

  10. Fix A CDN in front of storage

    Viewers now get the video from a CDN: servers in many cities that keep a copy close to the user.

  11. Recap Recap

    Five rules for files.

Terms

Presigned URL
A short-lived link signed by the API that lets the client upload a file straight to object storage. Bytes skip the API, which only checks who the user is and how big the file may be.