Cheat sheet · Whole systems
Design a video streaming site
The problem
Creators upload hour-long 4K videos, viewers watch on phones over shaky mobile links, and a premiere brings 200 000 people at once. A video site built step by step: resumable uploads, a transcoding pipeline, HLS with adaptive bitrate, a CDN, view counting and cold storage.
What you learn
- Why a big upload goes straight to storage in resumable parts
- Transcoding into a bitrate ladder, in parallel chunks
- HLS segments and playlists, and how the player switches quality
- CDN in front of the origin, counting views without a hot row, hot and cold storage
Diagram as code
The lesson's final design as text: paste it into Jira, GitLab, GitHub, Notion or your docs, where Mermaid renders by itself. There are D2 and PlantUML (Confluence, IntelliJ) versions too.
Mermaid: flowchart
flowchart LR
n_creator(["Creator"])
n_upload["Upload API"]
n_raw[("Raw uploads")]
n_viewer(["Viewer"])
n_queue[["Transcode queue"]]
n_workers("Transcoders")
n_media[("Video storage")]
n_cdn[/"CDN"\]
n_counter["View counter"]
n_videos[("Videos DB")]
n_cold[("Cold storage")]
n_creator --> n_upload
n_creator --> n_raw
n_raw --> n_queue
n_queue --> n_workers
n_workers --> n_media
n_viewer --> n_cdn
n_cdn --> n_media
n_viewer --> n_counter
n_counter --> n_videos
n_media --> n_coldMermaid: request sequence
sequenceDiagram
participant n_creator as Creator
participant n_upload as Upload API
participant n_raw as Raw uploads
participant n_viewer as Viewer
participant n_queue as Transcode queue
participant n_workers as Transcoders
participant n_media as Video storage
participant n_cdn as CDN
participant n_counter as View counter
participant n_videos as Videos DB
participant n_cold as Cold storage
Note over n_creator,n_upload: A 2 GB file in one request
n_creator->>n_upload: POST /v1/videos HTTP/1.1
n_upload-->>n_creator: HTTP/1.1 504 Gateway Timeout
Note over n_creator,n_raw: Upload straight to storage, in parts
n_creator->>n_raw: PUT /raw/v_8812.mp4?partNumber=88#38;uploadId=u_31#38;X-Amz-Signature=*** HTTP/1.1
n_raw-->>n_creator: HTTP/1.1 200 OK
Note over n_viewer,n_raw: Playing the original on a phone
n_viewer->>n_raw: GET /raw/v_8812.mp4 HTTP/1.1
n_raw-->>n_viewer: HTTP/1.1 206 Partial Content
Note over n_queue,n_workers: Transcode into a bitrate ladder, in chunks
n_queue->>n_workers: SEND transcode
n_workers-->>n_queue: chunk 17 done in 70 s by w_09
Note over n_viewer,n_media: HLS: playlists and 6-second segments
n_viewer->>n_media: GET /v_8812/master.m3u8 HTTP/1.1
n_media-->>n_viewer: #35;EXTM3U
Note over n_viewer,n_media: Adaptive bitrate: quality follows the link
n_viewer->>n_media: GET /v_8812/480p/seg_042.m4s HTTP/1.1
n_media-->>n_viewer: HTTP/1.1 200 OK
Note over n_viewer,n_media: A premiere: 200 000 viewers at once
n_viewer->>n_media: 20:00 premiere: 200000 viewers x 5 Mbit/s
Note over n_viewer,n_cdn: Segments from the CDN
n_viewer->>n_cdn: GET /v_8812/720p/seg_042.m4s HTTP/1.1
n_cdn-->>n_viewer: HTTP/1.1 200 OK
Note over n_counter,n_videos: One row for every view
n_counter->>n_videos: UPDATE videos SET views = views + 1 WHERE id = 'v_8812'#59;
n_videos-->>n_counter: lock wait on row v_8812
Note over n_counter,n_videos: Count in windows, write once a minute
n_counter->>n_videos: counter: 1 minute window for v_8812
Note over n_media,n_cold: Hot and cold storage
n_media->>n_cold: PUT lifecycle media
n_cold-->>n_media: 4.1 PB moved, first byte from cold: up to 3 sD2
direction: right
n_creator: "Creator" {
shape: rectangle
}
n_upload: "Upload API" {
shape: rectangle
}
n_raw: "Raw uploads" {
shape: stored_data
}
n_viewer: "Viewer" {
shape: rectangle
}
n_queue: "Transcode queue" {
shape: queue
}
n_workers: "Transcoders" {
shape: rectangle
style.multiple: true
}
n_media: "Video storage" {
shape: stored_data
}
n_cdn: "CDN" {
shape: oval
}
n_counter: "View counter" {
shape: rectangle
}
n_videos: "Videos DB" {
shape: cylinder
}
n_cold: "Cold storage" {
shape: stored_data
}
n_creator -> n_upload
n_creator -> n_raw
n_raw -> n_queue
n_queue -> n_workers
n_workers -> n_media
n_viewer -> n_cdn
n_cdn -> n_media
n_viewer -> n_counter
n_counter -> n_videos
n_media -> n_coldPlantUML: request sequence
@startuml
participant "Creator" as n_creator
participant "Upload API" as n_upload
database "Raw uploads" as n_raw
participant "Viewer" as n_viewer
queue "Transcode queue" as n_queue
participant "Transcoders" as n_workers
database "Video storage" as n_media
participant "CDN" as n_cdn
participant "View counter" as n_counter
database "Videos DB" as n_videos
database "Cold storage" as n_cold
note over n_creator, n_upload : A 2 GB file in one request
n_creator -> n_upload : POST /v1/videos HTTP/1.1
n_upload --> n_creator : HTTP/1.1 504 Gateway Timeout
note over n_creator, n_raw : Upload straight to storage, in parts
n_creator -> n_raw : PUT /raw/v_8812.mp4?partNumber=88&uploadId=u_31&X-Amz-Signature=*** HTTP/1.1
n_raw --> n_creator : HTTP/1.1 200 OK
note over n_viewer, n_raw : Playing the original on a phone
n_viewer -> n_raw : GET /raw/v_8812.mp4 HTTP/1.1
n_raw --> n_viewer : HTTP/1.1 206 Partial Content
note over n_queue, n_workers : Transcode into a bitrate ladder, in chunks
n_queue -> n_workers : SEND transcode
n_workers --> n_queue : chunk 17 done in 70 s by w_09
note over n_viewer, n_media : HLS: playlists and 6-second segments
n_viewer -> n_media : GET /v_8812/master.m3u8 HTTP/1.1
n_media --> n_viewer : #EXTM3U
note over n_viewer, n_media : Adaptive bitrate: quality follows the link
n_viewer -> n_media : GET /v_8812/480p/seg_042.m4s HTTP/1.1
n_media --> n_viewer : HTTP/1.1 200 OK
note over n_viewer, n_media : A premiere: 200 000 viewers at once
n_viewer -> n_media : 20:00 premiere: 200000 viewers x 5 Mbit/s
note over n_viewer, n_cdn : Segments from the CDN
n_viewer -> n_cdn : GET /v_8812/720p/seg_042.m4s HTTP/1.1
n_cdn --> n_viewer : HTTP/1.1 200 OK
note over n_counter, n_videos : One row for every view
n_counter -> n_videos : UPDATE videos SET views = views + 1 WHERE id = 'v_8812';
n_videos --> n_counter : lock wait on row v_8812
note over n_counter, n_videos : Count in windows, write once a minute
n_counter -> n_videos : counter: 1 minute window for v_8812
note over n_media, n_cold : Hot and cold storage
n_media -> n_cold : PUT lifecycle media
n_cold --> n_media : 4.1 PB moved, first byte from cold: up to 3 s
@endumlPlantUML: components
@startuml
left to right direction
agent "Creator" as n_creator
component "Upload API" as n_upload
storage "Raw uploads" as n_raw
agent "Viewer" as n_viewer
queue "Transcode queue" as n_queue
collections "Transcoders" as n_workers
storage "Video storage" as n_media
frame "CDN" as n_cdn
component "View counter" as n_counter
database "Videos DB" as n_videos
storage "Cold storage" as n_cold
n_creator --> n_upload
n_creator --> n_raw
n_raw --> n_queue
n_queue --> n_workers
n_workers --> n_media
n_viewer --> n_cdn
n_cdn --> n_media
n_viewer --> n_counter
n_counter --> n_videos
n_media --> n_cold
@endumlRecap
Six decisions for a video site.
1) Uploads go straight to storage in resumable parts; app servers only sign. 2) Transcode into a bitrate ladder, in parallel chunks. 3) Serve HLS: playlists plus small aligned segments. 4) Let the player adapt quality to the link and buffer. 5) Put a CDN in front; immutable segment names make caching safe. 6) Count views in windows, and move the long tail to cold storage.
Numbers: at the start and at the end
| Metric | Start | End |
|---|---|---|
| Time to first frame | 14 s | 2 s |
| Rebuffering | 22% | 2% |
| Upload to playable | 12 min | 12 min |
| Origin traffic | 1 000 Gbps | 20 Gbps |
| Storage cost | 18 000 $/mo | 7 000 $/mo |
Steps
- Problem A 2 GB file in one request
A creator uploads a one-hour 4K video, 2 GB, from a hotel.
- Fix Upload straight to storage, in parts
The upload API now only checks the creator and hands out signed links, one per 16 MB part.
- Problem Playing the original on a phone
The first version plays the uploaded file as it is.
- Fix Transcode into a bitrate ladder, in chunks
When the upload completes, storage puts a job in the transcode queue.
- Fix HLS: playlists and 6-second segments
Each rendition is cut into 6-second segments, small files with names like
seg_042.m4s. - Case Adaptive bitrate: quality follows the link
The viewer enters a metro station.
- Problem A premiere: 200 000 viewers at once
A popular channel premieres a new video at 20:00.
- Fix Segments from the CDN
A CDN stands in front of the video storage.
- Problem One row for every view
The player reports a heartbeat while it plays, and the counter adds one to the video's row for each.
- Fix Count in windows, write once a minute
The counter now sums heartbeats in memory for one minute.
- Fix Hot and cold storage
Most views go to a small share of videos; most videos are watched a handful of times a year.
- Recap Recap
Six decisions for a video site.
Terms
- Transcoding
- Re-encoding an uploaded video into several qualities and formats (for example 1080p, 720p, 480p). It is heavy work, so the video is cut into chunks and encoded in parallel on many workers.
- HLS (HTTP Live Streaming)
- Video is cut into segments a few seconds long, and the player fetches the next one with a plain HTTP GET, following a playlist file (.m3u8). Segments are ordinary files, so a CDN caches them well.
- Adaptive bitrate (ABR)
- Before each segment the player picks a quality from the network speed and the buffer: when the network slows it drops to 480p, when it recovers it goes back to 1080p. The video keeps playing, only the quality changes.