mufassall

Cheat sheet · Whole systems

Design a video streaming site

16 min · Advanced · mufassall.dtwin.uz/en/l/design-video-streaming/

← Open the lesson

The problem

Creators upload hour-long 4K videos, viewers watch on phones over shaky mobile links, and a premiere brings 200 000 people at once. A video site built step by step: resumable uploads, a transcoding pipeline, HLS with adaptive bitrate, a CDN, view counting and cold storage.

What you learn

  • Why a big upload goes straight to storage in resumable parts
  • Transcoding into a bitrate ladder, in parallel chunks
  • HLS segments and playlists, and how the player switches quality
  • CDN in front of the origin, counting views without a hot row, hot and cold storage
Final scene
10%40%50%60%40%20%

Diagram as code

The lesson's final design as text: paste it into Jira, GitLab, GitHub, Notion or your docs, where Mermaid renders by itself. There are D2 and PlantUML (Confluence, IntelliJ) versions too.

Mermaid: flowchart

flowchart LR
  n_creator(["Creator"])
  n_upload["Upload API"]
  n_raw[("Raw uploads")]
  n_viewer(["Viewer"])
  n_queue[["Transcode queue"]]
  n_workers("Transcoders")
  n_media[("Video storage")]
  n_cdn[/"CDN"\]
  n_counter["View counter"]
  n_videos[("Videos DB")]
  n_cold[("Cold storage")]
  n_creator --> n_upload
  n_creator --> n_raw
  n_raw --> n_queue
  n_queue --> n_workers
  n_workers --> n_media
  n_viewer --> n_cdn
  n_cdn --> n_media
  n_viewer --> n_counter
  n_counter --> n_videos
  n_media --> n_cold

Mermaid: request sequence

sequenceDiagram
  participant n_creator as Creator
  participant n_upload as Upload API
  participant n_raw as Raw uploads
  participant n_viewer as Viewer
  participant n_queue as Transcode queue
  participant n_workers as Transcoders
  participant n_media as Video storage
  participant n_cdn as CDN
  participant n_counter as View counter
  participant n_videos as Videos DB
  participant n_cold as Cold storage
  Note over n_creator,n_upload: A 2 GB file in one request
  n_creator->>n_upload: POST /v1/videos HTTP/1.1
  n_upload-->>n_creator: HTTP/1.1 504 Gateway Timeout
  Note over n_creator,n_raw: Upload straight to storage, in parts
  n_creator->>n_raw: PUT /raw/v_8812.mp4?partNumber=88#38;uploadId=u_31#38;X-Amz-Signature=*** HTTP/1.1
  n_raw-->>n_creator: HTTP/1.1 200 OK
  Note over n_viewer,n_raw: Playing the original on a phone
  n_viewer->>n_raw: GET /raw/v_8812.mp4 HTTP/1.1
  n_raw-->>n_viewer: HTTP/1.1 206 Partial Content
  Note over n_queue,n_workers: Transcode into a bitrate ladder, in chunks
  n_queue->>n_workers: SEND transcode
  n_workers-->>n_queue: chunk 17 done in 70 s by w_09
  Note over n_viewer,n_media: HLS: playlists and 6-second segments
  n_viewer->>n_media: GET /v_8812/master.m3u8 HTTP/1.1
  n_media-->>n_viewer: #35;EXTM3U
  Note over n_viewer,n_media: Adaptive bitrate: quality follows the link
  n_viewer->>n_media: GET /v_8812/480p/seg_042.m4s HTTP/1.1
  n_media-->>n_viewer: HTTP/1.1 200 OK
  Note over n_viewer,n_media: A premiere: 200 000 viewers at once
  n_viewer->>n_media: 20:00 premiere: 200000 viewers x 5 Mbit/s
  Note over n_viewer,n_cdn: Segments from the CDN
  n_viewer->>n_cdn: GET /v_8812/720p/seg_042.m4s HTTP/1.1
  n_cdn-->>n_viewer: HTTP/1.1 200 OK
  Note over n_counter,n_videos: One row for every view
  n_counter->>n_videos: UPDATE videos SET views = views + 1 WHERE id = 'v_8812'#59;
  n_videos-->>n_counter: lock wait on row v_8812
  Note over n_counter,n_videos: Count in windows, write once a minute
  n_counter->>n_videos: counter: 1 minute window for v_8812
  Note over n_media,n_cold: Hot and cold storage
  n_media->>n_cold: PUT lifecycle media
  n_cold-->>n_media: 4.1 PB moved, first byte from cold: up to 3 s

D2

direction: right

n_creator: "Creator" {
  shape: rectangle
}
n_upload: "Upload API" {
  shape: rectangle
}
n_raw: "Raw uploads" {
  shape: stored_data
}
n_viewer: "Viewer" {
  shape: rectangle
}
n_queue: "Transcode queue" {
  shape: queue
}
n_workers: "Transcoders" {
  shape: rectangle
  style.multiple: true
}
n_media: "Video storage" {
  shape: stored_data
}
n_cdn: "CDN" {
  shape: oval
}
n_counter: "View counter" {
  shape: rectangle
}
n_videos: "Videos DB" {
  shape: cylinder
}
n_cold: "Cold storage" {
  shape: stored_data
}

n_creator -> n_upload
n_creator -> n_raw
n_raw -> n_queue
n_queue -> n_workers
n_workers -> n_media
n_viewer -> n_cdn
n_cdn -> n_media
n_viewer -> n_counter
n_counter -> n_videos
n_media -> n_cold

PlantUML: request sequence

@startuml
participant "Creator" as n_creator
participant "Upload API" as n_upload
database "Raw uploads" as n_raw
participant "Viewer" as n_viewer
queue "Transcode queue" as n_queue
participant "Transcoders" as n_workers
database "Video storage" as n_media
participant "CDN" as n_cdn
participant "View counter" as n_counter
database "Videos DB" as n_videos
database "Cold storage" as n_cold
note over n_creator, n_upload : A 2 GB file in one request
n_creator -> n_upload : POST /v1/videos HTTP/1.1
n_upload --> n_creator : HTTP/1.1 504 Gateway Timeout
note over n_creator, n_raw : Upload straight to storage, in parts
n_creator -> n_raw : PUT /raw/v_8812.mp4?partNumber=88&uploadId=u_31&X-Amz-Signature=*** HTTP/1.1
n_raw --> n_creator : HTTP/1.1 200 OK
note over n_viewer, n_raw : Playing the original on a phone
n_viewer -> n_raw : GET /raw/v_8812.mp4 HTTP/1.1
n_raw --> n_viewer : HTTP/1.1 206 Partial Content
note over n_queue, n_workers : Transcode into a bitrate ladder, in chunks
n_queue -> n_workers : SEND transcode
n_workers --> n_queue : chunk 17 done in 70 s by w_09
note over n_viewer, n_media : HLS: playlists and 6-second segments
n_viewer -> n_media : GET /v_8812/master.m3u8 HTTP/1.1
n_media --> n_viewer : #EXTM3U
note over n_viewer, n_media : Adaptive bitrate: quality follows the link
n_viewer -> n_media : GET /v_8812/480p/seg_042.m4s HTTP/1.1
n_media --> n_viewer : HTTP/1.1 200 OK
note over n_viewer, n_media : A premiere: 200 000 viewers at once
n_viewer -> n_media : 20:00 premiere: 200000 viewers x 5 Mbit/s
note over n_viewer, n_cdn : Segments from the CDN
n_viewer -> n_cdn : GET /v_8812/720p/seg_042.m4s HTTP/1.1
n_cdn --> n_viewer : HTTP/1.1 200 OK
note over n_counter, n_videos : One row for every view
n_counter -> n_videos : UPDATE videos SET views = views + 1 WHERE id = 'v_8812';
n_videos --> n_counter : lock wait on row v_8812
note over n_counter, n_videos : Count in windows, write once a minute
n_counter -> n_videos : counter: 1 minute window for v_8812
note over n_media, n_cold : Hot and cold storage
n_media -> n_cold : PUT lifecycle media
n_cold --> n_media : 4.1 PB moved, first byte from cold: up to 3 s
@enduml

PlantUML: components

@startuml
left to right direction
agent "Creator" as n_creator
component "Upload API" as n_upload
storage "Raw uploads" as n_raw
agent "Viewer" as n_viewer
queue "Transcode queue" as n_queue
collections "Transcoders" as n_workers
storage "Video storage" as n_media
frame "CDN" as n_cdn
component "View counter" as n_counter
database "Videos DB" as n_videos
storage "Cold storage" as n_cold
n_creator --> n_upload
n_creator --> n_raw
n_raw --> n_queue
n_queue --> n_workers
n_workers --> n_media
n_viewer --> n_cdn
n_cdn --> n_media
n_viewer --> n_counter
n_counter --> n_videos
n_media --> n_cold
@enduml

Recap

Six decisions for a video site.

1) Uploads go straight to storage in resumable parts; app servers only sign. 2) Transcode into a bitrate ladder, in parallel chunks. 3) Serve HLS: playlists plus small aligned segments. 4) Let the player adapt quality to the link and buffer. 5) Put a CDN in front; immutable segment names make caching safe. 6) Count views in windows, and move the long tail to cold storage.

Numbers: at the start and at the end

MetricStartEnd
Time to first frame14 s2 s
Rebuffering22%2%
Upload to playable12 min12 min
Origin traffic1 000 Gbps20 Gbps
Storage cost18 000 $/mo7 000 $/mo

Steps

  1. Problem A 2 GB file in one request

    A creator uploads a one-hour 4K video, 2 GB, from a hotel.

  2. Fix Upload straight to storage, in parts

    The upload API now only checks the creator and hands out signed links, one per 16 MB part.

  3. Problem Playing the original on a phone

    The first version plays the uploaded file as it is.

  4. Fix Transcode into a bitrate ladder, in chunks

    When the upload completes, storage puts a job in the transcode queue.

  5. Fix HLS: playlists and 6-second segments

    Each rendition is cut into 6-second segments, small files with names like seg_042.m4s.

  6. Case Adaptive bitrate: quality follows the link

    The viewer enters a metro station.

  7. Problem A premiere: 200 000 viewers at once

    A popular channel premieres a new video at 20:00.

  8. Fix Segments from the CDN

    A CDN stands in front of the video storage.

  9. Problem One row for every view

    The player reports a heartbeat while it plays, and the counter adds one to the video's row for each.

  10. Fix Count in windows, write once a minute

    The counter now sums heartbeats in memory for one minute.

  11. Fix Hot and cold storage

    Most views go to a small share of videos; most videos are watched a handful of times a year.

  12. Recap Recap

    Six decisions for a video site.

Terms

Transcoding
Re-encoding an uploaded video into several qualities and formats (for example 1080p, 720p, 480p). It is heavy work, so the video is cut into chunks and encoded in parallel on many workers.
HLS (HTTP Live Streaming)
Video is cut into segments a few seconds long, and the player fetches the next one with a plain HTTP GET, following a playlist file (.m3u8). Segments are ordinary files, so a CDN caches them well.
Adaptive bitrate (ABR)
Before each segment the player picks a quality from the network speed and the buffer: when the network slows it drops to 480p, when it recovers it goes back to 1080p. The video keeps playing, only the quality changes.