mufassall

Konspekt · Butun tizimlar

Video sayt loyihalash

16 daq · Yuqori · mufassall.dtwin.uz/uz/l/design-video-streaming/

← Darsni ochish

Muammo

Mualliflar bir soatlik 4K videolarni yuklaydi, tomoshabinlar esa beqaror mobil internetda telefondan ko'radi, premyera bo'lsa bir vaqtda 200 000 kishi keladi. Video sayt qadamma-qadam: davom ettiriladigan yuklash, transkodlash konveyeri, adaptiv bitreytli HLS, CDN, ko'rishlarni sanash va sovuq saqlash.

Nimani o'rganasiz

  • Katta yuklash nega to'g'ridan saqlash joyiga, davom ettiriladigan qismlarda boradi
  • Bitreyt zinapoyasiga transkodlash, parallel bo'laklarda
  • HLS segmentlari va pleylistlari, pleyer sifatni qanday almashtiradi
  • Origin oldida CDN, qizg'in qatorsiz ko'rishlarni sanash, issiq va sovuq saqlash
Yakuniy sahna
10%40%50%60%40%20%

Diagramma kod sifatida

Darsdagi yakuniy sxemani matn ko'rinishida oling va Jira, GitLab, GitHub, Notion yoki hujjatga qo'ying: Mermaid u yerlarda o'zi chiziladi. D2 va PlantUML (Confluence, IntelliJ) uchun alohida variantlar bor.

Mermaid: sxema

flowchart LR
  n_creator(["Muallif"])
  n_upload["Yuklash API"]
  n_raw[("Asl fayllar")]
  n_viewer(["Tomoshabin"])
  n_queue[["Transkod navbati"]]
  n_workers("Transkoderlar")
  n_media[("Video ombori")]
  n_cdn[/"CDN"\]
  n_counter["Ko'rish hisoblagichi"]
  n_videos[("Videolar bazasi")]
  n_cold[("Sovuq ombor")]
  n_creator --> n_upload
  n_creator --> n_raw
  n_raw --> n_queue
  n_queue --> n_workers
  n_workers --> n_media
  n_viewer --> n_cdn
  n_cdn --> n_media
  n_viewer --> n_counter
  n_counter --> n_videos
  n_media --> n_cold

Mermaid: so'rovlar ketma-ketligi

sequenceDiagram
  participant n_creator as Muallif
  participant n_upload as Yuklash API
  participant n_raw as Asl fayllar
  participant n_viewer as Tomoshabin
  participant n_queue as Transkod navbati
  participant n_workers as Transkoderlar
  participant n_media as Video ombori
  participant n_cdn as CDN
  participant n_counter as Ko'rish hisoblagichi
  participant n_videos as Videolar bazasi
  participant n_cold as Sovuq ombor
  Note over n_creator,n_upload: 2 GB lik fayl bitta so'rovda
  n_creator->>n_upload: POST /v1/videos HTTP/1.1
  n_upload-->>n_creator: HTTP/1.1 504 Gateway Timeout
  Note over n_creator,n_raw: To'g'ridan saqlash joyiga, qismlarda
  n_creator->>n_raw: PUT /raw/v_8812.mp4?partNumber=88#38;uploadId=u_31#38;X-Amz-Signature=*** HTTP/1.1
  n_raw-->>n_creator: HTTP/1.1 200 OK
  Note over n_viewer,n_raw: Asl faylni telefonda ko'rsatish
  n_viewer->>n_raw: GET /raw/v_8812.mp4 HTTP/1.1
  n_raw-->>n_viewer: HTTP/1.1 206 Partial Content
  Note over n_queue,n_workers: Bitreyt zinapoyasiga, bo'laklarda transkodlash
  n_queue->>n_workers: SEND transcode
  n_workers-->>n_queue: chunk 17 done in 70 s by w_09
  Note over n_viewer,n_media: HLS: pleylistlar va 6 soniyalik segmentlar
  n_viewer->>n_media: GET /v_8812/master.m3u8 HTTP/1.1
  n_media-->>n_viewer: #35;EXTM3U
  Note over n_viewer,n_media: Adaptiv bitreyt: sifat aloqaga ergashadi
  n_viewer->>n_media: GET /v_8812/480p/seg_042.m4s HTTP/1.1
  n_media-->>n_viewer: HTTP/1.1 200 OK
  Note over n_viewer,n_media: Premyera: bir vaqtda 200 000 tomoshabin
  n_viewer->>n_media: 20:00 premiere: 200000 viewers x 5 Mbit/s
  Note over n_viewer,n_cdn: Segmentlar CDN'dan
  n_viewer->>n_cdn: GET /v_8812/720p/seg_042.m4s HTTP/1.1
  n_cdn-->>n_viewer: HTTP/1.1 200 OK
  Note over n_counter,n_videos: Har ko'rish uchun bitta qator
  n_counter->>n_videos: UPDATE videos SET views = views + 1 WHERE id = 'v_8812'#59;
  n_videos-->>n_counter: lock wait on row v_8812
  Note over n_counter,n_videos: Oynalarda sanash, daqiqada bir marta yozish
  n_counter->>n_videos: counter: 1 minute window for v_8812
  Note over n_media,n_cold: Issiq va sovuq saqlash
  n_media->>n_cold: PUT lifecycle media
  n_cold-->>n_media: 4.1 PB moved, first byte from cold: up to 3 s

D2

direction: right

n_creator: "Muallif" {
  shape: rectangle
}
n_upload: "Yuklash API" {
  shape: rectangle
}
n_raw: "Asl fayllar" {
  shape: stored_data
}
n_viewer: "Tomoshabin" {
  shape: rectangle
}
n_queue: "Transkod navbati" {
  shape: queue
}
n_workers: "Transkoderlar" {
  shape: rectangle
  style.multiple: true
}
n_media: "Video ombori" {
  shape: stored_data
}
n_cdn: "CDN" {
  shape: oval
}
n_counter: "Ko'rish hisoblagichi" {
  shape: rectangle
}
n_videos: "Videolar bazasi" {
  shape: cylinder
}
n_cold: "Sovuq ombor" {
  shape: stored_data
}

n_creator -> n_upload
n_creator -> n_raw
n_raw -> n_queue
n_queue -> n_workers
n_workers -> n_media
n_viewer -> n_cdn
n_cdn -> n_media
n_viewer -> n_counter
n_counter -> n_videos
n_media -> n_cold

PlantUML: so'rovlar ketma-ketligi

@startuml
participant "Muallif" as n_creator
participant "Yuklash API" as n_upload
database "Asl fayllar" as n_raw
participant "Tomoshabin" as n_viewer
queue "Transkod navbati" as n_queue
participant "Transkoderlar" as n_workers
database "Video ombori" as n_media
participant "CDN" as n_cdn
participant "Ko'rish hisoblagichi" as n_counter
database "Videolar bazasi" as n_videos
database "Sovuq ombor" as n_cold
note over n_creator, n_upload : 2 GB lik fayl bitta so'rovda
n_creator -> n_upload : POST /v1/videos HTTP/1.1
n_upload --> n_creator : HTTP/1.1 504 Gateway Timeout
note over n_creator, n_raw : To'g'ridan saqlash joyiga, qismlarda
n_creator -> n_raw : PUT /raw/v_8812.mp4?partNumber=88&uploadId=u_31&X-Amz-Signature=*** HTTP/1.1
n_raw --> n_creator : HTTP/1.1 200 OK
note over n_viewer, n_raw : Asl faylni telefonda ko'rsatish
n_viewer -> n_raw : GET /raw/v_8812.mp4 HTTP/1.1
n_raw --> n_viewer : HTTP/1.1 206 Partial Content
note over n_queue, n_workers : Bitreyt zinapoyasiga, bo'laklarda transkodlash
n_queue -> n_workers : SEND transcode
n_workers --> n_queue : chunk 17 done in 70 s by w_09
note over n_viewer, n_media : HLS: pleylistlar va 6 soniyalik segmentlar
n_viewer -> n_media : GET /v_8812/master.m3u8 HTTP/1.1
n_media --> n_viewer : #EXTM3U
note over n_viewer, n_media : Adaptiv bitreyt: sifat aloqaga ergashadi
n_viewer -> n_media : GET /v_8812/480p/seg_042.m4s HTTP/1.1
n_media --> n_viewer : HTTP/1.1 200 OK
note over n_viewer, n_media : Premyera: bir vaqtda 200 000 tomoshabin
n_viewer -> n_media : 20:00 premiere: 200000 viewers x 5 Mbit/s
note over n_viewer, n_cdn : Segmentlar CDN'dan
n_viewer -> n_cdn : GET /v_8812/720p/seg_042.m4s HTTP/1.1
n_cdn --> n_viewer : HTTP/1.1 200 OK
note over n_counter, n_videos : Har ko'rish uchun bitta qator
n_counter -> n_videos : UPDATE videos SET views = views + 1 WHERE id = 'v_8812';
n_videos --> n_counter : lock wait on row v_8812
note over n_counter, n_videos : Oynalarda sanash, daqiqada bir marta yozish
n_counter -> n_videos : counter: 1 minute window for v_8812
note over n_media, n_cold : Issiq va sovuq saqlash
n_media -> n_cold : PUT lifecycle media
n_cold --> n_media : 4.1 PB moved, first byte from cold: up to 3 s
@enduml

PlantUML: komponentlar

@startuml
left to right direction
agent "Muallif" as n_creator
component "Yuklash API" as n_upload
storage "Asl fayllar" as n_raw
agent "Tomoshabin" as n_viewer
queue "Transkod navbati" as n_queue
collections "Transkoderlar" as n_workers
storage "Video ombori" as n_media
frame "CDN" as n_cdn
component "Ko'rish hisoblagichi" as n_counter
database "Videolar bazasi" as n_videos
storage "Sovuq ombor" as n_cold
n_creator --> n_upload
n_creator --> n_raw
n_raw --> n_queue
n_queue --> n_workers
n_workers --> n_media
n_viewer --> n_cdn
n_cdn --> n_media
n_viewer --> n_counter
n_counter --> n_videos
n_media --> n_cold
@enduml

Xulosa

Video sayt uchun oltita qaror.

1) Yuklash to'g'ridan saqlash joyiga, davom ettiriladigan qismlarda boradi; ilova serverlari faqat imzolaydi. 2) Bitreyt zinapoyasiga parallel bo'laklarda transkodlang. 3) HLS bering: pleylistlar va kichik, tekislangan segmentlar. 4) Pleyer sifatni aloqa va buferga moslasin. 5) Oldiga CDN qo'ying; o'zgarmas segment nomlari keshni xavfsiz qiladi. 6) Ko'rishlarni oynalarda sanang, uzun dumni sovuq omborga ko'chiring.

Ko'rsatkichlar: boshida va oxirida

Ko'rsatkichBoshidaOxirida
Birinchi kadrgacha14 s2 s
Buferlash22%2%
Yuklashdan ko'rishgacha12 min12 min
Origin trafigi1 000 Gbps20 Gbps
Saqlash narxi18 000 $/mo7 000 $/mo

Qadamlar

  1. Muammo 2 GB lik fayl bitta so'rovda

    Muallif mehmonxonadan bir soatlik 4K videoni, 2 GB ni yuklaydi.

  2. Yechim To'g'ridan saqlash joyiga, qismlarda

    Yuklash API endi faqat muallifni tekshiradi va imzolangan havolalar beradi, har 16 MB lik qism uchun bittadan.

  3. Muammo Asl faylni telefonda ko'rsatish

    Birinchi versiya yuklangan faylni o'zicha ko'rsatadi.

  4. Yechim Bitreyt zinapoyasiga, bo'laklarda transkodlash

    Yuklash tugagach, saqlash joyi transkod navbatiga vazifa qo'yadi.

  5. Yechim HLS: pleylistlar va 6 soniyalik segmentlar

    Har variant 6 soniyalik segmentlarga, seg_042.m4s kabi nomli kichik fayllarga kesiladi.

  6. Holat Adaptiv bitreyt: sifat aloqaga ergashadi

    Tomoshabin metroga tushadi.

  7. Muammo Premyera: bir vaqtda 200 000 tomoshabin

    Mashhur kanal soat 20:00 da yangi video premyerasini qiladi.

  8. Yechim Segmentlar CDN'dan

    Video ombori oldiga CDN qo'yiladi.

  9. Muammo Har ko'rish uchun bitta qator

    Pleyer o'ynayotganda heartbeat yuboradi, hisoblagich esa har biri uchun video qatoriga bitta qo'shadi.

  10. Yechim Oynalarda sanash, daqiqada bir marta yozish

    Hisoblagich endi heartbeat'larni bir daqiqa xotirada yig'adi.

  11. Yechim Issiq va sovuq saqlash

    Ko'rishlarning ko'pi videolarning kichik qismiga to'g'ri keladi; ko'p videolarni yiliga bir necha marta ko'rishadi.

  12. Xulosa Xulosa

    Video sayt uchun oltita qaror.

Atamalar

Transkodlash
Yuklangan videoni bir nechta sifat va formatga (masalan 1080p, 720p, 480p) qayta kodlash. Og'ir ish, shuning uchun video bo'laklarga bo'linib, ko'p worker'da parallel bajariladi.
HLS (HTTP Live Streaming)
Video bir necha soniyalik bo'laklarga bo'linadi, pleyer esa ro'yxat faylidan (.m3u8) keyingi bo'lakni oddiy HTTP GET bilan oladi. Bo'laklar oddiy fayl bo'lgani uchun CDN ularni yaxshi keshlaydi.
Moslashuvchan bitreyt (ABR)
Pleyer har bo'lakdan oldin tarmoq tezligi va buferga qarab sifatni tanlaydi: internet sekinlashsa 480p ga tushadi, tezlashsa 1080p ga qaytadi. Video to'xtab qolmaydi, faqat sifati o'zgaradi.