Konspekt · Butun tizimlar
Video sayt loyihalash
Muammo
Mualliflar bir soatlik 4K videolarni yuklaydi, tomoshabinlar esa beqaror mobil internetda telefondan ko'radi, premyera bo'lsa bir vaqtda 200 000 kishi keladi. Video sayt qadamma-qadam: davom ettiriladigan yuklash, transkodlash konveyeri, adaptiv bitreytli HLS, CDN, ko'rishlarni sanash va sovuq saqlash.
Nimani o'rganasiz
- Katta yuklash nega to'g'ridan saqlash joyiga, davom ettiriladigan qismlarda boradi
- Bitreyt zinapoyasiga transkodlash, parallel bo'laklarda
- HLS segmentlari va pleylistlari, pleyer sifatni qanday almashtiradi
- Origin oldida CDN, qizg'in qatorsiz ko'rishlarni sanash, issiq va sovuq saqlash
Diagramma kod sifatida
Darsdagi yakuniy sxemani matn ko'rinishida oling va Jira, GitLab, GitHub, Notion yoki hujjatga qo'ying: Mermaid u yerlarda o'zi chiziladi. D2 va PlantUML (Confluence, IntelliJ) uchun alohida variantlar bor.
Mermaid: sxema
flowchart LR
n_creator(["Muallif"])
n_upload["Yuklash API"]
n_raw[("Asl fayllar")]
n_viewer(["Tomoshabin"])
n_queue[["Transkod navbati"]]
n_workers("Transkoderlar")
n_media[("Video ombori")]
n_cdn[/"CDN"\]
n_counter["Ko'rish hisoblagichi"]
n_videos[("Videolar bazasi")]
n_cold[("Sovuq ombor")]
n_creator --> n_upload
n_creator --> n_raw
n_raw --> n_queue
n_queue --> n_workers
n_workers --> n_media
n_viewer --> n_cdn
n_cdn --> n_media
n_viewer --> n_counter
n_counter --> n_videos
n_media --> n_coldMermaid: so'rovlar ketma-ketligi
sequenceDiagram
participant n_creator as Muallif
participant n_upload as Yuklash API
participant n_raw as Asl fayllar
participant n_viewer as Tomoshabin
participant n_queue as Transkod navbati
participant n_workers as Transkoderlar
participant n_media as Video ombori
participant n_cdn as CDN
participant n_counter as Ko'rish hisoblagichi
participant n_videos as Videolar bazasi
participant n_cold as Sovuq ombor
Note over n_creator,n_upload: 2 GB lik fayl bitta so'rovda
n_creator->>n_upload: POST /v1/videos HTTP/1.1
n_upload-->>n_creator: HTTP/1.1 504 Gateway Timeout
Note over n_creator,n_raw: To'g'ridan saqlash joyiga, qismlarda
n_creator->>n_raw: PUT /raw/v_8812.mp4?partNumber=88#38;uploadId=u_31#38;X-Amz-Signature=*** HTTP/1.1
n_raw-->>n_creator: HTTP/1.1 200 OK
Note over n_viewer,n_raw: Asl faylni telefonda ko'rsatish
n_viewer->>n_raw: GET /raw/v_8812.mp4 HTTP/1.1
n_raw-->>n_viewer: HTTP/1.1 206 Partial Content
Note over n_queue,n_workers: Bitreyt zinapoyasiga, bo'laklarda transkodlash
n_queue->>n_workers: SEND transcode
n_workers-->>n_queue: chunk 17 done in 70 s by w_09
Note over n_viewer,n_media: HLS: pleylistlar va 6 soniyalik segmentlar
n_viewer->>n_media: GET /v_8812/master.m3u8 HTTP/1.1
n_media-->>n_viewer: #35;EXTM3U
Note over n_viewer,n_media: Adaptiv bitreyt: sifat aloqaga ergashadi
n_viewer->>n_media: GET /v_8812/480p/seg_042.m4s HTTP/1.1
n_media-->>n_viewer: HTTP/1.1 200 OK
Note over n_viewer,n_media: Premyera: bir vaqtda 200 000 tomoshabin
n_viewer->>n_media: 20:00 premiere: 200000 viewers x 5 Mbit/s
Note over n_viewer,n_cdn: Segmentlar CDN'dan
n_viewer->>n_cdn: GET /v_8812/720p/seg_042.m4s HTTP/1.1
n_cdn-->>n_viewer: HTTP/1.1 200 OK
Note over n_counter,n_videos: Har ko'rish uchun bitta qator
n_counter->>n_videos: UPDATE videos SET views = views + 1 WHERE id = 'v_8812'#59;
n_videos-->>n_counter: lock wait on row v_8812
Note over n_counter,n_videos: Oynalarda sanash, daqiqada bir marta yozish
n_counter->>n_videos: counter: 1 minute window for v_8812
Note over n_media,n_cold: Issiq va sovuq saqlash
n_media->>n_cold: PUT lifecycle media
n_cold-->>n_media: 4.1 PB moved, first byte from cold: up to 3 sD2
direction: right
n_creator: "Muallif" {
shape: rectangle
}
n_upload: "Yuklash API" {
shape: rectangle
}
n_raw: "Asl fayllar" {
shape: stored_data
}
n_viewer: "Tomoshabin" {
shape: rectangle
}
n_queue: "Transkod navbati" {
shape: queue
}
n_workers: "Transkoderlar" {
shape: rectangle
style.multiple: true
}
n_media: "Video ombori" {
shape: stored_data
}
n_cdn: "CDN" {
shape: oval
}
n_counter: "Ko'rish hisoblagichi" {
shape: rectangle
}
n_videos: "Videolar bazasi" {
shape: cylinder
}
n_cold: "Sovuq ombor" {
shape: stored_data
}
n_creator -> n_upload
n_creator -> n_raw
n_raw -> n_queue
n_queue -> n_workers
n_workers -> n_media
n_viewer -> n_cdn
n_cdn -> n_media
n_viewer -> n_counter
n_counter -> n_videos
n_media -> n_coldPlantUML: so'rovlar ketma-ketligi
@startuml
participant "Muallif" as n_creator
participant "Yuklash API" as n_upload
database "Asl fayllar" as n_raw
participant "Tomoshabin" as n_viewer
queue "Transkod navbati" as n_queue
participant "Transkoderlar" as n_workers
database "Video ombori" as n_media
participant "CDN" as n_cdn
participant "Ko'rish hisoblagichi" as n_counter
database "Videolar bazasi" as n_videos
database "Sovuq ombor" as n_cold
note over n_creator, n_upload : 2 GB lik fayl bitta so'rovda
n_creator -> n_upload : POST /v1/videos HTTP/1.1
n_upload --> n_creator : HTTP/1.1 504 Gateway Timeout
note over n_creator, n_raw : To'g'ridan saqlash joyiga, qismlarda
n_creator -> n_raw : PUT /raw/v_8812.mp4?partNumber=88&uploadId=u_31&X-Amz-Signature=*** HTTP/1.1
n_raw --> n_creator : HTTP/1.1 200 OK
note over n_viewer, n_raw : Asl faylni telefonda ko'rsatish
n_viewer -> n_raw : GET /raw/v_8812.mp4 HTTP/1.1
n_raw --> n_viewer : HTTP/1.1 206 Partial Content
note over n_queue, n_workers : Bitreyt zinapoyasiga, bo'laklarda transkodlash
n_queue -> n_workers : SEND transcode
n_workers --> n_queue : chunk 17 done in 70 s by w_09
note over n_viewer, n_media : HLS: pleylistlar va 6 soniyalik segmentlar
n_viewer -> n_media : GET /v_8812/master.m3u8 HTTP/1.1
n_media --> n_viewer : #EXTM3U
note over n_viewer, n_media : Adaptiv bitreyt: sifat aloqaga ergashadi
n_viewer -> n_media : GET /v_8812/480p/seg_042.m4s HTTP/1.1
n_media --> n_viewer : HTTP/1.1 200 OK
note over n_viewer, n_media : Premyera: bir vaqtda 200 000 tomoshabin
n_viewer -> n_media : 20:00 premiere: 200000 viewers x 5 Mbit/s
note over n_viewer, n_cdn : Segmentlar CDN'dan
n_viewer -> n_cdn : GET /v_8812/720p/seg_042.m4s HTTP/1.1
n_cdn --> n_viewer : HTTP/1.1 200 OK
note over n_counter, n_videos : Har ko'rish uchun bitta qator
n_counter -> n_videos : UPDATE videos SET views = views + 1 WHERE id = 'v_8812';
n_videos --> n_counter : lock wait on row v_8812
note over n_counter, n_videos : Oynalarda sanash, daqiqada bir marta yozish
n_counter -> n_videos : counter: 1 minute window for v_8812
note over n_media, n_cold : Issiq va sovuq saqlash
n_media -> n_cold : PUT lifecycle media
n_cold --> n_media : 4.1 PB moved, first byte from cold: up to 3 s
@endumlPlantUML: komponentlar
@startuml
left to right direction
agent "Muallif" as n_creator
component "Yuklash API" as n_upload
storage "Asl fayllar" as n_raw
agent "Tomoshabin" as n_viewer
queue "Transkod navbati" as n_queue
collections "Transkoderlar" as n_workers
storage "Video ombori" as n_media
frame "CDN" as n_cdn
component "Ko'rish hisoblagichi" as n_counter
database "Videolar bazasi" as n_videos
storage "Sovuq ombor" as n_cold
n_creator --> n_upload
n_creator --> n_raw
n_raw --> n_queue
n_queue --> n_workers
n_workers --> n_media
n_viewer --> n_cdn
n_cdn --> n_media
n_viewer --> n_counter
n_counter --> n_videos
n_media --> n_cold
@endumlXulosa
Video sayt uchun oltita qaror.
1) Yuklash to'g'ridan saqlash joyiga, davom ettiriladigan qismlarda boradi; ilova serverlari faqat imzolaydi. 2) Bitreyt zinapoyasiga parallel bo'laklarda transkodlang. 3) HLS bering: pleylistlar va kichik, tekislangan segmentlar. 4) Pleyer sifatni aloqa va buferga moslasin. 5) Oldiga CDN qo'ying; o'zgarmas segment nomlari keshni xavfsiz qiladi. 6) Ko'rishlarni oynalarda sanang, uzun dumni sovuq omborga ko'chiring.
Ko'rsatkichlar: boshida va oxirida
| Ko'rsatkich | Boshida | Oxirida |
|---|---|---|
| Birinchi kadrgacha | 14 s | 2 s |
| Buferlash | 22% | 2% |
| Yuklashdan ko'rishgacha | 12 min | 12 min |
| Origin trafigi | 1 000 Gbps | 20 Gbps |
| Saqlash narxi | 18 000 $/mo | 7 000 $/mo |
Qadamlar
- Muammo 2 GB lik fayl bitta so'rovda
Muallif mehmonxonadan bir soatlik 4K videoni, 2 GB ni yuklaydi.
- Yechim To'g'ridan saqlash joyiga, qismlarda
Yuklash API endi faqat muallifni tekshiradi va imzolangan havolalar beradi, har 16 MB lik qism uchun bittadan.
- Muammo Asl faylni telefonda ko'rsatish
Birinchi versiya yuklangan faylni o'zicha ko'rsatadi.
- Yechim Bitreyt zinapoyasiga, bo'laklarda transkodlash
Yuklash tugagach, saqlash joyi transkod navbatiga vazifa qo'yadi.
- Yechim HLS: pleylistlar va 6 soniyalik segmentlar
Har variant 6 soniyalik segmentlarga,
seg_042.m4skabi nomli kichik fayllarga kesiladi. - Holat Adaptiv bitreyt: sifat aloqaga ergashadi
Tomoshabin metroga tushadi.
- Muammo Premyera: bir vaqtda 200 000 tomoshabin
Mashhur kanal soat 20:00 da yangi video premyerasini qiladi.
- Yechim Segmentlar CDN'dan
Video ombori oldiga CDN qo'yiladi.
- Muammo Har ko'rish uchun bitta qator
Pleyer o'ynayotganda heartbeat yuboradi, hisoblagich esa har biri uchun video qatoriga bitta qo'shadi.
- Yechim Oynalarda sanash, daqiqada bir marta yozish
Hisoblagich endi heartbeat'larni bir daqiqa xotirada yig'adi.
- Yechim Issiq va sovuq saqlash
Ko'rishlarning ko'pi videolarning kichik qismiga to'g'ri keladi; ko'p videolarni yiliga bir necha marta ko'rishadi.
- Xulosa Xulosa
Video sayt uchun oltita qaror.
Atamalar
- Transkodlash
- Yuklangan videoni bir nechta sifat va formatga (masalan 1080p, 720p, 480p) qayta kodlash. Og'ir ish, shuning uchun video bo'laklarga bo'linib, ko'p worker'da parallel bajariladi.
- HLS (HTTP Live Streaming)
- Video bir necha soniyalik bo'laklarga bo'linadi, pleyer esa ro'yxat faylidan (.m3u8) keyingi bo'lakni oddiy HTTP GET bilan oladi. Bo'laklar oddiy fayl bo'lgani uchun CDN ularni yaxshi keshlaydi.
- Moslashuvchan bitreyt (ABR)
- Pleyer har bo'lakdan oldin tarmoq tezligi va buferga qarab sifatni tanlaydi: internet sekinlashsa 480p ga tushadi, tezlashsa 1080p ga qaytadi. Video to'xtab qolmaydi, faqat sifati o'zgaradi.