Konspekt · Ma'lumot va saqlash
Fayl yuklash
Muammo
Telefonlar 180 MB'lik video yuklaydigan va minglab odam ko'radigan ilova. Nega baytlar API orqali o'tmasligi kerak va ishni presigned URL, multipart upload, worker va CDN qanday bo'lishib oladi.
Nimani o'rganasiz
- Nega ilova serveri orqali yuklash sekin va uning diskini to'ldiradi
- Presigned URL: API imzolaydi, telefon to'g'ridan-to'g'ri storage'ga yuklaydi
- Multipart upload: butun fayl emas, faqat yiqilgan qism qayta yuboriladi
- Thumbnail va transkodlash worker'da, tarqatish CDN orqali, bazada faqat metama'lumot
Diagramma kod sifatida
Darsdagi yakuniy sxemani matn ko'rinishida oling va Jira, GitLab, GitHub, Notion yoki hujjatga qo'ying: Mermaid u yerlarda o'zi chiziladi. D2 va PlantUML (Confluence, IntelliJ) uchun alohida variantlar bor.
Mermaid: sxema
flowchart LR
n_phone(["Telefon ilovasi"])
n_api["API"]
n_bucket[("Object storage")]
n_db[("Baza")]
n_queue[["Navbat"]]
n_worker("Media worker")
n_viewer(["Tomoshabinlar"])
n_cdn[/"CDN"\]
n_phone --> n_api
n_phone --> n_bucket
n_api --> n_db
n_bucket --> n_queue
n_queue --> n_worker
n_worker --> n_bucket
n_worker --> n_db
n_viewer --> n_cdn
n_cdn --> n_bucketMermaid: so'rovlar ketma-ketligi
sequenceDiagram
participant n_phone as Telefon ilovasi
participant n_api as API
participant n_disk as API diski
participant n_bucket as Object storage
participant n_db as Baza
participant n_queue as Navbat
participant n_viewer as Tomoshabinlar
participant n_cdn as CDN
Note over n_phone,n_api: Har bayt API orqali o'tadi
n_phone->>n_api: POST /videos HTTP/1.1
n_api-->>n_phone: HTTP/1.1 201 Created
Note over n_api,n_disk: Disk to'ldi, boshqa serverda fayl yo'q
n_api-xn_disk: write /var/app/uploads/v_9f3.mp4
Note over n_phone,n_api: API imzolaydi, telefon yuklaydi
n_phone->>n_api: POST /uploads HTTP/1.1
n_api-->>n_phone: HTTP/1.1 200 OK
Note over n_phone,n_bucket: Mobil tarmoq: 80% bo'ldi, boshidan
n_phone-xn_bucket: PUT /raw/v_9f3.mp4 HTTP/1.1
Note over n_phone,n_bucket: Multipart: fayl 8 MB'lik qismlarda
n_phone->>n_bucket: PUT /raw/v_9f3.mp4?uploadId=up_31a#38;partNumber=17 HTTP/1.1
n_bucket-->>n_phone: HTTP/1.1 200 OK
Note over n_api,n_db: Bazada fayl emas, qator
n_api->>n_db: INSERT INTO videos
n_db-->>n_api: INSERT 0 1
Note over n_api,n_bucket: So'rov ichida transkodlash
n_api->>n_bucket: api-1: transcode v_9f3 to 720p started
Note over n_bucket,n_queue: Storage hodisasi, navbat, worker
n_bucket->>n_queue: PUBLISH media.uploaded
Note over n_viewer,n_bucket: Video viral bo'ldi
n_viewer->>n_bucket: GET /v/v_9f3/720p.mp4 HTTP/1.1
n_bucket-->>n_viewer: HTTP/1.1 200 OK
Note over n_viewer,n_cdn: Storage oldida CDN
n_viewer->>n_cdn: GET /v/v_9f3/720p.mp4 HTTP/1.1
n_cdn-->>n_viewer: HTTP/1.1 206 Partial ContentD2
direction: right
n_phone: "Telefon ilovasi" {
shape: rectangle
}
n_api: "API" {
shape: rectangle
}
n_bucket: "Object storage" {
shape: stored_data
}
n_db: "Baza" {
shape: cylinder
}
n_queue: "Navbat" {
shape: queue
}
n_worker: "Media worker" {
shape: rectangle
style.multiple: true
}
n_viewer: "Tomoshabinlar" {
shape: rectangle
}
n_cdn: "CDN" {
shape: oval
}
n_phone -> n_api
n_phone -> n_bucket
n_api -> n_db
n_bucket -> n_queue
n_queue -> n_worker
n_worker -> n_bucket
n_worker -> n_db
n_viewer -> n_cdn
n_cdn -> n_bucketPlantUML: so'rovlar ketma-ketligi
@startuml
participant "Telefon ilovasi" as n_phone
participant "API" as n_api
database "API diski" as n_disk
database "Object storage" as n_bucket
database "Baza" as n_db
queue "Navbat" as n_queue
participant "Tomoshabinlar" as n_viewer
participant "CDN" as n_cdn
note over n_phone, n_api : Har bayt API orqali o'tadi
n_phone -> n_api : POST /videos HTTP/1.1
n_api --> n_phone : HTTP/1.1 201 Created
note over n_api, n_disk : Disk to'ldi, boshqa serverda fayl yo'q
n_api ->x n_disk : [yo'qoldi] write /var/app/uploads/v_9f3.mp4
note over n_phone, n_api : API imzolaydi, telefon yuklaydi
n_phone -> n_api : POST /uploads HTTP/1.1
n_api --> n_phone : HTTP/1.1 200 OK
note over n_phone, n_bucket : Mobil tarmoq: 80% bo'ldi, boshidan
n_phone ->x n_bucket : [yo'qoldi] PUT /raw/v_9f3.mp4 HTTP/1.1
note over n_phone, n_bucket : Multipart: fayl 8 MB'lik qismlarda
n_phone -> n_bucket : PUT /raw/v_9f3.mp4?uploadId=up_31a&partNumber=17 HTTP/1.1
n_bucket --> n_phone : HTTP/1.1 200 OK
note over n_api, n_db : Bazada fayl emas, qator
n_api -> n_db : INSERT INTO videos
n_db --> n_api : INSERT 0 1
note over n_api, n_bucket : So'rov ichida transkodlash
n_api -> n_bucket : api-1: transcode v_9f3 to 720p started
note over n_bucket, n_queue : Storage hodisasi, navbat, worker
n_bucket -> n_queue : PUBLISH media.uploaded
note over n_viewer, n_bucket : Video viral bo'ldi
n_viewer -> n_bucket : GET /v/v_9f3/720p.mp4 HTTP/1.1
n_bucket --> n_viewer : HTTP/1.1 200 OK
note over n_viewer, n_cdn : Storage oldida CDN
n_viewer -> n_cdn : GET /v/v_9f3/720p.mp4 HTTP/1.1
n_cdn --> n_viewer : HTTP/1.1 206 Partial Content
@endumlPlantUML: komponentlar
@startuml
left to right direction
agent "Telefon ilovasi" as n_phone
component "API" as n_api
storage "Object storage" as n_bucket
database "Baza" as n_db
queue "Navbat" as n_queue
collections "Media worker" as n_worker
agent "Tomoshabinlar" as n_viewer
frame "CDN" as n_cdn
n_phone --> n_api
n_phone --> n_bucket
n_api --> n_db
n_bucket --> n_queue
n_queue --> n_worker
n_worker --> n_bucket
n_worker --> n_db
n_viewer --> n_cdn
n_cdn --> n_bucket
@endumlXulosa
Fayllar uchun besh qoida.
1) Baytlar API orqali o'tmaydi: u qisqa muddatli URL imzolaydi, klient object storage'ga yuklaydi. 2) Katta fayllar birma-bir qayta yuborsa bo'ladigan qismlarda boradi. 3) Bazada fayl emas, metama'lumot va obyekt kaliti. 4) Og'ir ishlov hodisa ishga tushiradigan worker'da. 5) Ko'rishlar CDN orqali, o'zgarmaydigan fayllar uzoq keshlanadi.
Ko'rsatkichlar: boshida va oxirida
| Ko'rsatkich | Boshida | Oxirida |
|---|---|---|
| Yuklash p95 | 94 s | 22 s |
| Yiqilgan yuklash | 7% | 0,4% |
| Oylik xarajat | 1 900 $ | 900 $ |
Qadamlar
- Muammo Har bayt API orqali o'tadi
Foydalanuvchi telefondan 180 MB video yuklaydi.
- Muammo Disk to'ldi, boshqa serverda fayl yo'q
Videolar API'ning o'z diskida yozuv yiqilguncha to'planadi.
- Yechim API imzolaydi, telefon yuklaydi
Fayllar endi object storage'da: katta obyektlar uchun qurilgan alohida servis, to'ladigan disk yo'q.
- Muammo Mobil tarmoq: 80% bo'ldi, boshidan
Bitta
PUTbutun 180 MB'ni olib ketadi. - Yechim Multipart: fayl 8 MB'lik qismlarda
Yuklash 8 MB'lik 23 qismga bo'linadi.
- Holat Bazada fayl emas, qator
Yuklash tugagach, telefon API'ga xabar beradi.
- Muammo So'rov ichida transkodlash
Tomoshabinlarga 720p versiya va thumbnail kerak.
- Yechim Storage hodisasi, navbat, worker
Endi yangi obyekt paydo bo'lganda storage hodisa e'lon qiladi.
- Muammo Video viral bo'ldi
Minglab tomoshabin bitta videoni ochadi, har biri uni bitta regiondagi storage'dan oladi.
- Yechim Storage oldida CDN
Tomoshabinlar endi videoni CDN'dan oladi: ko'p shaharlardagi serverlar nusxani foydalanuvchiga yaqin saqlaydi.
- Xulosa Xulosa
Fayllar uchun besh qoida.
Atamalar
- Presigned URL
- API imzolab beradigan qisqa muddatli havola, klient faylni u orqali to'g'ridan-to'g'ri object storage'ga yuklaydi. Baytlar API orqali o'tmaydi, API faqat kim va qancha hajm ekanini tekshiradi.