mufassall

Konspekt · Ma'lumot va saqlash

Fayl yuklash

13 daq · O'rta · mufassall.dtwin.uz/uz/l/files/

← Darsni ochish

Muammo

Telefonlar 180 MB'lik video yuklaydigan va minglab odam ko'radigan ilova. Nega baytlar API orqali o'tmasligi kerak va ishni presigned URL, multipart upload, worker va CDN qanday bo'lishib oladi.

Nimani o'rganasiz

  • Nega ilova serveri orqali yuklash sekin va uning diskini to'ldiradi
  • Presigned URL: API imzolaydi, telefon to'g'ridan-to'g'ri storage'ga yuklaydi
  • Multipart upload: butun fayl emas, faqat yiqilgan qism qayta yuboriladi
  • Thumbnail va transkodlash worker'da, tarqatish CDN orqali, bazada faqat metama'lumot
Yakuniy sahna
12%20%40%60%

Diagramma kod sifatida

Darsdagi yakuniy sxemani matn ko'rinishida oling va Jira, GitLab, GitHub, Notion yoki hujjatga qo'ying: Mermaid u yerlarda o'zi chiziladi. D2 va PlantUML (Confluence, IntelliJ) uchun alohida variantlar bor.

Mermaid: sxema

flowchart LR
  n_phone(["Telefon ilovasi"])
  n_api["API"]
  n_bucket[("Object storage")]
  n_db[("Baza")]
  n_queue[["Navbat"]]
  n_worker("Media worker")
  n_viewer(["Tomoshabinlar"])
  n_cdn[/"CDN"\]
  n_phone --> n_api
  n_phone --> n_bucket
  n_api --> n_db
  n_bucket --> n_queue
  n_queue --> n_worker
  n_worker --> n_bucket
  n_worker --> n_db
  n_viewer --> n_cdn
  n_cdn --> n_bucket

Mermaid: so'rovlar ketma-ketligi

sequenceDiagram
  participant n_phone as Telefon ilovasi
  participant n_api as API
  participant n_disk as API diski
  participant n_bucket as Object storage
  participant n_db as Baza
  participant n_queue as Navbat
  participant n_viewer as Tomoshabinlar
  participant n_cdn as CDN
  Note over n_phone,n_api: Har bayt API orqali o'tadi
  n_phone->>n_api: POST /videos HTTP/1.1
  n_api-->>n_phone: HTTP/1.1 201 Created
  Note over n_api,n_disk: Disk to'ldi, boshqa serverda fayl yo'q
  n_api-xn_disk: write /var/app/uploads/v_9f3.mp4
  Note over n_phone,n_api: API imzolaydi, telefon yuklaydi
  n_phone->>n_api: POST /uploads HTTP/1.1
  n_api-->>n_phone: HTTP/1.1 200 OK
  Note over n_phone,n_bucket: Mobil tarmoq: 80% bo'ldi, boshidan
  n_phone-xn_bucket: PUT /raw/v_9f3.mp4 HTTP/1.1
  Note over n_phone,n_bucket: Multipart: fayl 8 MB'lik qismlarda
  n_phone->>n_bucket: PUT /raw/v_9f3.mp4?uploadId=up_31a#38;partNumber=17 HTTP/1.1
  n_bucket-->>n_phone: HTTP/1.1 200 OK
  Note over n_api,n_db: Bazada fayl emas, qator
  n_api->>n_db: INSERT INTO videos
  n_db-->>n_api: INSERT 0 1
  Note over n_api,n_bucket: So'rov ichida transkodlash
  n_api->>n_bucket: api-1: transcode v_9f3 to 720p started
  Note over n_bucket,n_queue: Storage hodisasi, navbat, worker
  n_bucket->>n_queue: PUBLISH media.uploaded
  Note over n_viewer,n_bucket: Video viral bo'ldi
  n_viewer->>n_bucket: GET /v/v_9f3/720p.mp4 HTTP/1.1
  n_bucket-->>n_viewer: HTTP/1.1 200 OK
  Note over n_viewer,n_cdn: Storage oldida CDN
  n_viewer->>n_cdn: GET /v/v_9f3/720p.mp4 HTTP/1.1
  n_cdn-->>n_viewer: HTTP/1.1 206 Partial Content

D2

direction: right

n_phone: "Telefon ilovasi" {
  shape: rectangle
}
n_api: "API" {
  shape: rectangle
}
n_bucket: "Object storage" {
  shape: stored_data
}
n_db: "Baza" {
  shape: cylinder
}
n_queue: "Navbat" {
  shape: queue
}
n_worker: "Media worker" {
  shape: rectangle
  style.multiple: true
}
n_viewer: "Tomoshabinlar" {
  shape: rectangle
}
n_cdn: "CDN" {
  shape: oval
}

n_phone -> n_api
n_phone -> n_bucket
n_api -> n_db
n_bucket -> n_queue
n_queue -> n_worker
n_worker -> n_bucket
n_worker -> n_db
n_viewer -> n_cdn
n_cdn -> n_bucket

PlantUML: so'rovlar ketma-ketligi

@startuml
participant "Telefon ilovasi" as n_phone
participant "API" as n_api
database "API diski" as n_disk
database "Object storage" as n_bucket
database "Baza" as n_db
queue "Navbat" as n_queue
participant "Tomoshabinlar" as n_viewer
participant "CDN" as n_cdn
note over n_phone, n_api : Har bayt API orqali o'tadi
n_phone -> n_api : POST /videos HTTP/1.1
n_api --> n_phone : HTTP/1.1 201 Created
note over n_api, n_disk : Disk to'ldi, boshqa serverda fayl yo'q
n_api ->x n_disk : [yo'qoldi] write /var/app/uploads/v_9f3.mp4
note over n_phone, n_api : API imzolaydi, telefon yuklaydi
n_phone -> n_api : POST /uploads HTTP/1.1
n_api --> n_phone : HTTP/1.1 200 OK
note over n_phone, n_bucket : Mobil tarmoq: 80% bo'ldi, boshidan
n_phone ->x n_bucket : [yo'qoldi] PUT /raw/v_9f3.mp4 HTTP/1.1
note over n_phone, n_bucket : Multipart: fayl 8 MB'lik qismlarda
n_phone -> n_bucket : PUT /raw/v_9f3.mp4?uploadId=up_31a&partNumber=17 HTTP/1.1
n_bucket --> n_phone : HTTP/1.1 200 OK
note over n_api, n_db : Bazada fayl emas, qator
n_api -> n_db : INSERT INTO videos
n_db --> n_api : INSERT 0 1
note over n_api, n_bucket : So'rov ichida transkodlash
n_api -> n_bucket : api-1: transcode v_9f3 to 720p started
note over n_bucket, n_queue : Storage hodisasi, navbat, worker
n_bucket -> n_queue : PUBLISH media.uploaded
note over n_viewer, n_bucket : Video viral bo'ldi
n_viewer -> n_bucket : GET /v/v_9f3/720p.mp4 HTTP/1.1
n_bucket --> n_viewer : HTTP/1.1 200 OK
note over n_viewer, n_cdn : Storage oldida CDN
n_viewer -> n_cdn : GET /v/v_9f3/720p.mp4 HTTP/1.1
n_cdn --> n_viewer : HTTP/1.1 206 Partial Content
@enduml

PlantUML: komponentlar

@startuml
left to right direction
agent "Telefon ilovasi" as n_phone
component "API" as n_api
storage "Object storage" as n_bucket
database "Baza" as n_db
queue "Navbat" as n_queue
collections "Media worker" as n_worker
agent "Tomoshabinlar" as n_viewer
frame "CDN" as n_cdn
n_phone --> n_api
n_phone --> n_bucket
n_api --> n_db
n_bucket --> n_queue
n_queue --> n_worker
n_worker --> n_bucket
n_worker --> n_db
n_viewer --> n_cdn
n_cdn --> n_bucket
@enduml

Xulosa

Fayllar uchun besh qoida.

1) Baytlar API orqali o'tmaydi: u qisqa muddatli URL imzolaydi, klient object storage'ga yuklaydi. 2) Katta fayllar birma-bir qayta yuborsa bo'ladigan qismlarda boradi. 3) Bazada fayl emas, metama'lumot va obyekt kaliti. 4) Og'ir ishlov hodisa ishga tushiradigan worker'da. 5) Ko'rishlar CDN orqali, o'zgarmaydigan fayllar uzoq keshlanadi.

Ko'rsatkichlar: boshida va oxirida

Ko'rsatkichBoshidaOxirida
Yuklash p9594 s22 s
Yiqilgan yuklash7%0,4%
Oylik xarajat1 900 $900 $

Qadamlar

  1. Muammo Har bayt API orqali o'tadi

    Foydalanuvchi telefondan 180 MB video yuklaydi.

  2. Muammo Disk to'ldi, boshqa serverda fayl yo'q

    Videolar API'ning o'z diskida yozuv yiqilguncha to'planadi.

  3. Yechim API imzolaydi, telefon yuklaydi

    Fayllar endi object storage'da: katta obyektlar uchun qurilgan alohida servis, to'ladigan disk yo'q.

  4. Muammo Mobil tarmoq: 80% bo'ldi, boshidan

    Bitta PUT butun 180 MB'ni olib ketadi.

  5. Yechim Multipart: fayl 8 MB'lik qismlarda

    Yuklash 8 MB'lik 23 qismga bo'linadi.

  6. Holat Bazada fayl emas, qator

    Yuklash tugagach, telefon API'ga xabar beradi.

  7. Muammo So'rov ichida transkodlash

    Tomoshabinlarga 720p versiya va thumbnail kerak.

  8. Yechim Storage hodisasi, navbat, worker

    Endi yangi obyekt paydo bo'lganda storage hodisa e'lon qiladi.

  9. Muammo Video viral bo'ldi

    Minglab tomoshabin bitta videoni ochadi, har biri uni bitta regiondagi storage'dan oladi.

  10. Yechim Storage oldida CDN

    Tomoshabinlar endi videoni CDN'dan oladi: ko'p shaharlardagi serverlar nusxani foydalanuvchiga yaqin saqlaydi.

  11. Xulosa Xulosa

    Fayllar uchun besh qoida.

Atamalar

Presigned URL
API imzolab beradigan qisqa muddatli havola, klient faylni u orqali to'g'ridan-to'g'ri object storage'ga yuklaydi. Baytlar API orqali o'tmaydi, API faqat kim va qancha hajm ekanini tekshiradi.