mufassall

Konspekt · Butun tizimlar

Yangiliklar lentasi loyihasi

15 daq · Yuqori · mufassall.dtwin.uz/uz/l/design-news-feed/

← Darsni ochish

Muammo

Odamlar post yozadi, ularga obuna bo'lganlarning hammasi uni bir necha soniyada lentasida ko'rishi kerak. Lentani qadamma-qadam quramiz: o'qishda yig'ish, yozishda tarqatish, mashhurlar muammosi va gibrid, ranking, CDN'dagi media va lenta keshi o'lganda nima bo'ladi.

Nimani o'rganasiz

  • Fan-out on read va fan-out on write, har birining narxi
  • Nega bir nechta ulkan akkaunt push'ni buzadi va gibrid uni qanday tuzatadi
  • Postlarni emas, id'larni saqlaydigan har foydalanuvchi lenta keshi
  • Cursor bilan sahifalash, ranking va CDN'dagi media
Yakuniy sahna
20%55%55%35%40%50%10%

Diagramma kod sifatida

Darsdagi yakuniy sxemani matn ko'rinishida oling va Jira, GitLab, GitHub, Notion yoki hujjatga qo'ying: Mermaid u yerlarda o'zi chiziladi. D2 va PlantUML (Confluence, IntelliJ) uchun alohida variantlar bor.

Mermaid: sxema

flowchart LR
  n_author(["Muallif ilovasi"])
  n_reader(["O'quvchi ilovasi"])
  n_api["Lenta API"]
  n_db[("Postlar va obunalar bazasi")]
  n_queue[["Postlar navbati"]]
  n_fanout("Fan-out worker")
  n_feeds{{"Lenta keshi"}}
  n_cdn[/"CDN"\]
  n_media[("Media ombori")]
  n_author --> n_api
  n_api --> n_db
  n_reader --> n_api
  n_api --> n_queue
  n_queue --> n_fanout
  n_fanout --> n_db
  n_fanout --> n_feeds
  n_api --> n_feeds
  n_api --> n_db
  n_reader --> n_cdn
  n_cdn --> n_media

Mermaid: so'rovlar ketma-ketligi

sequenceDiagram
  participant n_author as Muallif ilovasi
  participant n_api as Lenta API
  participant n_db as Postlar va obunalar bazasi
  participant n_fanout as Fan-out worker
  participant n_feeds as Lenta keshi
  participant n_queue as Postlar navbati
  participant n_reader as O'quvchi ilovasi
  participant n_cdn as CDN
  Note over n_author,n_api: Post yozish: bitta qator, bitta id
  n_author->>n_api: POST /v1/posts HTTP/1.1
  n_api-->>n_author: HTTP/1.1 201 Created
  Note over n_api,n_db: Fan-out on read: lenta ochilganda yig'iladi
  n_api->>n_db: SELECT p.id, p.author_id, p.created_at
  n_db-->>n_api: (50 rows)
  Note over n_api,n_db: Katta hajmda pull: har safar hamma ishni baza qiladi
  n_api->>n_db: EXPLAIN ANALYZE SELECT p.id ...
  n_db-->>n_api: Sort (top-N heapsort)
  Note over n_fanout,n_feeds: Fan-out on write: post yozilganda tarqatamiz
  n_fanout->>n_feeds: LPUSH feed:u_42 7393018847102976
  n_feeds-->>n_fanout: (integer) 801
  Note over n_api,n_feeds: Lentani o'qish endi bitta oraliq o'qishi
  n_api->>n_feeds: LRANGE feed:u_42 0 49
  n_feeds-->>n_api: 1) #34;7393018847102976#34;
  Note over n_queue,n_fanout: Mashhur odam post yozdi: 30 million yozish
  n_queue->>n_fanout: fanout post=7393102255110144 author=u_star
  Note over n_api,n_db: Gibrid: ko'pchilikka push, mashhurlarga pull
  n_api->>n_db: SELECT id, author_id, created_at FROM posts
  n_db-->>n_api: (7 rows)
  Note over n_reader,n_api: Ranking va cursor
  n_reader->>n_api: GET /v1/feed?limit=20#38;cursor=c_7f2a HTTP/1.1
  n_api-->>n_reader: HTTP/1.1 200 OK
  Note over n_reader,n_cdn: Rasmlar API'dan emas, CDN'dan
  n_reader->>n_cdn: GET /m/m_5521/720.webp HTTP/1.1
  n_cdn-->>n_reader: HTTP/1.1 200 OK
  Note over n_api,n_feeds: Lenta keshining bitta node'i o'ldi
  n_api->>n_feeds: LRANGE feed:u_42 0 49
  n_feeds-->>n_api: (error) connection refused

D2

direction: right

n_author: "Muallif ilovasi" {
  shape: rectangle
}
n_reader: "O'quvchi ilovasi" {
  shape: rectangle
}
n_api: "Lenta API" {
  shape: rectangle
}
n_db: "Postlar va obunalar bazasi" {
  shape: cylinder
}
n_queue: "Postlar navbati" {
  shape: queue
}
n_fanout: "Fan-out worker" {
  shape: rectangle
  style.multiple: true
}
n_feeds: "Lenta keshi" {
  shape: hexagon
}
n_cdn: "CDN" {
  shape: oval
}
n_media: "Media ombori" {
  shape: stored_data
}

n_author -> n_api
n_api -> n_db
n_reader -> n_api
n_api -> n_queue
n_queue -> n_fanout
n_fanout -> n_db
n_fanout -> n_feeds
n_api -> n_feeds
n_api -> n_db
n_reader -> n_cdn
n_cdn -> n_media

PlantUML: so'rovlar ketma-ketligi

@startuml
participant "Muallif ilovasi" as n_author
participant "Lenta API" as n_api
database "Postlar va obunalar bazasi" as n_db
participant "Fan-out worker" as n_fanout
collections "Lenta keshi" as n_feeds
queue "Postlar navbati" as n_queue
participant "O'quvchi ilovasi" as n_reader
participant "CDN" as n_cdn
note over n_author, n_api : Post yozish: bitta qator, bitta id
n_author -> n_api : POST /v1/posts HTTP/1.1
n_api --> n_author : HTTP/1.1 201 Created
note over n_api, n_db : Fan-out on read: lenta ochilganda yig'iladi
n_api -> n_db : SELECT p.id, p.author_id, p.created_at
n_db --> n_api : (50 rows)
note over n_api, n_db : Katta hajmda pull: har safar hamma ishni baza qiladi
n_api -> n_db : EXPLAIN ANALYZE SELECT p.id ...
n_db --> n_api : Sort (top-N heapsort)
note over n_fanout, n_feeds : Fan-out on write: post yozilganda tarqatamiz
n_fanout -> n_feeds : LPUSH feed:u_42 7393018847102976
n_feeds --> n_fanout : (integer) 801
note over n_api, n_feeds : Lentani o'qish endi bitta oraliq o'qishi
n_api -> n_feeds : LRANGE feed:u_42 0 49
n_feeds --> n_api : 1) "7393018847102976"
note over n_queue, n_fanout : Mashhur odam post yozdi: 30 million yozish
n_queue -> n_fanout : fanout post=7393102255110144 author=u_star
note over n_api, n_db : Gibrid: ko'pchilikka push, mashhurlarga pull
n_api -> n_db : SELECT id, author_id, created_at FROM posts
n_db --> n_api : (7 rows)
note over n_reader, n_api : Ranking va cursor
n_reader -> n_api : GET /v1/feed?limit=20&cursor=c_7f2a HTTP/1.1
n_api --> n_reader : HTTP/1.1 200 OK
note over n_reader, n_cdn : Rasmlar API'dan emas, CDN'dan
n_reader -> n_cdn : GET /m/m_5521/720.webp HTTP/1.1
n_cdn --> n_reader : HTTP/1.1 200 OK
note over n_api, n_feeds : Lenta keshining bitta node'i o'ldi
n_api -> n_feeds : LRANGE feed:u_42 0 49
n_feeds --> n_api : (error) connection refused
@enduml

PlantUML: komponentlar

@startuml
left to right direction
agent "Muallif ilovasi" as n_author
agent "O'quvchi ilovasi" as n_reader
component "Lenta API" as n_api
database "Postlar va obunalar bazasi" as n_db
queue "Postlar navbati" as n_queue
collections "Fan-out worker" as n_fanout
stack "Lenta keshi" as n_feeds
frame "CDN" as n_cdn
storage "Media ombori" as n_media
n_author --> n_api
n_api --> n_db
n_reader --> n_api
n_api --> n_queue
n_queue --> n_fanout
n_fanout --> n_db
n_fanout --> n_feeds
n_api --> n_feeds
n_api --> n_db
n_reader --> n_cdn
n_cdn --> n_media
@enduml

Xulosa

Lenta bu yozishdagi ish va o'qishdagi ish orasidagi savdo.

1) Pull oddiy, lekin har ochishda har lentani qayta quradi. 2) Push o'qishni bitta kesh oralig'iga aylantiradi, narxi yozishda har obunachiga. 3) Katta akkauntlar o'qishda pull qilinib birlashtiriladi. 4) Lenta ro'yxatlari id saqlaydi, cheklangan va doim qayta qurilishi mumkin. 5) Bir necha yuz nomzodni rank qiling, cursor bilan sahifalang, mediani CDN'dan bering.

Ko'rsatkichlar: boshida va oxirida

Ko'rsatkichBoshidaOxirida
Lenta yuklash p9980 ms60 ms
Lentaga yozishlar0/s1 800/s
Obunachilarga yetish vaqti0 s2 s

Qadamlar

  1. Holat Post yozish: bitta qator, bitta id

    Muallif matn va oldinroq yuklangan rasmlar id'larini yuboradi.

  2. Holat Fan-out on read: lenta ochilganda yig'iladi

    Eng oddiy lenta: o'quvchi ilovani ochganda u obuna bo'lgan hammani topamiz, ularning yangi postlarini olamiz, vaqt bo'yicha saralab 50 tasini qaytaramiz.

  3. Muammo Katta hajmda pull: har safar hamma ishni baza qiladi

    20 million o'quvchi lentani kuniga ~10 marta ochadi: o'rtacha 2 300 lenta/s, kechki cho'qqida ~10 000.

  4. Yechim Fan-out on write: post yozilganda tarqatamiz

    Teskarisi: kimdir post yozsa, navbat uni fan-out worker'ga beradi.

  5. Holat Lentani o'qish endi bitta oraliq o'qishi

    API feed:u_42 ning birinchi 50 id'sini o'qiydi, keyin shu postlarni id bo'yicha post keshidan oladi (ko'pi issiq) va qaytaradi.

  6. Muammo Mashhur odam post yozdi: 30 million yozish

    30 million obunachisi bor akkaunt post yozdi.

  7. Yechim Gibrid: ko'pchilikka push, mashhurlarga pull

    100 000 dan ortiq obunachili akkauntlar fan-out qilinmaydi.

  8. Holat Ranking va cursor

    Qat'iy vaqt tartibi yaqin do'stlarning postlarini shovqin ostida ko'mib yuboradi.

  9. Yechim Rasmlar API'dan emas, CDN'dan

    Baytlarning ko'pi rasmlar.

  10. Muammo Lenta keshining bitta node'i o'ldi

    Lenta keshining bitta node'i ishdan chiqdi.

  11. Yechim Pull'ga qaytish, fonda qayta qurish

    Ro'yxati yo'q foydalanuvchilar uchun API pull'ga qaytadi: obuna bo'lingan mualliflarning faqat oxirgi ikki kunlik postlari.

  12. Xulosa Xulosa

    Lenta bu yozishdagi ish va o'qishdagi ish orasidagi savdo.

Atamalar

Fan-out (on write / on read)
Fan-out on write post yozilganda uning id'sini har obunachining lenta keshiga qo'yadi, fan-out on read esa lentani ochilganda yig'adi. Ko'pchilikka push, juda mashhur akkauntlarga pull ishlatiladigan gibrid eng yaxshi ishlaydi.