Konspekt · Butun tizimlar
Yangiliklar lentasi loyihasi
Muammo
Odamlar post yozadi, ularga obuna bo'lganlarning hammasi uni bir necha soniyada lentasida ko'rishi kerak. Lentani qadamma-qadam quramiz: o'qishda yig'ish, yozishda tarqatish, mashhurlar muammosi va gibrid, ranking, CDN'dagi media va lenta keshi o'lganda nima bo'ladi.
Nimani o'rganasiz
- Fan-out on read va fan-out on write, har birining narxi
- Nega bir nechta ulkan akkaunt push'ni buzadi va gibrid uni qanday tuzatadi
- Postlarni emas, id'larni saqlaydigan har foydalanuvchi lenta keshi
- Cursor bilan sahifalash, ranking va CDN'dagi media
Diagramma kod sifatida
Darsdagi yakuniy sxemani matn ko'rinishida oling va Jira, GitLab, GitHub, Notion yoki hujjatga qo'ying: Mermaid u yerlarda o'zi chiziladi. D2 va PlantUML (Confluence, IntelliJ) uchun alohida variantlar bor.
Mermaid: sxema
flowchart LR
n_author(["Muallif ilovasi"])
n_reader(["O'quvchi ilovasi"])
n_api["Lenta API"]
n_db[("Postlar va obunalar bazasi")]
n_queue[["Postlar navbati"]]
n_fanout("Fan-out worker")
n_feeds{{"Lenta keshi"}}
n_cdn[/"CDN"\]
n_media[("Media ombori")]
n_author --> n_api
n_api --> n_db
n_reader --> n_api
n_api --> n_queue
n_queue --> n_fanout
n_fanout --> n_db
n_fanout --> n_feeds
n_api --> n_feeds
n_api --> n_db
n_reader --> n_cdn
n_cdn --> n_mediaMermaid: so'rovlar ketma-ketligi
sequenceDiagram
participant n_author as Muallif ilovasi
participant n_api as Lenta API
participant n_db as Postlar va obunalar bazasi
participant n_fanout as Fan-out worker
participant n_feeds as Lenta keshi
participant n_queue as Postlar navbati
participant n_reader as O'quvchi ilovasi
participant n_cdn as CDN
Note over n_author,n_api: Post yozish: bitta qator, bitta id
n_author->>n_api: POST /v1/posts HTTP/1.1
n_api-->>n_author: HTTP/1.1 201 Created
Note over n_api,n_db: Fan-out on read: lenta ochilganda yig'iladi
n_api->>n_db: SELECT p.id, p.author_id, p.created_at
n_db-->>n_api: (50 rows)
Note over n_api,n_db: Katta hajmda pull: har safar hamma ishni baza qiladi
n_api->>n_db: EXPLAIN ANALYZE SELECT p.id ...
n_db-->>n_api: Sort (top-N heapsort)
Note over n_fanout,n_feeds: Fan-out on write: post yozilganda tarqatamiz
n_fanout->>n_feeds: LPUSH feed:u_42 7393018847102976
n_feeds-->>n_fanout: (integer) 801
Note over n_api,n_feeds: Lentani o'qish endi bitta oraliq o'qishi
n_api->>n_feeds: LRANGE feed:u_42 0 49
n_feeds-->>n_api: 1) #34;7393018847102976#34;
Note over n_queue,n_fanout: Mashhur odam post yozdi: 30 million yozish
n_queue->>n_fanout: fanout post=7393102255110144 author=u_star
Note over n_api,n_db: Gibrid: ko'pchilikka push, mashhurlarga pull
n_api->>n_db: SELECT id, author_id, created_at FROM posts
n_db-->>n_api: (7 rows)
Note over n_reader,n_api: Ranking va cursor
n_reader->>n_api: GET /v1/feed?limit=20#38;cursor=c_7f2a HTTP/1.1
n_api-->>n_reader: HTTP/1.1 200 OK
Note over n_reader,n_cdn: Rasmlar API'dan emas, CDN'dan
n_reader->>n_cdn: GET /m/m_5521/720.webp HTTP/1.1
n_cdn-->>n_reader: HTTP/1.1 200 OK
Note over n_api,n_feeds: Lenta keshining bitta node'i o'ldi
n_api->>n_feeds: LRANGE feed:u_42 0 49
n_feeds-->>n_api: (error) connection refusedD2
direction: right
n_author: "Muallif ilovasi" {
shape: rectangle
}
n_reader: "O'quvchi ilovasi" {
shape: rectangle
}
n_api: "Lenta API" {
shape: rectangle
}
n_db: "Postlar va obunalar bazasi" {
shape: cylinder
}
n_queue: "Postlar navbati" {
shape: queue
}
n_fanout: "Fan-out worker" {
shape: rectangle
style.multiple: true
}
n_feeds: "Lenta keshi" {
shape: hexagon
}
n_cdn: "CDN" {
shape: oval
}
n_media: "Media ombori" {
shape: stored_data
}
n_author -> n_api
n_api -> n_db
n_reader -> n_api
n_api -> n_queue
n_queue -> n_fanout
n_fanout -> n_db
n_fanout -> n_feeds
n_api -> n_feeds
n_api -> n_db
n_reader -> n_cdn
n_cdn -> n_mediaPlantUML: so'rovlar ketma-ketligi
@startuml
participant "Muallif ilovasi" as n_author
participant "Lenta API" as n_api
database "Postlar va obunalar bazasi" as n_db
participant "Fan-out worker" as n_fanout
collections "Lenta keshi" as n_feeds
queue "Postlar navbati" as n_queue
participant "O'quvchi ilovasi" as n_reader
participant "CDN" as n_cdn
note over n_author, n_api : Post yozish: bitta qator, bitta id
n_author -> n_api : POST /v1/posts HTTP/1.1
n_api --> n_author : HTTP/1.1 201 Created
note over n_api, n_db : Fan-out on read: lenta ochilganda yig'iladi
n_api -> n_db : SELECT p.id, p.author_id, p.created_at
n_db --> n_api : (50 rows)
note over n_api, n_db : Katta hajmda pull: har safar hamma ishni baza qiladi
n_api -> n_db : EXPLAIN ANALYZE SELECT p.id ...
n_db --> n_api : Sort (top-N heapsort)
note over n_fanout, n_feeds : Fan-out on write: post yozilganda tarqatamiz
n_fanout -> n_feeds : LPUSH feed:u_42 7393018847102976
n_feeds --> n_fanout : (integer) 801
note over n_api, n_feeds : Lentani o'qish endi bitta oraliq o'qishi
n_api -> n_feeds : LRANGE feed:u_42 0 49
n_feeds --> n_api : 1) "7393018847102976"
note over n_queue, n_fanout : Mashhur odam post yozdi: 30 million yozish
n_queue -> n_fanout : fanout post=7393102255110144 author=u_star
note over n_api, n_db : Gibrid: ko'pchilikka push, mashhurlarga pull
n_api -> n_db : SELECT id, author_id, created_at FROM posts
n_db --> n_api : (7 rows)
note over n_reader, n_api : Ranking va cursor
n_reader -> n_api : GET /v1/feed?limit=20&cursor=c_7f2a HTTP/1.1
n_api --> n_reader : HTTP/1.1 200 OK
note over n_reader, n_cdn : Rasmlar API'dan emas, CDN'dan
n_reader -> n_cdn : GET /m/m_5521/720.webp HTTP/1.1
n_cdn --> n_reader : HTTP/1.1 200 OK
note over n_api, n_feeds : Lenta keshining bitta node'i o'ldi
n_api -> n_feeds : LRANGE feed:u_42 0 49
n_feeds --> n_api : (error) connection refused
@endumlPlantUML: komponentlar
@startuml
left to right direction
agent "Muallif ilovasi" as n_author
agent "O'quvchi ilovasi" as n_reader
component "Lenta API" as n_api
database "Postlar va obunalar bazasi" as n_db
queue "Postlar navbati" as n_queue
collections "Fan-out worker" as n_fanout
stack "Lenta keshi" as n_feeds
frame "CDN" as n_cdn
storage "Media ombori" as n_media
n_author --> n_api
n_api --> n_db
n_reader --> n_api
n_api --> n_queue
n_queue --> n_fanout
n_fanout --> n_db
n_fanout --> n_feeds
n_api --> n_feeds
n_api --> n_db
n_reader --> n_cdn
n_cdn --> n_media
@endumlXulosa
Lenta bu yozishdagi ish va o'qishdagi ish orasidagi savdo.
1) Pull oddiy, lekin har ochishda har lentani qayta quradi. 2) Push o'qishni bitta kesh oralig'iga aylantiradi, narxi yozishda har obunachiga. 3) Katta akkauntlar o'qishda pull qilinib birlashtiriladi. 4) Lenta ro'yxatlari id saqlaydi, cheklangan va doim qayta qurilishi mumkin. 5) Bir necha yuz nomzodni rank qiling, cursor bilan sahifalang, mediani CDN'dan bering.
Ko'rsatkichlar: boshida va oxirida
| Ko'rsatkich | Boshida | Oxirida |
|---|---|---|
| Lenta yuklash p99 | 80 ms | 60 ms |
| Lentaga yozishlar | 0/s | 1 800/s |
| Obunachilarga yetish vaqti | 0 s | 2 s |
Qadamlar
- Holat Post yozish: bitta qator, bitta id
Muallif matn va oldinroq yuklangan rasmlar id'larini yuboradi.
- Holat Fan-out on read: lenta ochilganda yig'iladi
Eng oddiy lenta: o'quvchi ilovani ochganda u obuna bo'lgan hammani topamiz, ularning yangi postlarini olamiz, vaqt bo'yicha saralab 50 tasini qaytaramiz.
- Muammo Katta hajmda pull: har safar hamma ishni baza qiladi
20 million o'quvchi lentani kuniga ~10 marta ochadi: o'rtacha 2 300 lenta/s, kechki cho'qqida ~10 000.
- Yechim Fan-out on write: post yozilganda tarqatamiz
Teskarisi: kimdir post yozsa, navbat uni fan-out worker'ga beradi.
- Holat Lentani o'qish endi bitta oraliq o'qishi
API
feed:u_42ning birinchi 50 id'sini o'qiydi, keyin shu postlarni id bo'yicha post keshidan oladi (ko'pi issiq) va qaytaradi. - Muammo Mashhur odam post yozdi: 30 million yozish
30 million obunachisi bor akkaunt post yozdi.
- Yechim Gibrid: ko'pchilikka push, mashhurlarga pull
100 000 dan ortiq obunachili akkauntlar fan-out qilinmaydi.
- Holat Ranking va cursor
Qat'iy vaqt tartibi yaqin do'stlarning postlarini shovqin ostida ko'mib yuboradi.
- Yechim Rasmlar API'dan emas, CDN'dan
Baytlarning ko'pi rasmlar.
- Muammo Lenta keshining bitta node'i o'ldi
Lenta keshining bitta node'i ishdan chiqdi.
- Yechim Pull'ga qaytish, fonda qayta qurish
Ro'yxati yo'q foydalanuvchilar uchun API pull'ga qaytadi: obuna bo'lingan mualliflarning faqat oxirgi ikki kunlik postlari.
- Xulosa Xulosa
Lenta bu yozishdagi ish va o'qishdagi ish orasidagi savdo.
Atamalar
- Fan-out (on write / on read)
- Fan-out on write post yozilganda uning id'sini har obunachining lenta keshiga qo'yadi, fan-out on read esa lentani ochilganda yig'adi. Ko'pchilikka push, juda mashhur akkauntlarga pull ishlatiladigan gibrid eng yaxshi ishlaydi.