Cheat sheet · Architecture
Service discovery
The problem
An orders service calls inventory instances that start, stop and move. How a registry with heartbeats replaces hard-coded IPs, and the three ways to ask it: from the client, through DNS, or through a load balancer.
What you learn
- Why hard-coded addresses break as soon as instances move
- How instances register and how heartbeats remove dead ones
- Client-side discovery vs server-side discovery
- DNS-based discovery and what its TTL costs
Diagram as code
The lesson's final design as text: paste it into Jira, GitLab, GitHub, Notion or your docs, where Mermaid renders by itself. There are D2 and PlantUML (Confluence, IntelliJ) versions too.
Mermaid: flowchart
flowchart LR
n_orders["Orders service"]
n_inv_a["Inventory A"]
n_inv_c["Inventory C"]
n_registry[("Service registry")]
n_inv_d["Inventory D"]
n_lb{"Load balancer"}
n_inv_c -.-> n_registry
n_inv_d -.-> n_registry
n_orders --> n_lb
n_lb --> n_inv_c
n_lb --> n_inv_d
n_registry -.-> n_lb
classDef down fill:#fde2e2,stroke:#c62828,color:#4a0d0d,stroke-dasharray:4 3
class n_inv_a downMermaid: request sequence
sequenceDiagram
participant n_orders as Orders service
participant n_inv_a as Inventory A
participant n_inv_b as Inventory B
participant n_inv_c as Inventory C
participant n_registry as Service registry
participant n_lb as Load balancer
Note over n_orders,n_inv_a: Two IPs in a config file
n_orders->>n_inv_a: GET /v1/stock/sku_19 HTTP/1.1
n_inv_a-->>n_orders: HTTP/1.1 200 OK
Note over n_orders,n_inv_b: An instance moved#59; the config did not
n_orders-xn_inv_b: GET /v1/stock/sku_19 HTTP/1.1
Note over n_inv_c,n_registry: Instances register themselves
n_inv_c->>n_registry: PUT /v1/agent/service/register HTTP/1.1
n_registry-->>n_inv_c: HTTP/1.1 200 OK
Note over n_orders,n_registry: Client-side discovery
n_orders->>n_registry: GET /v1/health/service/inventory?passing=true HTTP/1.1
n_registry-->>n_orders: HTTP/1.1 200 OK
Note over n_orders,n_inv_a: A crashed instance stays on the list
n_orders-xn_inv_a: GET /v1/stock/sku_19 HTTP/1.1
Note over n_inv_c,n_registry: Heartbeats and health checks
n_inv_c->>n_registry: PUT /v1/agent/check/pass/service:inv-c HTTP/1.1
n_registry-->>n_inv_c: HTTP/1.1 200 OK
Note over n_orders,n_registry: DNS-based discovery
n_orders->>n_registry: dig @registry.internal -p 8600 inventory.service.internal A
n_registry-->>n_orders: #59;#59; ANSWER SECTION:
Note over n_orders,n_lb: Server-side discovery: let a load balancer ask
n_orders->>n_lb: GET /v1/stock/sku_19 HTTP/1.1
n_lb-->>n_orders: HTTP/1.1 200 OKD2
direction: right
n_orders: "Orders service" {
shape: rectangle
}
n_inv_a: "Inventory A" {
shape: rectangle
style.fill: "#fde2e2"
style.stroke: "#c62828"
style.stroke-dash: 3
}
n_inv_c: "Inventory C" {
shape: rectangle
}
n_registry: "Service registry" {
shape: cylinder
}
n_inv_d: "Inventory D" {
shape: rectangle
}
n_lb: "Load balancer" {
shape: diamond
}
n_inv_c -> n_registry {
style.stroke-dash: 3
}
n_inv_d -> n_registry {
style.stroke-dash: 3
}
n_orders -> n_lb
n_lb -> n_inv_c
n_lb -> n_inv_d
n_registry -> n_lb {
style.stroke-dash: 3
}PlantUML: request sequence
@startuml
participant "Orders service" as n_orders
participant "Inventory A" as n_inv_a
participant "Inventory B" as n_inv_b
participant "Inventory C" as n_inv_c
database "Service registry" as n_registry
participant "Load balancer" as n_lb
note over n_orders, n_inv_a : Two IPs in a config file
n_orders -> n_inv_a : GET /v1/stock/sku_19 HTTP/1.1
n_inv_a --> n_orders : HTTP/1.1 200 OK
note over n_orders, n_inv_b : An instance moved; the config did not
n_orders ->x n_inv_b : [lost] GET /v1/stock/sku_19 HTTP/1.1
note over n_inv_c, n_registry : Instances register themselves
n_inv_c -> n_registry : PUT /v1/agent/service/register HTTP/1.1
n_registry --> n_inv_c : HTTP/1.1 200 OK
note over n_orders, n_registry : Client-side discovery
n_orders -> n_registry : GET /v1/health/service/inventory?passing=true HTTP/1.1
n_registry --> n_orders : HTTP/1.1 200 OK
note over n_orders, n_inv_a : A crashed instance stays on the list
n_orders ->x n_inv_a : [lost] GET /v1/stock/sku_19 HTTP/1.1
note over n_inv_c, n_registry : Heartbeats and health checks
n_inv_c -> n_registry : PUT /v1/agent/check/pass/service:inv-c HTTP/1.1
n_registry --> n_inv_c : HTTP/1.1 200 OK
note over n_orders, n_registry : DNS-based discovery
n_orders -> n_registry : dig @registry.internal -p 8600 inventory.service.internal A
n_registry --> n_orders : ;; ANSWER SECTION:
note over n_orders, n_lb : Server-side discovery: let a load balancer ask
n_orders -> n_lb : GET /v1/stock/sku_19 HTTP/1.1
n_lb --> n_orders : HTTP/1.1 200 OK
@endumlPlantUML: components
@startuml
left to right direction
component "Orders service" as n_orders
component "Inventory A" as n_inv_a #fde2e2;line:c62828;line.dashed
component "Inventory C" as n_inv_c
database "Service registry" as n_registry
component "Inventory D" as n_inv_d
node "Load balancer" as n_lb
n_inv_c ..> n_registry
n_inv_d ..> n_registry
n_orders --> n_lb
n_lb --> n_inv_c
n_lb --> n_inv_d
n_registry ..> n_lb
@endumlRecap
Never write instance addresses by hand.
1) Instances register their name and address in a registry on start. 2) Heartbeats with a TTL and active health checks remove dead instances automatically. 3) Client-side discovery: the caller asks the registry and balances itself, no extra hop. 4) Server-side discovery: a load balancer asks for you, simpler callers, one more hop. 5) DNS discovery works everywhere, but watch the TTL and caching.
Numbers: at the start and at the end
| Metric | Start | End |
|---|---|---|
| Failed stock calls | 0.1% | 0.3% |
| Time to stop calling a dead instance | 1 800 s | 15 s |
| Live inventory instances | 2 | 2 |
Steps
- Case Two IPs in a config file
The orders service checks stock before every order.
- Problem An instance moved; the config did not
The platform replaced instance B after a host failure.
- Fix Instances register themselves
A service registry is a small, highly available database of who is running where.
- Fix Client-side discovery
Orders asks the registry for healthy
inventoryinstances, caches the list for a few seconds and picks one per call itself, round-robin. - Problem A crashed instance stays on the list
Instance A's host loses power.
- Fix Heartbeats and health checks
Each instance now reports "I am alive" every 5 seconds, with a 15-second TTL.
- Case DNS-based discovery
The registry can also answer DNS.
- Fix Server-side discovery: let a load balancer ask
Now orders calls one stable name,
inventory.internal. - Recap Recap
Never write instance addresses by hand.
Terms
- Service discovery
- Instances write their name and address to a registry on start and are removed when their heartbeat stops. Callers get addresses from the registry, not from config: themselves (client-side), via a load balancer (server-side) or via DNS.