mufassall

Cheat sheet · Architecture

Service discovery

11 min · Intermediate · mufassall.dtwin.uz/en/l/service-discovery/

← Open the lesson

The problem

An orders service calls inventory instances that start, stop and move. How a registry with heartbeats replaces hard-coded IPs, and the three ways to ask it: from the client, through DNS, or through a load balancer.

What you learn

  • Why hard-coded addresses break as soon as instances move
  • How instances register and how heartbeats remove dead ones
  • Client-side discovery vs server-side discovery
  • DNS-based discovery and what its TTL costs
Final scene
30%30%40%40%

Diagram as code

The lesson's final design as text: paste it into Jira, GitLab, GitHub, Notion or your docs, where Mermaid renders by itself. There are D2 and PlantUML (Confluence, IntelliJ) versions too.

Mermaid: flowchart

flowchart LR
  n_orders["Orders service"]
  n_inv_a["Inventory A"]
  n_inv_c["Inventory C"]
  n_registry[("Service registry")]
  n_inv_d["Inventory D"]
  n_lb{"Load balancer"}
  n_inv_c -.-> n_registry
  n_inv_d -.-> n_registry
  n_orders --> n_lb
  n_lb --> n_inv_c
  n_lb --> n_inv_d
  n_registry -.-> n_lb
  classDef down fill:#fde2e2,stroke:#c62828,color:#4a0d0d,stroke-dasharray:4 3
  class n_inv_a down

Mermaid: request sequence

sequenceDiagram
  participant n_orders as Orders service
  participant n_inv_a as Inventory A
  participant n_inv_b as Inventory B
  participant n_inv_c as Inventory C
  participant n_registry as Service registry
  participant n_lb as Load balancer
  Note over n_orders,n_inv_a: Two IPs in a config file
  n_orders->>n_inv_a: GET /v1/stock/sku_19 HTTP/1.1
  n_inv_a-->>n_orders: HTTP/1.1 200 OK
  Note over n_orders,n_inv_b: An instance moved#59; the config did not
  n_orders-xn_inv_b: GET /v1/stock/sku_19 HTTP/1.1
  Note over n_inv_c,n_registry: Instances register themselves
  n_inv_c->>n_registry: PUT /v1/agent/service/register HTTP/1.1
  n_registry-->>n_inv_c: HTTP/1.1 200 OK
  Note over n_orders,n_registry: Client-side discovery
  n_orders->>n_registry: GET /v1/health/service/inventory?passing=true HTTP/1.1
  n_registry-->>n_orders: HTTP/1.1 200 OK
  Note over n_orders,n_inv_a: A crashed instance stays on the list
  n_orders-xn_inv_a: GET /v1/stock/sku_19 HTTP/1.1
  Note over n_inv_c,n_registry: Heartbeats and health checks
  n_inv_c->>n_registry: PUT /v1/agent/check/pass/service:inv-c HTTP/1.1
  n_registry-->>n_inv_c: HTTP/1.1 200 OK
  Note over n_orders,n_registry: DNS-based discovery
  n_orders->>n_registry: dig @registry.internal -p 8600 inventory.service.internal A
  n_registry-->>n_orders: #59;#59; ANSWER SECTION:
  Note over n_orders,n_lb: Server-side discovery: let a load balancer ask
  n_orders->>n_lb: GET /v1/stock/sku_19 HTTP/1.1
  n_lb-->>n_orders: HTTP/1.1 200 OK

D2

direction: right

n_orders: "Orders service" {
  shape: rectangle
}
n_inv_a: "Inventory A" {
  shape: rectangle
  style.fill: "#fde2e2"
  style.stroke: "#c62828"
  style.stroke-dash: 3
}
n_inv_c: "Inventory C" {
  shape: rectangle
}
n_registry: "Service registry" {
  shape: cylinder
}
n_inv_d: "Inventory D" {
  shape: rectangle
}
n_lb: "Load balancer" {
  shape: diamond
}

n_inv_c -> n_registry {
  style.stroke-dash: 3
}
n_inv_d -> n_registry {
  style.stroke-dash: 3
}
n_orders -> n_lb
n_lb -> n_inv_c
n_lb -> n_inv_d
n_registry -> n_lb {
  style.stroke-dash: 3
}

PlantUML: request sequence

@startuml
participant "Orders service" as n_orders
participant "Inventory A" as n_inv_a
participant "Inventory B" as n_inv_b
participant "Inventory C" as n_inv_c
database "Service registry" as n_registry
participant "Load balancer" as n_lb
note over n_orders, n_inv_a : Two IPs in a config file
n_orders -> n_inv_a : GET /v1/stock/sku_19 HTTP/1.1
n_inv_a --> n_orders : HTTP/1.1 200 OK
note over n_orders, n_inv_b : An instance moved; the config did not
n_orders ->x n_inv_b : [lost] GET /v1/stock/sku_19 HTTP/1.1
note over n_inv_c, n_registry : Instances register themselves
n_inv_c -> n_registry : PUT /v1/agent/service/register HTTP/1.1
n_registry --> n_inv_c : HTTP/1.1 200 OK
note over n_orders, n_registry : Client-side discovery
n_orders -> n_registry : GET /v1/health/service/inventory?passing=true HTTP/1.1
n_registry --> n_orders : HTTP/1.1 200 OK
note over n_orders, n_inv_a : A crashed instance stays on the list
n_orders ->x n_inv_a : [lost] GET /v1/stock/sku_19 HTTP/1.1
note over n_inv_c, n_registry : Heartbeats and health checks
n_inv_c -> n_registry : PUT /v1/agent/check/pass/service:inv-c HTTP/1.1
n_registry --> n_inv_c : HTTP/1.1 200 OK
note over n_orders, n_registry : DNS-based discovery
n_orders -> n_registry : dig @registry.internal -p 8600 inventory.service.internal A
n_registry --> n_orders : ;; ANSWER SECTION:
note over n_orders, n_lb : Server-side discovery: let a load balancer ask
n_orders -> n_lb : GET /v1/stock/sku_19 HTTP/1.1
n_lb --> n_orders : HTTP/1.1 200 OK
@enduml

PlantUML: components

@startuml
left to right direction
component "Orders service" as n_orders
component "Inventory A" as n_inv_a #fde2e2;line:c62828;line.dashed
component "Inventory C" as n_inv_c
database "Service registry" as n_registry
component "Inventory D" as n_inv_d
node "Load balancer" as n_lb
n_inv_c ..> n_registry
n_inv_d ..> n_registry
n_orders --> n_lb
n_lb --> n_inv_c
n_lb --> n_inv_d
n_registry ..> n_lb
@enduml

Recap

Never write instance addresses by hand.

1) Instances register their name and address in a registry on start. 2) Heartbeats with a TTL and active health checks remove dead instances automatically. 3) Client-side discovery: the caller asks the registry and balances itself, no extra hop. 4) Server-side discovery: a load balancer asks for you, simpler callers, one more hop. 5) DNS discovery works everywhere, but watch the TTL and caching.

Numbers: at the start and at the end

MetricStartEnd
Failed stock calls0.1%0.3%
Time to stop calling a dead instance1 800 s15 s
Live inventory instances22

Steps

  1. Case Two IPs in a config file

    The orders service checks stock before every order.

  2. Problem An instance moved; the config did not

    The platform replaced instance B after a host failure.

  3. Fix Instances register themselves

    A service registry is a small, highly available database of who is running where.

  4. Fix Client-side discovery

    Orders asks the registry for healthy inventory instances, caches the list for a few seconds and picks one per call itself, round-robin.

  5. Problem A crashed instance stays on the list

    Instance A's host loses power.

  6. Fix Heartbeats and health checks

    Each instance now reports "I am alive" every 5 seconds, with a 15-second TTL.

  7. Case DNS-based discovery

    The registry can also answer DNS.

  8. Fix Server-side discovery: let a load balancer ask

    Now orders calls one stable name, inventory.internal.

  9. Recap Recap

    Never write instance addresses by hand.

Terms

Service discovery
Instances write their name and address to a registry on start and are removed when their heartbeat stops. Callers get addresses from the registry, not from config: themselves (client-side), via a load balancer (server-side) or via DNS.