How we work

Five stages, in order, from traffic map to handover — with the measurement baseline captured before anything changes.

The shape of an engagement

Routing changes are unusual: the blast radius is global, the feedback loop is delayed by DNS caching, and the failure mode is often "some users, somewhere, sometimes".

The sequence below runs on most engagements; durations are typical rather than contractual. How the stages map to a scope of work is on the engagements page, and what we report at the end is under results.

One rule sits above the rest: no routing change ships before a baseline exists to compare it against. If measurement cannot be completed, we fix that first.

1. Discovery

Where requests actually enter and leave your estate — often not what the architecture diagram says.

What happens

  • A traffic map: every public hostname, which resolver answers for it, and what the answer is by geography.
  • Current topology — VPCs and VNets, peering and transit gateways, egress paths.
  • Which provider primitives are already in play: Route 53 routing policies, Cloud DNS, Azure Traffic Manager, Cloudflare load balancing, GCLB, ALB/NLB.
  • Existing health checks — endpoints, intervals, thresholds, where they are evaluated from — and record TTLs as they stand, including any dropped for an incident and never restored.

What we produce

A written topology and traffic map, and a list of what is surprising: orphaned records, checks pointing at endpoints that always return 200, a day-long TTL in front of a failover record.

Roughly how long

3–5 working days

What can go wrong here

Read access arrives late, or for one account out of twelve — and a partial map is how you design failover for traffic that never touched that path.

2. Measurement baseline

You cannot improve what you have not measured. The baseline is captured before any change is made.

What happens

  • Real user monitoring segmented by region, network and device class — not one global average that hides a bad continent.
  • Synthetic probes from the regions your users are in, at a fixed interval, so the series stays comparable.
  • p50, p95 and p99 per segment — routing problems live in the p99 — decomposed into DNS resolution, TCP connect, TLS handshake cost, time to first byte, and the edge versus origin split.
  • Failure-side data — error rates, retries, and how long resolvers really hold your records.

What we produce

A baseline document with the numbers, method and date range, reproducible by your team rather than taken on trust. Where a service level objective is in scope the target is written here — a latency or availability SLO the design aims at, never a result already achieved. See our high latency notes.

Roughly how long

1–2 weeks, including at least one full traffic week

What can go wrong here

An unrepresentative window — a release week is not a quiet week — or instrumenting only the happy path, leaving the baseline silent about the conditions the design must survive.

3. Design

A written document, reviewed with your engineers before a single record changes. If your team cannot explain it back to us, it is not finished.

What happens

  • Routing policy — latency-based, geolocation, weighted or layered, with the reason for each. Trade-offs in global load balancing.
  • Failover topology: active-active versus active-passive, chosen against your RTO and RPO — active-passive is cheaper and has a cold path you must exercise.
  • Health-check design — what the check asserts (a dependency-aware endpoint, not a static 200), interval, thresholds, and enough evaluation points that one region cannot trigger a global flip.
  • TTL strategy per record class: long for stable apex records, short for anything in a failover path, plus a pre-cutover reduction window.
  • Blast radius and rollback: what each step can affect, the revert action, and how long the revert takes to propagate.

What we produce

A design document, a change plan ordered by risk, and the rollback criteria — cross-referenced to our engineering guidelines and the disaster recovery checklist.

Roughly how long

1–2 weeks, including review cycles

What can go wrong here

Designing for a steady state nobody operates in: a topology needing four teams to coordinate during an incident will not survive its first real failure.

4. Implementation

Changes go out staged behind low TTLs, one region or one policy at a time, each with its own rollback.

What happens

  • TTLs on affected records are reduced first and allowed to age out fully, so the old value is out of resolver caches before cutover.
  • A canary region or small traffic weight goes first, held long enough to see a real traffic cycle, not ten minutes of graphs.
  • Health checks and monitors are observed before traffic moves onto the path they protect.
  • Infrastructure as code where you already use it — Terraform, Pulumi or CloudFormation, in your repositories, through your review process — with each step announced, observed and recorded, rollback command written out first.

What we produce

Merged configuration in your own repositories, a timestamped change log, and DNS and health-check state matching the design. Work spans whichever of routing consulting, traffic optimization, multi-cloud architecture or failover the design calls for.

Roughly how long

2–6 weeks, depending on region and provider count

What can go wrong here

Resolvers that ignore your TTL. Some do, and a cutover assuming everyone honours 60 seconds leaves a tail on the old path for hours; keep it serving until that drains. The other classic is a provider-side rate limit on record updates, found mid-cutover.

5. Validation and handover

The engagement ends when your team can operate the design without us, not when the last record changes.

What happens

  • Re-measurement against the stage 2 baseline with the same probes, regions and percentiles — same method, or the comparison means nothing.
  • A deliberate failover drill: an endpoint is failed, the check trips on its real interval, and time to full shift is measured against the targeted RTO.
  • A runbook — what a flapping check looks like, how to force a failover and fail back, what to check when one region alone gets slow.
  • Documentation of final topology, policy set, TTLs and check configuration, held in your systems, and a handover session with the engineers who will be on call.

What we produce

A before-and-after comparison against the baseline, drill results, the runbook and the architecture documentation — including, where an SLO was set, whether the design meets it under drill conditions.

Roughly how long

1–2 weeks

What can go wrong here

A drill on a quiet Friday afternoon proves only that failover works when nothing else is happening. And a runbook nobody has walked through is one nobody follows at 03:00.

Change safety

Every stage inherits the same safety properties, and they are not negotiated away to hit a date. A fourth sits alongside them: never two routing changes at once, or you cannot attribute a p99 that moves.

Low TTLs before a cutover

TTLs are reduced and allowed to fully expire ahead of the change, so a rollback propagates in seconds rather than in whatever the old value was. Raising them again is a tracked task.

Canary regions and weights

New policy reaches a bounded slice first — one region, or a small weight on a weighted record — chosen so total failure of that slice is a known, acceptable fraction of requests.

Rollback criteria agreed in advance

Before a step runs we write down the signal that ends it: error rate above a threshold, p99 beyond a bound, a check flapping more than n times in a window — so rolling back is a reading, not a debate.

Who does the work

The engineers who write the design make the changes and sit on the handover call. Nothing is passed to an unnamed pool once it is signed off.

Backgrounds are on the team page, the company story on the about page, questions about access and scope in the FAQ. Our reasoning is published openly in insights.

CloudRoute Inc. has operated since 2010 from San Francisco, CA, with clients worldwide, delivered remotely. Engagements are scoped over email — [email protected], replies within one business day, Mon–Fri 9:00–18:00 Pacific. No phone line, no contact form.

Start at stage one

Send us the symptom — a slow region, a failover never drilled, a second cloud you are about to commit to — and we will tell you whether stage one is worth running.

Start a conversation Read the technical insights