DFdom@fradley
← ~/writing
$ cat half-built-landing-zone.md

What to finish first in a half-built landing zone

The network topology holds up, the subscription boundaries mean something, the platform deploys from code through a pipeline, and real services run on it today.

Four things are half done. The identity model got as far as a diagram, with a pile of role assignments granted directly to whoever needed access that week underneath it. Nobody can say what a given product costs to run. Policy is assigned but enforces almost nothing. The code that builds the platform stops at the workload boundary, so everything people ship still lands by hand.

The platform team is small and nobody is pausing delivery while you tidy up, so the work is a matter of order.

The four gaps don't get more expensive at the same rate

Ranking by risk tells you what would hurt most if exploited today, and says nothing about timing. Workloads keep landing on the unfinished foundation while you work, so rank by what each gap costs to close after another quarter.

  • Identity compounds fastest. Every new subscription, workload and support ticket adds role assignments, and each one granted directly to a person is something you later have to find, understand and take away from somebody who is using it.
  • Governance compounds more slowly. Every resource created outside a policy becomes an exception to the policy you write later.
  • Cost visibility is cheap to fix at any point, but Cost Management attributes spend using the tags the resource carried when the usage was recorded, and tags are not applied retroactively for cost rollups. A month of waiting is a month of history you can't get back.
  • Workload IaC barely compounds. Importing a workload into Terraform costs about the same this quarter or next year, because the cost is per workload rather than per week of delay.
gap cost of waiting another quarter order identity every week adds assignments to unpick 1st cost visibility cheap to fix; history never backfills 2nd governance untouched resources become exceptions 3rd workload IaC per workload, not per week of delay leave open on purpose
Solid lines are what each gap costs to close after another quarter of delivery. The dashed line is the cost attribution history you can still recover, which falls away whether you work on it or not. Paying for workload IaC early buys nothing.

Identity first

Measure it before you design anything:

az role assignment list --all --include-inherited \
  --query "[?principalType=='User'].{user:principalName, role:roleDefinitionName, scope:scope}" \
  -o table

Every row is access that no code knows about, and the row count is the size of the retrofit. Done means role assignments live in Terraform, target groups rather than people, and sit at management group or subscription scope so a new spoke inherits its access model instead of collecting one.

Standing owner and contributor rights for humans are worth attacking early, because they become the hardest thing to remove once a support process depends on them. Make them eligible rather than active:

resource "azurerm_pim_eligible_role_assignment" "platform_break_glass" {
  scope              = data.azurerm_subscription.workload.id
  role_definition_id = "${data.azurerm_subscription.workload.id}${data.azurerm_role_definition.owner.id}"
  principal_id       = azuread_group.platform_engineers.object_id

  schedule {
    expiration {
      duration_days = 365
    }
  }

  justification = "Platform team break-glass, activated per incident"
}

Somebody has to ask for the access and say why. Privileged Identity Management is a licensed feature, so check what the tenant already holds (licensing fundamentals) before designing around it.

Then set the forward rule: a new subscription arrives with its role assignments in the same code that created it (landing zones as code). Remove the existing direct assignments last, with the list from that query in front of you. It's the only step in the sequence that can break somebody's Tuesday.

Cost visibility second

The fix lands in about a week. Assign a required tag set (owner, cost centre, environment) in audit mode first. Then use a Modify effect with a remediation task where the value can be inferred from the subscription or resource group, and deny for new resources once the estate is clean. Untagged Azure resources has the policy, and budget alerts as code covers a budget per subscription with an alert at 80%. Size it the same way as identity:

resources
| where isnull(tags['cost_centre']) or tags['cost_centre'] == ''
| summarize untagged = count() by subscriptionId, type
| order by untagged desc

The reporting you want to show somebody in January has to be tagged in September, because the tag is stamped onto the usage record as it's written. Tag inheritance recovers part of it by pushing subscription and resource-group tags down onto child resource usage, but only back to the start of the current billing month.

Governance stays light

A small team can't run an initiative per compliance framework and a spreadsheet of exemptions next to a live roadmap. Assign the broad built-in initiatives in audit mode for a compliance picture inside a day (Azure Policy out of the box covers what ships). Then keep a deny set small enough to hold in your head, limited to rules where compliance is cheap and reversal is expensive:

  • Regions outside the approved list, because a resource in the wrong region is a migration rather than an edit.
  • Public network access on newly created data services.
  • Public IP addresses on virtual machines in spoke subscriptions.
  • Missing required tags, switched on only after the tagging remediation has run.

That last one puts governance third: tighten policy before tagging and identity are settled and you generate exemptions. Everything else stays on audit, where non-compliance is a work queue you can see.

Workload IaC: leave it open on purpose

The instinct is to schedule an import programme. Import cost is per workload and roughly constant, so paying it now saves nothing. What does compound is the rate at which new hand-built workloads arrive.

Build the paved path first: a module for a standard spoke plus the pipeline template that deploys it, so the fastest route to a working environment is the code route. If the portal is quicker, the wiki standard won't save you.

Then import on touch. When a workload is being changed anyway, it comes under Terraform as the first step of that change, using the zero-diff adoption in ClickOps to code. Import cold only a workload inside the blast radius of platform work you're about to do, such as a spoke whose route tables you're about to rewrite.

Splitting a small team

On a team of five I'd run two engineers on the platform sequence above, two embedded with the delivery teams building the paved path, and one on the pipeline and operational load, which otherwise interrupts everyone at random.

The embedded pair gets cut first when the roadmap squeezes, and should go last, because a path designed without the delivery teams gets routed around.

Replace the status update with a definition of done

Replace "making good progress" with a definition of done made of properties of the estate, each one a query somebody can run:

  • No role assignments held directly by users, at any scope.
  • Every subscription carries an owner and a cost centre tag, and sits inside a budget.
  • Every spoke created since the module shipped came from the module.
  • The deny set is assigned at the top management group, with no exemption older than 30 days.

Progress then comes out of the estate rather than off a slide, and a delivery manager can see what is left and what slips if the engineers move.

The first month

Before designing anything, count the role assignments held directly by users, the resources with no cost centre tag by subscription, and the resource groups no Terraform state file knows about. Then work the order.

week 0 three counts, before you design anything direct user RBAC az role assignment list untagged resources resource graph, by sub RGs outside state terraform state list then, in order month 1 month 2 month 3 identity compounds fastest groups + scope in code PIM eligible, then vending carries RBAC remove direct last cost visibility history never backfills required tags: audit, then modify remediation task, budgets at 80% deny on tags waits here governance waits on clean tags broad initiatives on audit small deny set, tags last workload IaC does not compound paved path module + pipeline, then import on touch never a programme done = four properties of the estate, each one a query > no role assignments held by users > every spoke came from the module > every sub tagged and inside a budget > deny set at top MG, no stale exemptions
The same order, on a calendar. Governance cannot tighten until the tagging remediation has run, which is what the arrow marks. Workload IaC has no start and no finish: a paved path with import on touch, running underneath everything else.

Before trusting any of this, confirm the platform Terraform state is authoritative. If people have been fixing things in the portal on top of the platform code, the foundation is less finished than it looks. A plan against the platform repo tells you inside ten minutes.

The queries and the eligible-assignment example are in the blog-examples repo.

discuss on linkedin → more writing