DFdom@fradley
← ~/writing
$ cat half-built-landing-zone.md

What to finish first in a half-built landing zone

The estate is neither greenfield nor a mess. Somebody did the hard early thinking, and it shows: the network topology holds up and the subscription boundaries mean something. The platform deploys from code through a pipeline rather than from somebody's memory, and real services are running on it today.

Four things are half done. The identity model got as far as a diagram, and underneath it sits a pile of role assignments granted directly to whoever needed access that week. Nobody can say what a given product costs to run. Policy is assigned but enforces almost nothing. The code that builds the platform stops at the workload boundary, so everything people ship still lands by hand.

The platform team is small and the demand on it is not. Nobody is pausing delivery while you tidy up, and asking for that pause is the request that gets a platform team quietly defunded. So the work is a matter of order, and the ordering rule matters more than the list.

The four gaps do not get more expensive at the same rate

The usual move is to rank the gaps by risk and work down. Risk tells you which one would hurt most if it were exploited today. It says nothing about timing, and timing is what makes this estate different from a greenfield one, because workloads keep landing on the unfinished foundation while you work on it.

The more useful ranking is by how much each gap costs to close after another quarter of delivery.

Identity compounds fastest. Every new subscription, workload and support ticket adds role assignments, and each one granted directly to a person is something you will later have to find, understand, and then take away from somebody who is using it.

Governance compounds more slowly. Every resource created outside a policy becomes an exception to the policy you write later, so the distance between an audit-mode assignment and a deny-mode one widens as the estate grows.

Cost visibility is cheap to fix at any point, but its value arrives late. Cost Management attributes spend using the tags the resource carried when the usage was recorded, and tags are not applied retroactively for cost rollups, so a month of waiting is a month of history you cannot get back.

Workload IaC barely compounds at all. Importing a workload into Terraform costs about the same whether you do it this quarter or next year, because the cost is per workload rather than per week of delay.

gap cost of waiting another quarter order identity every week adds assignments to unpick 1st cost visibility cheap to fix; history never backfills 2nd governance untouched resources become exceptions 3rd workload IaC per workload, not per week of delay leave open on purpose
Solid lines are what each gap costs to close after another quarter of delivery. Identity is the only one that climbs steeply, which is what puts it first. Cost visibility is cheap to close at any point; the dashed line is the attribution history you can still recover, and it falls away whether you work on it or not. Workload IaC stays flat, so paying for it early buys nothing.

That is not the risk ordering, and it is the one you can defend when a delivery manager comes asking for two of your engineers.

Identity first, because every week adds assignments to unpick

Measure it before you design anything:

az role assignment list --all --include-inherited \
  --query "[?principalType=='User'].{user:principalName, role:roleDefinitionName, scope:scope}" \
  -o table

Every row is access that no code knows about. The row count is the size of the retrofit, and it grows every week you spend somewhere else.

Finished, for this gap, means three properties hold. Role assignments live in Terraform and target groups rather than people. They sit at management group or subscription scope rather than on individual resource groups, so a new spoke inherits its access model instead of collecting one.

Standing owner and contributor rights for humans are worth attacking early, because they become the hardest thing to remove once a support process depends on them. Make them eligible rather than active:

resource "azurerm_pim_eligible_role_assignment" "platform_break_glass" {
  scope              = data.azurerm_subscription.workload.id
  role_definition_id = "${data.azurerm_subscription.workload.id}${data.azurerm_role_definition.owner.id}"
  principal_id       = azuread_group.platform_engineers.object_id

  schedule {
    expiration {
      duration_days = 365
    }
  }

  justification = "Platform team break-glass, activated per incident"
}

The access still exists, and somebody has to ask for it and say why. Privileged Identity Management is a licensed feature, so check what the tenant already holds (licensing fundamentals) before designing around it.

Then set the forward rule: a new subscription arrives with its role assignments in the same code that created the subscription, which is one of the jobs landing zones as code exists to do. Every subscription created after that rule lands is one you never have to go back and audit.

Remove the existing direct assignments last, with the list from that query in front of you. It is the only step in the sequence that can break somebody's Tuesday.

Cost visibility second, because the history starts when you start

Poor cost visibility reads as a finance problem. At this stage it is a prioritisation problem, because every argument you are about to have about where the team's next quarter goes gets settled on opinion while nobody can put a number against a product.

The fix is mechanical and lands in about a week. A required tag set covering owner, cost centre and environment, assigned in audit mode first, then a Modify effect with a remediation task where the value can be inferred from the subscription or the resource group, then deny for new resources once the estate is clean. Untagged Azure resources has the policy. A budget per subscription with an alert at 80% is a few lines of Terraform, covered in budget alerts as code.

Size it the same way you sized identity:

resources
| where isnull(tags['cost_centre']) or tags['cost_centre'] == ''
| summarize untagged = count() by subscriptionId, type
| order by untagged desc

This goes second on the timing argument rather than the urgency one. The reporting you want to show somebody in January has to have been tagged in September, because the tag is stamped onto the usage record as it is written. Tag inheritance recovers part of it, pushing subscription and resource-group tags down onto child resource usage, but only back to the start of the current billing month. Everything before that stays unattributed.

What it buys is the ability to make the rest of the sequence a proportional argument. A product with a visible monthly run rate turns "the platform team needs two more engineers" into a comparison rather than a request.

Governance stays light, with a wide audit set and a small deny set

The failure mode here is a governance programme: an initiative per compliance framework and a spreadsheet tracking the exemptions. A small team cannot run that next to a live roadmap, and the attempt is how a platform team acquires a reputation for slowing delivery down.

Assign the broad built-in initiatives in audit mode and you have a compliance picture inside a day without blocking anybody. Azure ships thousands of definitions you already own, which Azure Policy out of the box covers in more detail.

Then a deny set small enough to hold in your head. The test for whether a rule belongs in it is that compliance is cheap and reversal is expensive:

  • Regions outside the approved list, because a resource in the wrong region is a migration rather than an edit.
  • Public network access on newly created data services.
  • Public IP addresses on virtual machines in spoke subscriptions.
  • Missing required tags, switched on only after the tagging remediation has run.

That last one is the dependency that puts governance third rather than first. The deny set you can afford is a function of how clean the estate already is. Tighten policy before tagging and identity are settled and you generate exemptions instead of compliance, and an exemption list is a governance model that has already lost.

Everything else stays on audit. Non-compliance in audit mode is a work queue you can see, which is worth more than a deny that gets an exemption on the day it blocks a release.

Workload IaC is the gap to leave open on purpose

Platform in Terraform with workloads by hand is the common shape at this stage, and the instinct is to schedule an import programme. The cost curve argues against it. Import cost is per workload and roughly constant, so paying it now saves nothing, and the workload teams get nothing on the day it lands.

What does compound is the rate at which new hand-built workloads arrive. Put the effort there.

Build the paved path first: a module for a standard spoke plus the pipeline template that deploys it, so the fastest route to a working environment is the code route. If the portal is quicker, the standard on the wiki page will not save you.

Then import on touch. When a workload is being changed anyway, it comes under Terraform as the first step of that change, using the zero-diff adoption described in ClickOps to code. The testing was already funded, so the import rides on work that was happening regardless.

Import cold only where a workload sits inside the blast radius of platform work you are about to do. A spoke whose route tables you are about to rewrite is worth having in state before you rewrite them.

The proportion of the estate under code then climbs on its own, and no line item ever appears on a roadmap asking product for six months to rebuild what already runs.

How to split a small team across that order

A team spread across all four gaps at once moves everything at a quarter speed and finishes nothing, while the estate keeps growing under every one of them.

On a team of five I would run two workstreams. Two engineers on the platform sequence in the order above: identity, then tagging and budgets, then policy. Two embedded with the delivery teams building the paved path. One on the pipeline and the operational load, which is the work that otherwise interrupts everyone else at random.

The embedded pair is the part that gets cut first when the roadmap squeezes, and it should be the last. The platform team's product is the paved path, and the delivery teams are its customers. A path designed without them gets routed around, politely, by people with a release date.

Replace the status update with a definition of done

"Making good progress" is a status with no end. It is why leadership keeps asking when the platform work finishes and keeps getting an answer about percentages.

Give it a definition of done made of properties of the estate, each one a query somebody can run:

  • No role assignments held directly by users, at any scope.
  • Every subscription carries an owner and a cost centre tag, and sits inside a budget.
  • Every spoke created since the module shipped came from the module.
  • The deny set is assigned at the top management group, with no exemption older than 30 days.

Progress reporting then comes out of the estate rather than off a slide, and the number moves because work happened. It also puts the trade in a form a delivery manager can see: here is what is left, here is what it costs, here is what slips if the engineers move.

The first month

Run three counts before designing anything, because they turn four vague gaps into numbers you can sequence:

  1. Role assignments held directly by users, at every scope.
  2. Resources with no cost centre tag, grouped by subscription.
  3. Resource groups holding resources that no Terraform state file knows about.

Then work the order: identity, tagging and budgets, audit-mode policy with a small deny set, paved path. Workload backfill happens on touch and never becomes a programme.

week 0 three counts, before you design anything direct user RBAC az role assignment list untagged resources resource graph, by sub RGs outside state terraform state list then, in order month 1 month 2 month 3 identity compounds fastest groups + scope in code PIM eligible, then vending carries RBAC remove direct last cost visibility history never backfills required tags: audit, then modify remediation task, budgets at 80% deny on tags waits here governance waits on clean tags broad initiatives on audit small deny set, tags last workload IaC does not compound paved path module + pipeline, then import on touch never a programme done = four properties of the estate, each one a query > no role assignments held by users > every spoke came from the module > every sub tagged and inside a budget > deny set at top MG, no stale exemptions
The same order, on a calendar. Identity starts in week one because it is the only gap that gets more expensive while you look at it, and the single step that can break somebody's Tuesday sits at the end of that lane rather than the start. Governance cannot tighten until the tagging remediation has run, which is what the arrow marks. Workload IaC has no start and no finish: a paved path with import on touch, running underneath everything else.

One thing to check in your own estate before trusting any of this. Confirm the platform Terraform state is authoritative, because if people have been fixing things in the portal on top of the platform code, the foundation is less finished than it looks. A plan against the platform repo will tell you inside ten minutes, and it is the cheapest thing on this list to run first.

The queries and the eligible-assignment example are in the blog-examples repo.

More useful tidbits coming — one a week.

discuss on linkedin → more writing