Verification trail
#2
✅ Verified
content/blog/what-a-good-infrastructure-pull-request-looks-like-now/index.md:22
A 2025 study of agent-authored pull requests by Li, Zhang, and Hassan (arXiv 2507.15003) collected over 456,000 pull requests from five coding agents across…
framing: The paper introduces a dataset (AIDev). Calling it a "study" that "collected" the pull requests describes the same thing fairly.; evidence: The arXiv abstract (submitted 20 Jul 2025) lists the authors as Hao Li, Haoxiang Zhang and Ahmed E. Hassan. It reads: "Spanning over 456,000 pull requests by five leading agents--OpenAI Codex, Devin, GitHub Copilot, Cursor, and Claude…; source: https://arxiv.org/abs/2507.15003
source: https://arxiv.org/abs/2507.15003
framing: exact-match — The paper introduces a dataset (AIDev). Calling it a "study" that "collected" the pull requests describes the same thing fairly.
pass2 · attribution · high confidence
#3
✅ Verified
content/blog/what-a-good-infrastructure-pull-request-looks-like-now/index.md:30
A one-line change to a resource property can force a stateful resource to be replaced in infrastructure-as-code.
framing: The claim is general IaC behavior with no cited source. The docs page backs it but isn't product source; a gh code search to check product source was…; evidence: Pulumi's replaceOnChanges docs say providers decide "whether the input property is one that the provider knows how to update in place, or if not, requires a replacement to modify." So changing a single input property can force a resource…; source: repo:content/docs/iac/concepts/resources/options/replaceonchanges.md
source: repo:content/docs/iac/concepts/resources/options/replaceonchanges.md
framing: The claim is general IaC behavior with no cited source. The docs page backs it but isn't product source; a gh code search to check product source was rate-limited.
pass1 · behavior · medium confidence
#4
✅ Verified
content/blog/what-a-good-infrastructure-pull-request-looks-like-now/index.md:30
A renamed resource in Pulumi code can appear as a delete plus a create in the cloud.
evidence: The Pulumi aliases docs say that if you rename a resource without an alias, `pulumi up` deletes the old resource and creates a new one: "By default, when we run `pulumi up`, we see that the old resource is deleted and the new one…; source: repo:content/docs/iac/concepts/resources/options/aliases.md#L18-L24
source: repo:content/docs/iac/concepts/resources/options/aliases.md#L18-L24
pass1 · behavior · medium confidence
#6
✅ Verified
content/blog/what-a-good-infrastructure-pull-request-looks-like-now/index.md:38
Pulumi's continuous delivery guidance recommends the pipeline run `pulumi preview` when the pull request opens, post the summary of proposed changes for…
framing: The claim pulls together the guide's L41 (run preview when the PR opens and post the summary) and L47 (re-run preview after each branch update) without…; evidence: The CD guide says: "Open a pull request. The pipeline runs `pulumi preview` and posts the summary of proposed changes for review." It also says to "re-run `pulumi preview` after each update to the branch."; source: repo:content/docs/iac/operations/continuous-delivery/_index.md#L41-L47
source: repo:content/docs/iac/operations/continuous-delivery/_index.md#L41-L47
framing: exact-match — The claim pulls together the guide's L41 (run preview when the PR opens and post the summary) and L47 (re-run preview after each branch update) without changing what either says.
pass1 · attribution · high confidence
#7
✅ Verified
content/blog/what-a-good-infrastructure-pull-request-looks-like-now/index.md:42
`pulumi preview --diff` displays a rich diff of the overall change.
evidence: The auto-generated CLI reference for `pulumi preview` lists the flag as: "--diff Display operation as a rich diff showing the overall change". That is the same thing the claim says.; source: repo:content/docs/iac/cli/commands/pulumi_preview.md L41
source: repo:content/docs/iac/cli/commands/pulumi_preview.md L41
pass1 · api-surface · high confidence
#8
✅ Verified
content/blog/what-a-good-infrastructure-pull-request-looks-like-now/index.md:43
The Pulumi GitHub App displays the results of stack update previews in pull requests.
evidence: The linked docs page says almost word for word what the claim says: "Pulumi's GitHub app displays the results of Pulumi stack update previews in pull requests and enables automatic stack deployments via Pulumi Deployments." Its…; source: repo:content/docs/integrations/version-control/github-app.md#L25
source: repo:content/docs/integrations/version-control/github-app.md#L25
framing: exact-match
pass1 · url · high confidence
#9
✅ Verified
content/blog/what-a-good-infrastructure-pull-request-looks-like-now/index.md:44
`pulumi preview --save-plan` saves the operations a preview proposed to a plan file, and `pulumi up --plan=plan.json` constrains the update to the operations…
framing: The claim's wording "constrains the update to the operations in the plan" matches the CLI help: "will not perform operations that exceed its plan". The claim…; evidence: The generated CLI reference for `pulumi preview` says: "--save-plan string [PREVIEW] Save the operations proposed by the preview to a plan file at the given path". For `pulumi up` it says: "--plan string [EXPERIMENTAL] Path to a plan…; source: repo:content/docs/iac/cli/commands/pulumi_preview.md L77; repo:content/docs/iac/cli/commands/pulumi_up.md L57
source: repo:content/docs/iac/cli/commands/pulumi_preview.md L77; repo:content/docs/iac/cli/commands/pulumi_up.md L57
framing: exact-match — The claim's wording "constrains the update to the operations in the plan" matches the CLI help: "will not perform operations that exceed its plan". The claim omits the [PREVIEW]/[EXPERIMENTAL] labels.
pass1 · api-surface · high confidence
#10
✅ Verified
content/blog/what-a-good-infrastructure-pull-request-looks-like-now/index.md:53
The Pulumi resource option `deleteBeforeReplace` deletes the existing resource before creating its replacement.
framing: The docs make the behavior conditional on setting the option to true. The claim leaves that out, which is fine because it describes what the option does when…; evidence: The resource options docs say: "Setting the `deleteBeforeReplace` option to true means that Pulumi will delete the existing resource before creating its replacement." By default, Pulumi creates the new resource before deleting the old one.; source: repo:content/docs/iac/concepts/resources/options/deletebeforereplace.md L21
source: repo:content/docs/iac/concepts/resources/options/deletebeforereplace.md L21
framing: entailed-narrower — The docs make the behavior conditional on setting the option to true. The claim leaves that out, which is fine because it describes what the option does when it is enabled.
pass1 · api-surface · high confidence
#11
✅ Verified
content/blog/what-a-good-infrastructure-pull-request-looks-like-now/index.md:57
Google's engineering practices say the right size for a change is one self-contained change, that reviewers may reject a change outright for being too large…
framing: Matches the source closely. The source says reviewers "have discretion to" reject large changes, and the claim's "may reject" keeps that meaning.; evidence: The page says: "In general, the right size for a CL is one self-contained change." It also says "reviewers have discretion to reject your change outright for the sole reason of it being too large" and lists "Simpler to roll back" as a…; source: https://google.github.io/eng-practices/review/developer/small-cls.html
source: https://google.github.io/eng-practices/review/developer/small-cls.html
framing: exact-match — Matches the source closely. The source says reviewers "have discretion to" reject large changes, and the claim's "may reject" keeps that meaning.
pass2 · attribution · high confidence
#12
✅ Verified
content/blog/what-a-good-infrastructure-pull-request-looks-like-now/index.md:63
Pulumi stack configuration lives in files named `Pulumi.<stack-name>.yaml`, and the Pulumi configuration docs say these files should be committed to version…
framing: The claim describes what the docs say, and the cited page says it at L23. Both points, the file name and committing to version control, match.; evidence: The config docs say: "The key-value pairs for any given stack are stored in your project's stack settings file, which is automatically named `Pulumi.<stack-name>.yaml`. Stack configuration files should be committed to version control…; source: repo:content/docs/iac/concepts/config.md#L23
source: repo:content/docs/iac/concepts/config.md#L23
framing: exact-match — The claim describes what the docs say, and the cited page says it at L23. Both points, the file name and committing to version control, match.
pass1 · attribution · high confidence
#13
✅ Verified
content/blog/what-a-good-infrastructure-pull-request-looks-like-now/index.md:65
The Pulumi resource option `ignoreChanges` ignores changes to specified properties during a diff.
framing: Claim restates the doc's meta_desc almost word for word; evidence: The ignoreChanges resource-option docs say: "The ignoreChanges resource option declares that changes to certain properties should be ignored during a diff." The page also lists ignoreChanges among the resource options in every SDK (TS…; source: repo:content/docs/iac/concepts/resources/options/ignorechanges.md
source: repo:content/docs/iac/concepts/resources/options/ignorechanges.md
framing: exact-match — Claim restates the doc's meta_desc almost word for word
pass1 · api-surface · high confidence
#15
✅ Verified
content/blog/what-a-good-infrastructure-pull-request-looks-like-now/index.md:69
The `--expect-no-changes` flag on `pulumi preview` returns an error if any changes are proposed by the preview.
framing: The claim says the same thing as the CLI help text. This page is generated from the CLI source.; evidence: The auto-generated CLI reference for `pulumi preview` lists: "--expect-no-changes Return an error if any changes are proposed by this preview".; source: repo:content/docs/iac/cli/commands/pulumi_preview.md#L44
source: repo:content/docs/iac/cli/commands/pulumi_preview.md#L44
framing: exact-match — The claim says the same thing as the CLI help text. This page is generated from the CLI source.
pass1 · api-surface · high confidence
#16
✅ Verified
content/blog/what-a-good-infrastructure-pull-request-looks-like-now/index.md:73
The Pulumi testing guide separates unit tests, which are fast in-memory tests that mock all external calls, from integration tests, which deploy ephemeral…
framing: The source lists three styles (unit, property, integration). The claim names only two, and describes both in the guide's own words.; evidence: The testing guide says: "**Unit tests** are fast in-memory tests that mock all external calls." and "**Integration tests** deploy ephemeral infrastructure and run external tests against it." The guide also lists a third style, property…; source: repo:content/docs/iac/guides/testing/_index.md
source: repo:content/docs/iac/guides/testing/_index.md
framing: entailed-narrower — The source lists three styles (unit, property, integration). The claim names only two, and describes both in the guide's own words.
pass1 · attribution · high confidence
#17
✅ Verified
content/blog/what-a-good-infrastructure-pull-request-looks-like-now/index.md:75
A Pulumi Deployments review stack is created automatically when a pull request is opened, updated on each new commit, and destroyed when the pull request is…
framing: The claim repeats the cited page's lifecycle sentence almost word for word. That page says review stacks only work once Deployment Settings and a review…; evidence: The cited docs page (L18) says almost the same thing: "Review stacks are ephemeral cloud environments powered by Pulumi Deployments. They are created automatically when a pull request is opened, updated on each new commit, and destroyed…; source: repo:content/docs/deployments/concepts/review-stacks.md#L18
source: repo:content/docs/deployments/concepts/review-stacks.md#L18
framing: exact-match — The claim repeats the cited page's lifecycle sentence almost word for word. That page says review stacks only work once Deployment Settings and a review stack template are set up, and the claim leaves that out.
pass1 · url · high confidence
#18
✅ Verified
content/blog/what-a-good-infrastructure-pull-request-looks-like-now/index.md:81
Google's guidance on change descriptions says "Fix bug" is an inadequate description because it leaves out which bug and what you did to fix it, that reading…
framing: The claim paraphrases three separate statements on the source page, and each one matches what the page says.; evidence: All three points in the claim appear on Google's "Writing good CL descriptions" page. It says "'Fix bug' is an inadequate CL description. What bug? What did you do to fix it?" It also says "Reading source code may reveal what the…; source: https://google.github.io/eng-practices/review/developer/cl-descriptions.html
source: https://google.github.io/eng-practices/review/developer/cl-descriptions.html
framing: exact-match — The claim paraphrases three separate statements on the source page, and each one matches what the page says.
pass3 · attribution · high confidence
#22
✅ Verified
content/blog/what-a-good-infrastructure-pull-request-looks-like-now/index.md:130
A pull request that meets these seven is quick to review for anyone, and quick review is what keeps infrastructure changes moving at the pace the rest of…
framing: The link text calls it the "continuous delivery guide" for setting up preview-on-every-PR, and the target page has a section on running previews on pull…; evidence: The link target exists: content/docs/iac/operations/continuous-delivery/_index.md (title "Continuous Delivery"). It covers the preview-on-every-PR workflow: "A CI/CD pipeline for infrastructure **must** run a preview on every pull request."; source: repo:content/docs/iac/operations/continuous-delivery/_index.md
source: repo:content/docs/iac/operations/continuous-delivery/_index.md
framing: exact-match — The link text calls it the "continuous delivery guide" for setting up preview-on-every-PR, and the target page has a section on running previews on pull requests.
pass1 · url · high confidence
#1
➖ Not a claim
content/blog/what-a-good-infrastructure-pull-request-looks-like-now/index.md:3
date: 2026-10-29
evidence: This line is the blog post's front-matter `date:` field, which is the author's own publication metadata. It doesn't assert anything about the world that could be checked. The date matches today (2026-10-29), so it isn't future-dated either.; source: content/blog/what-a-good-infrastructure-pull-request-looks-like-now/index.md:L3 (front matter)
source: content/blog/what-a-good-infrastructure-pull-request-looks-like-now/index.md:L3 (front matter)
pass3 · numerical · high confidence
#19
➖ Not a claim
content/blog/what-a-good-infrastructure-pull-request-looks-like-now/index.md:112
An agent can produce a confident, well-organized description for any diff, including a diff that does something other than what the description says.
framing: General commentary on how LLMs behave. No source is cited.; evidence: This is the author's editorial point about AI agents in general: they can write a fluent PR description that doesn't match the diff. It doesn't describe any Pulumi product behavior, and no Pulumi source or release note could confirm or…; source: content/blog/what-a-good-infrastructure-pull-request-looks-like-now/index.md L112 (editorial context only; no checkable product claim)
source: content/blog/what-a-good-infrastructure-pull-request-looks-like-now/index.md L112 (editorial context only; no checkable product claim)
framing: General commentary on how LLMs behave. No source is cited.
pass1 · behavior · medium confidence
#20
➖ Not a claim
content/blog/what-a-good-infrastructure-pull-request-looks-like-now/index.md:122
1. Preview output is attached and was generated from the latest commit.
evidence: This line is an item in the author's own checklist for reviewing PRs ("Preview output is attached and was generated from the latest commit"). It tells reviewers what to do and makes no factual claim that could be checked. "Latest commit"…; source: repo:content/blog/what-a-good-infrastructure-pull-request-looks-like-now/index.md#L122
source: repo:content/blog/what-a-good-infrastructure-pull-request-looks-like-now/index.md#L122
pass1 · temporal · high confidence
#21
➖ Not a claim
content/blog/what-a-good-infrastructure-pull-request-looks-like-now/index.md:124
3. Config changes and new `ignoreChanges` entries are named and justified.
evidence: This line is a numbered checklist item giving the author's advice on what a good infrastructure PR should include. It doesn't state a fact about time. The regex matched the word "new" in "new `ignoreChanges` entries," which describes…; source: repo:content/blog/what-a-good-infrastructure-pull-request-looks-like-now/index.md#L124
source: repo:content/blog/what-a-good-infrastructure-pull-request-looks-like-now/index.md#L124
pass1 · temporal · high confidence
#5
🤷 Unverifiable
content/blog/what-a-good-infrastructure-pull-request-looks-like-now/index.md:32
Google's SRE team reports that roughly 70% of outages are due to changes in a live system (per the SRE book introduction).
framing: Couldn't run the framing check because the supporting passage isn't in the truncated fetch. The figure is widely attributed to this chapter's Change…; evidence: The cited page returned HTTP 200 and is the right target (SRE book Chapter 1, Introduction), but the pre-fetched body is truncated before the Change Management section. The fetched text has no passage about the 70% figure, so I couldn't…; source: https://sre.google/sre-book/introduction/
source: https://sre.google/sre-book/introduction/
framing: Couldn't run the framing check because the supporting passage isn't in the truncated fetch. The figure is widely attributed to this chapter's Change Management section, so a reviewer reading the full page will probably confirm it.
pass2 · attribution · low confidence
#14
🤷 Unverifiable
content/blog/what-a-good-infrastructure-pull-request-looks-like-now/index.md:67
`pulumi preview --refresh` refreshes the state of the stack's resources before previewing.
framing: The fetched body has no passage to compare against the claim, so no framing could be assessed.; evidence: The cited drift page returned HTTP 200, but the pre-fetched body is cut off inside the site navigation. It has no passage about `pulumi preview --refresh`, so I can't quote support. The flag does exist in the Pulumi CLI, so the claim is…; source: https://www.pulumi.com/docs/deployments/concepts/drift/
source: https://www.pulumi.com/docs/deployments/concepts/drift/
framing: The fetched body has no passage to compare against the claim, so no framing could be assessed.
pass2 · api-surface · low confidence
No claims match the current filter.
Findings
| ID | Bucket | File:lines | Status | Disposition |
|---|---|---|---|---|
| F3 | 🚨 Outstanding | content/blog/what-a-good-infrastructure-pull-request-looks-like-now/index.md:123 | open | — |
The checklist numbers its items `1.` through `7.`; this repo's convention is that every ordered-list item begins with `1.` (Markdown renders the numbers). Change lines 123–128 to start with `1.`. | ||||
| F4 | 🚨 Outstanding | content/blog/what-a-good-infrastructure-pull-request-looks-like-now/index.md:67 | open | — |
The drift link is an absolute `https://www.pulumi.com/docs/deployments/concepts/drift/` URL; internal links must be site-relative. Change it to `/docs/deployments/concepts/drift/`. | ||||
| F1 | ❓ Author answer | content/blog/what-a-good-infrastructure-pull-request-looks-like-now/index.md:32 | open | — |
*"Google's SRE team reports that roughly 70% of outages are due to changes in a live…"* — verdict: unverifiable. The verifier couldn't find the 70% figure in the fetched SRE book page; confirm the source (see F1). Do this- **Line (verbatim):** The stakes are well documented. Google's SRE team reports that [roughly 70% of outages are due to changes in a live system](https://sre.google/sre-book/introduction/). Infrastructure changes are changes to a live system by definition, so the pull request is the last cheap place to catch the problem.
- **Why:** The figure is widely attributed to the SRE book introduction's change-management passage, but the verifier's fetch was truncated before it, so the statistic and attribution are unconfirmed.
- **Fix:** Confirm the linked page states the 70% figure and reply `/update-review F1: <the quoted passage>`. | ||||
| F2 | ❓ Author answer | content/blog/what-a-good-infrastructure-pull-request-looks-like-now/index.md:67 | open | — |
*"`pulumi preview --refresh` refreshes the state of the stack's resources before previewing."* — verdict: unverifiable. The verifier's fetch had no passage on this flag; confirm the behavior (see F2). Do this- **Line (verbatim):** **Differences between state and reality.** If someone changed a resource in the console, the preview compares your code against stale state. Run `pulumi preview --refresh` to refresh the state of the stack's resources before previewing, or `pulumi refresh --preview-only` to see the drift without applying it. Pulumi Cloud treats a [preview of a refresh as a drift detection run](https://www.pulumi.com/docs/deployments/concepts/drift/).
- **Why:** The fetched source had no passage describing `--refresh` on `pulumi preview`, so this behavior (repeated in checklist item 4) is unconfirmed.
- **Fix:** Check the `pulumi preview` CLI reference (`/docs/iac/cli/commands/pulumi_preview/`) and reply `/update-review F2: <what it says>`. | ||||
Editorial stances
Superlative, ranking, or comparative language the diff adds. No verdict — a page's own framing isn't fact-checkable — listed so a reviewer can confirm each is a stance the docs should take.
- content/blog/what-a-good-infrastructure-pull-request-looks-like-now/index.md L108 ## Does the standard change when an agent opens the pull request? positioning · found by regex
- content/blog/what-a-good-infrastructure-pull-request-looks-like-now/index.md L112 First, fluent prose is cheap now. An agent can produce a confident, well-organized description for any diff, including a diff that does something other than… positioning · found by regex
- content/blog/what-a-good-infrastructure-pull-request-looks-like-now/index.md L116 For the author, whether person or agent, the practical consequence is to build the checklist into the pipeline instead of the habit. Preview on every push… positioning · found by regex
- content/blog/what-a-good-infrastructure-pull-request-looks-like-now/index.md L22 What has changed is the volume. A [2025 study of agent-authored pull requests](https://arxiv.org/abs/2507.15003) (Li, Zhang, and Hassan) collected over… comparison · found by regex
- content/blog/what-a-good-infrastructure-pull-request-looks-like-now/index.md L36 The output of `pulumi preview` is the only artifact that shows what a change will do to real resources. positioning · found by regex, llm-atomic, llm-holistic
- content/blog/what-a-good-infrastructure-pull-request-looks-like-now/index.md L38 Pulumi's [continuous delivery guidance](/docs/iac/operations/continuous-delivery/) is direct about this: a CI/CD pipeline for infrastructure must run a… comparison · found by regex
- content/blog/what-a-good-infrastructure-pull-request-looks-like-now/index.md L38 Pulumi's [continuous delivery guidance](/docs/iac/operations/continuous-delivery/) is direct about this: a CI/CD pipeline for infrastructure must run a… positioning · found by regex
- content/blog/what-a-good-infrastructure-pull-request-looks-like-now/index.md L42 - Use `pulumi preview --diff` for a rich diff of the overall change. It is easier to scan than the default view when a stack has many resources. positioning · found by regex
Investigation log
- cross-sibling-reads
- not run (not in a templated section)
- external-claim-verification
- 16 of 22 claims verified (2 unverifiable, 0 contradicted) · 4 specialists (numerical, cross-reference, capability, framing); 0 cross-specialist corroborations · routed: 0 inline, 16 Pass 1, 4 Pass 2 (verified 2, contradicted 0, unverifiable 2), 2 Pass 3 (verified 1, contradicted 0, unverifiable 1).
- cited-claim-spot-checks
- 4 of 4 cited claims fetched and compared
- frontmatter-sweep
- ran on body + meta_desc
- temporal-trigger-sweep
- ran (recency words present in diff; spot-check in-review)
- code-execution
- not run (no `static/programs/` change)
- code-examples-checks
- ran (3 specialists: structural, existence, body-code-coverage); 0 findings
- editorial-balance-pass
- ran (single-subject, N/A)
Editorial balance
{
"files": [
{
"file": "content/blog/what-a-good-infrastructure-pull-request-looks-like-now/index.md",
"outliers": [
{
"heading": "The short checklist",
"lines": 9,
"ratio": 3.0
}
],
"sections": [
{
"heading": "Why is an infrastructure diff harder to review than application code?",
"lines": 2
},
{
"heading": "What should every infrastructure pull request attach? The preview output",
"lines": 6
},
{
"heading": "How do you state the blast radius in plain words?",
"lines": 7
},
{
"heading": "How do you keep config drift from slipping in silently?",
"lines": 5
},
{
"heading": "Which tests belong in an infrastructure pull request?",
"lines": 3
},
{
"heading": "How do you write the description for a reviewer who was not there?",
"lines": 3
},
{
"heading": "What and why",
"lines": 1
},
{
"heading": "Preview",
"lines": 2
},
{
"heading": "Blast radius",
"lines": 4
},
{
"heading": "Testing",
"lines": 1
},
{
"heading": "Verification and rollback",
"lines": 3
},
{
"heading": "Does the standard change when an agent opens the pull request?",
"lines": 4
},
{
"heading": "The short checklist",
"lines": 9
}
],
"stats": {
"mean": 3.8,
"median": 3,
"std": 2.3
},
"threshold_flags": [
{
"heading": "The short checklist",
"lines": 9,
"ratio": 3.0,
"type": "section-depth-3x-median"
}
]
}
],
"trigger": null
}
History
- 2026-10-29T00:11:41Z
18b077f
initial review (pending publication)