CI/CD Pipelines

The opinion

I've worked across Jenkins, Bitbucket Pipelines, and GitLab CI, sometimes on the same platform in the same week. The strongest lesson from that is how little the choice of CI system matters compared to where you put the logic.

This isn't hypothetical for me. Porting pipelines between CI systems is dramatically cheaper when each stage is a call to ./ci/build.sh than when each stage is forty lines of platform-specific YAML with inline shell fragments.

Keep the logic in scripts, keep the YAML thin

stages: [build, test, scan, deploy]

build:
  stage: build
  script: ["./ci/build.sh"]

test:
  stage: test
  script: ["./ci/test.sh"]

deploy:staging:
  stage: deploy
  script: ["./ci/deploy.sh staging"]
  environment: staging

Every one of those scripts runs on a laptop. That property is worth more than any feature a CI vendor advertises, because it converts "the pipeline is broken" from a push-and-wait guessing game into ordinary local debugging.

Give the scripts a consistent internal shape, too. The one I use:

1. Authenticate: acquire credentials, fail fast and clearly if they're missing.
2. Execute: do the actual work.
3. Validate: verify the intended end state, don't just check the exit code.
4. Roll back: a defined path when validation fails.

Step three is the one most scripts omit. A deploy script that exits 0 because the API accepted the request hasn't verified anything. The deployment can still fail thirty seconds later, and the pipeline will report success.

The minimum viable pipeline

Every service gets these stages before it's allowed to deploy anywhere. Not as bureaucracy. Each one catches a distinct failure class:

Branch protection underneath all of it: no direct commits to the main branch, ever. A pipeline that can be bypassed isn't a source of truth, it's a suggestion.

Fifteen minutes

Past roughly fifteen minutes, a pipeline stops being a feedback mechanism and becomes an interruption. Developers context-switch away, come back to a failure, and have lost the mental state they needed to fix it quickly. The cost isn't the fifteen minutes. It's the re-acquisition.

• Parallelize independent stages. Scanning doesn't need to wait for tests.
• Cache dependencies aggressively, and verify the cache is actually being hit. A misconfigured cache silently costs you the whole benefit.
• Run the expensive, slow suites on a schedule rather than per-commit, if they're genuinely needed.
• Track pipeline duration over time. It degrades gradually and nobody notices until it's twenty-five minutes.

Every failure needs a class

The most corrosive thing in a delivery system is a pipeline that fails for reasons unrelated to the change. Teams learn to re-run on red, and from then on the pipeline is no longer carrying information.

The fix starts with classification. Every failure is one of:

• A real defect: the pipeline did its job.
• Environment or configuration drift: fix the environment, not the test.
• Flake: quarantine it immediately and fix or delete it. A known-flaky test left in the suite is training the team to ignore failures.
• Infrastructure: runner capacity, network, registry. Usually invisible until you count.

You can't manage this by feel. Counting failures by class for a couple of weeks tends to produce a surprise. Most teams discover the majority of their failures aren't defects at all, which reframes what the pipeline is actually costing them.

Anti-patterns

• Jobs configured through a web UI. Not versioned, not reviewable, and gone if the server is.
• Copy-pasted pipeline definitions across repositories. Twenty repos, twenty subtly different deploy behaviors, and no way to fix them all at once.
• Credentials as plain pipeline variables. Use the secret store; short-lived credentials over static keys wherever the platform supports it.
• E2E suites as the primary gate. They fail combinatorially, mostly for configuration reasons. The argument is in the strategy page.
• Retry-until-green. The moment this becomes normal, the pipeline has stopped being a safety mechanism.

With Claude

Pipeline work is close to ideal for AI assistance: the inputs are text, the failures are text, and the feedback loop is fast.

Triaging a build failure

Classification first, cause second. Asking "why did this fail" invites a plausible story; asking it to classify first forces it to look at what kind of evidence is actually present.

Here is a failing build log.

1. Classify the failure: real defect / config drift / flake /
   infrastructure. State your confidence.
2. Quote the specific log lines that support that classification.
3. Give the most likely cause and the smallest check that would
   confirm or eliminate it.

If the log doesn't contain enough to classify it, say so and tell me
what to capture on the next run instead of guessing.

That last instruction is the one worth copying. Without it you get a confident answer built on insufficient evidence, which is worse than no answer because you'll act on it.

Porting a pipeline between CI systems

A genuinely well-suited task: mechanical, high-volume, and easy to verify. The trick is making it explicit about what doesn't translate.

Port this Bitbucket Pipelines config to GitLab CI.

Rules:
  - move all shell logic into ./ci/*.sh scripts, keep the YAML thin
  - preserve stage ordering and the existing branch conditions
  - map repository variables to GitLab CI/CD variables and list them
  - flag anything with no direct equivalent rather than approximating

Output: the YAML, the scripts, then a table of everything I need to
configure manually before the first run.

Writing the validation step

Deploy scripts routinely check that a command succeeded rather than that the system reached the intended state. Handing over the deploy script and asking specifically for post-conditions (what should be true afterwards, and how to check it) produces the step that was missing.