Terms
| Term | Meaning |
|---|---|
| Continuous integration | Merge often; build and test every change automatically |
| Continuous delivery | Every passing change is releasable; a human chooses when |
| Continuous deployment | Every passing change is released automatically |
Pipeline stages
- Checkout the exact revision
- Build the artifact once
- Test: lint and unit tests first, then integration
- Analyze: dependency, secret and image scans
- Publish with an immutable tag
- Deploy to a test environment and run smoke tests
- Promote the same artifact to production and verify
Build once, deploy many. Order checks by speed.
Jenkinsfile skeleton
pipeline {
agent { label 'docker' }
options {
timeout(time: 30, unit: 'MINUTES')
disableConcurrentBuilds()
}
environment {
IMAGE = "registry.example.com/shop-api"
TAG = "build-${env.BUILD_NUMBER}"
}
stages {
stage('Test') {
parallel {
stage('Unit') { steps { sh 'make test' } }
stage('Lint') { steps { sh 'make lint' } }
}
}
stage('Build') {
steps { sh 'docker build -t "$IMAGE:$TAG" .' }
}
stage('Push') {
when { branch 'main' }
steps {
withCredentials([usernamePassword(credentialsId: 'registry-creds',
usernameVariable: 'U', passwordVariable: 'P')]) {
sh 'echo "$P" | docker login registry.example.com -u "$U" --password-stdin'
sh 'docker push "$IMAGE:$TAG"'
}
}
}
}
post {
always { echo 'done' }
}
}Use single quotes around shell commands that reference secrets, so the shell expands them and Groovy does not.
Deployment strategies
| Strategy | Strength | Weakness |
|---|---|---|
| Recreate | Simple | Downtime |
| Rolling | No downtime, no extra capacity | Slower rollback, mixed versions |
| Blue-green | Instant switch and rollback | Double capacity |
| Canary | Limits blast radius | Needs traffic control and metrics |
Feature flags separate deploy from release. Remove them when done.
Database changes
- Expand: add new schema compatibly
- Migrate: write both, read new, backfill
- Contract: drop the old schema in a later release
Verification and rollback
kubectl rollout status deployment/shop-api -n shop --timeout=180s
kubectl rollout undo deployment/shop-api -n shop
curl -fsS https://shop.example.com/healthz/readyDelivery metrics
| Metric | Question |
|---|---|
| Deployment frequency | How often do we release? |
| Lead time for changes | Commit to production? |
| Change failure rate | Share of releases causing problems? |
| Time to restore | How fast do we recover? |
Declarative delivery
argocd app list
argocd app diff shop-api-prod
argocd app sync shop-api-prod
argocd app history shop-api-prod| Setting | Effect |
|---|---|
automated.prune |
Delete objects removed from configuration |
automated.selfHeal |
Revert manual cluster changes |
Pull model: credentials stay in the cluster, drift is detected, and rollback is a configuration revert.
Pipeline security
- Secrets come from a secret store, never from files in the source
- Short-lived credentials via identity federation
- Separate plan or build permissions from deploy permissions
- Scan images and dependencies; fail on high severity
- Deploy only from the main branch after review
Pipeline hygiene commands
yamllint .ci/pipeline.yml
actionlint
docker run --rm -v "$PWD":/src hadolint/hadolint hadolint /src/Dockerfile
shellcheck ci/*.sh
trivy image --severity HIGH,CRITICAL --exit-code 1 app:1.4
trivy fs --scanners vuln,secret .
syft app:1.4 -o spdx-json > sbom.json
cosign sign --key cosign.key registry.example.com/app@sha256:ABC123
cosign verify --key cosign.pub registry.example.com/app@sha256:ABC123Build, test and promote
make test
pytest -q --maxfail=1 --junitxml=report.xml
npm ci && npm test
docker build --pull --cache-from app:cache -t app:$SHORT_SHA .
docker push registry.example.com/app:$SHORT_SHA
docker buildx imagetools inspect registry.example.com/app:$SHORT_SHA
curl -fsS https://staging.example.com/healthz
curl -fsS -o /dev/null -w '%{http_code}\n' https://staging.example.com/versionDeploy and rollback
kubectl set image deploy/api app=registry.example.com/app:$SHORT_SHA -n shop
kubectl rollout status deploy/api -n shop --timeout=180s
kubectl rollout undo deploy/api -n shop
helm upgrade --install api ./chart -n shop --atomic --timeout 5m
curl -fsS https://example.com/healthz || exit 1