🔧DevOps Engineer

Containers share the host OS kernel with isolated user spaces, making them lightweight (megabytes, millisecond start) and portable. Virtual machines virtualize full hardware with a guest OS each, giving stronger isolation at the cost of gigabytes and slower boot. Use containers for dense microservices and VMs, or containers on VMs, for strong multi-tenant isolation.

Multistage builds use multiple FROM stages in one Dockerfile so compilers and build tooling live in early stages while only the compiled artifact copies into a slim final image. This shrinks images dramatically, removes build secrets and toolchains from production, and keeps the build hermetic without extra scripts. Order stages from least to most frequently changed and copy dependency manifests first to maximize layer cache hits.

CMD: sets the default command or arguments for a container and is easily overridden by arguments passed to docker run, with only one effective CMD per image. ENTRYPOINT: defines the fixed executable for the image so the container behaves like a wrapped binary, with CMD then supplying default flags. Prefer ENTRYPOINT for the program plus CMD for defaults, both in exec JSON form so signals reach the process cleanly.

Volumes: Docker-managed storage in the daemon area with driver support, easy backup, and clean sharing between containers, preferred for databases and durable state. Bind mounts: direct mappings of a host path into the container, great for local development live reload but coupled to host layout and file permissions. Prefer named volumes in production and bind mounts for iterative coding.

Each Dockerfile instruction creates a cached layer keyed by its inputs, so unchanged steps reuse cache and rebuilds finish in seconds. Order instructions from least to most frequently changed, copying dependency manifests and installing before application source so code edits do not invalidate heavy install layers. Add a clear invalidation story with pinned base tags and BuildKit cache mounts for package managers.

Docker Compose defines a multi-container application in one declarative YAML file with services, networks, and volumes started by a single command. It is ideal for local development, integration tests, and small single-host deployments where full orchestration would be overkill. Mention profiles, overrides per environment, and healthcheck-based startup ordering for dependent services.

Tag images immutably with registry, repository, and a unique tag such as a git SHA plus a moving tag like staging, then push with docker push after authenticating via the cloud provider credential helper. Pulls in production should pin the immutable digest so deploys are reproducible even if a tag moves. Mention retention policies, vulnerability scanning on push, and replication for multi-region pulls.

Enable BuildKit for parallel stage execution, cache mounts, and secret mounts that keep credentials out of layers, then restructure the Dockerfile so expensive dependency installs sit below rarely changing lines. Use cache mounts for package caches, bind mounts for context without copying, and separate test and lint stages that CI can run without bloating the final image. Measure with build history and trim the context with a tight ignore file.

Set requests to guarantee scheduling placement and limits to cap burst, sizing both from load-tested profiles rather than guesses. Memory limits trigger OOM kills when exceeded so size them above steady-state peaks, while CPU limits throttle instead of killing and overly tight values cause latency. Pair limits with HPA and alerts on throttling and near-limit usage.

Kubernetes orchestrates containers: it schedules pods onto nodes and handles service discovery, load balancing, rolling updates, autoscaling, self-healing and config and secret management declaratively. You describe desired state in manifests and controllers reconcile reality. Mention core objects like Pod, Deployment, Service, Ingress and ConfigMap, and when managed offerings like EKS or GKE beat self-hosting.

A Pod moves through Pending, Running, Succeeded or Failed phases while its containers cycle through Waiting, Running, and Terminated with exit codes and reasons. The restartPolicy of Always, OnFailure, or Never decides whether the kubelet restarts containers, and probes plus termination grace periods control graceful shutdown. Teach reading kubectl describe output for events, lastState, and exit codes first.

Deployments: manage stateless replicas with interchangeable pods, rolling updates, and easy scale-out behind a Service, ideal for web APIs and workers. StatefulSets: give each pod a stable name, ordered startup, and sticky storage identity for databases and queues that need persistent per-instance state. Choose Deployments by default and StatefulSets when identity and ordered storage matter.

ClusterIP: exposes the Service only inside the cluster on an internal virtual IP, the default for service-to-service traffic. NodePort: opens a static port on every node routing to the Service, useful for demos but crude for production exposure. LoadBalancer: provisions a cloud load balancer in front of the Service for public traffic, at extra cost per service. Prefer Ingress or Gateway in front of ClusterIP services to economize.

Ingress: the classic single-resource routing object with host and path rules, limited by divergent controller annotations and weak role separation. Gateway API: a layered model separating Gateway, HTTPRoute, and policy attachments with standard cross-namespace support and clearer persona boundaries. Mention migrating Ingress rules to HTTPRoutes for portable advanced traffic control.

ConfigMaps: hold non-sensitive configuration as key value pairs or mounted files, ideal for feature settings and service URLs. Secrets: hold sensitive data with base64 encoding, size limits, encryption-at-rest options, and tighter RBAC, injected via env or volumes. Never commit either raw to git, prefer external secret managers synced by operators, and rotate values regularly.

Helm packages manifests into versioned charts with values files, templating, and release history so one chart deploys consistently across environments. Values overrides per environment replace forked YAML, and rollback reverts to a prior release revision in one command. Mention chart dependencies, lint and template testing in CI, and pinning chart versions for reproducible deploys.

Helm: a packaging and release manager with charts, versioned releases, rollback, and lifecycle hooks, strong for distributing reusable applications. Kustomize: a template-free overlay system that patches base manifests per environment with strategic merges, built into kubectl and simpler for in-house services. Many teams render with Kustomize inside Helm or use Helm for third-party charts and Kustomize for their own.

Liveness: decides when to restart a stuck container, so keep it cheap and free of dependencies that would cause crash loops. Readiness: decides when a pod may receive Service traffic, ideal for draining during warmup or dependency outages. Startup: guards slow-starting containers by disabling the other probes until first success. Tune thresholds from observed timings, never probe external systems from liveness.

Namespaces scope names and policy so teams share clusters safely, while ResourceQuotas cap aggregate CPU, memory, and object counts per namespace. LimitRanges set per-pod defaults and bounds so one misconfigured workload cannot starve neighbors. Combine both with RBAC and network policy for solid multi-tenant isolation.

StorageClasses define provisioner profiles such as SSD tiers, PersistentVolumes represent actual capacity, and PersistentVolumeClaims are pod requests that bind to matching volumes dynamically. Access modes and reclaim policies decide sharing and lifecycle, so choose Retain for precious data and Delete for scratch. Mention snapshots and backup before stateful upgrades.

DaemonSets: run exactly one pod per matching node, ideal for node-level agents like log collectors, monitors, and network plugins that must be everywhere. Deployments: run a chosen replica count placed anywhere suitable, ideal for application serving with rolling updates. Pick DaemonSet for per-node coverage and Deployment for elastic application capacity.

Node affinity steers pods toward nodes with matching labels such as GPU types or zones, with required rules for hard constraints and preferred rules for soft wishes. Pod affinity and anti-affinity co-locate or spread workloads relative to each other for locality or fault isolation. Taints repel pods unless a matching toleration exists, enabling dedicated pools for special workloads.

Managed control planes remove etcd operations, upgrades, and HA plumbing so small platform teams ship faster with built-in IAM, logging, and autoscaling integrations. Self-hosting fits strict sovereignty, exotic networking, or version-pinning needs but demands deep etcd, upgrade, and control-plane on-call expertise. Price the managed fee against engineer time and incident risk before deciding.

Build a pipeline with stages: lint and unit tests on every commit, container image build with pinned tags, security and IaC scans, deploy to staging with integration and smoke tests, then progressive production rollout with rolling, blue-green or canary strategy plus automatic rollback on failed health checks. Keep pipelines fast and hermetic, promoting artifacts rather than rebuilding between stages.

Jenkins: self-hosted automation with vast plugin history and full runner control, powerful but carrying server maintenance and plugin debt. GitHub Actions: hosted event-driven workflows colocated with code, with matrix builds, reusable workflows, and managed runners that minimize ops. Choose Jenkins for exotic on-prem needs and Actions for fast setup near the repository.

Trunk-based: developers integrate into main daily behind flags with fast CI, suiting SaaS teams shipping continuously. GitFlow: long-lived develop and release branches with structured ceremonies, suiting versioned packaged software with parallel supported releases. Prefer trunk for services and reserve GitFlow for release-train products.

Build once per commit, stamp the artifact with the git SHA, run full tests and scans on that exact binary, then promote the same artifact through environments by changing only config. Registries or artifact stores with promotion lanes plus signed attestations prove what was tested is what ships. Rebuilding per environment is the classic mistake to call out.

GitOps declares desired cluster state in git and lets controllers continuously reconcile live state toward it with drift detection and automated sync. ArgoCD watches repositories, renders manifests, applies changes progressively with health checks, and rolls back via git revert. Protect the pattern with signed commits, branch policies, and separated app-of-apps layers.

Infrastructure as Code manages infrastructure through version-controlled declarative files instead of manual console clicks. Tools like Terraform, CloudFormation and Pulumi provide repeatability, reviewability, drift detection and environment parity. Emphasize code review for plans, remote state with locking, modular structure and applying changes through CI and CD.

Store state remotely in versioned object storage with locking such as S3 plus DynamoDB or native cloud backends so concurrent applies cannot corrupt it. Split state by environment and component to limit blast radius, review plans in CI before apply, and never hand-edit state except through moved blocks or imports. Encrypt at rest, restrict access tightly, and keep state free of secrets.

Terraform: cloud-agnostic IaC with a large provider ecosystem, expressive language, and plan-driven workflows that suit multi-cloud estates. CloudFormation: the native AWS-only service with deep service coverage, drift detection, and StackSets for account-wide rollouts. Choose Terraform for portability and modules, CloudFormation for purely AWS shops wanting native integration.

Place hot objects in Standard, cooler data in Infrequent Access or Glacier tiers, and automate transitions plus expirations with lifecycle rules matched to access patterns. Versioning protects against overwrites while noncurrent-version expiration bounds its cost, and replication covers regional resilience. Monitor with storage lens metrics and right-size after observing real access.

EC2: virtual servers you fully control for long-running, stateful, or specialized workloads, priced while running with scaling you manage. Lambda: event-driven functions with automatic scaling, sub-second billing, and no server management, ideal for spiky or glue workloads under platform limits. Choose EC2 for control and persistence, Lambda for bursty event handling with minimal ops.

Read replicas offload read-heavy traffic and serve regional or reporting queries with async replication lag you must design around. Multi-AZ keeps a synchronous standby in another zone with automatic failover for durability rather than scaling. Combine both with connection pooling, tuned instance classes, and tested failover runbooks.

Layer the four golden signals (latency, traffic, errors, saturation) plus business KPIs, using metrics with Prometheus, centralized structured logs and traces with OpenTelemetry, backed by dashboards and SLO-based alerts. Alert on symptoms with runbooks attached, keep on-call rotations sustainable, and review incidents blamelessly to convert findings into automation.

Emit namespaced metrics with dimensions, centralize logs with structured JSON and retention policies, and build dashboards around golden signals per service. Alarms on SLO burn or anomaly detection route to paging with runbook links, while composite alarms reduce noise during correlated failures. Review alarm history regularly and delete unactioned alerts.

Prometheus scrapes instrumented HTTP endpoints on intervals into a time-series store with labels, using exporters and push gateways where scraping cannot reach. PromQL aggregates with rate, histogram_quantile, and label matchers to build SLO burn and saturation queries behind Grafana dashboards. Size retention and cardinality budgets carefully since unbounded labels explode storage.

Design dashboards top-down from user-facing SLOs to service signals to infrastructure, with consistent panels, units, and links from alert to dashboard to runbook. Template variables make one dashboard serve many services, and alerts fire on burn-rate symptoms rather than raw CPU wiggles. Review dashboards quarterly so they track current architecture.

Loki: label-indexed log aggregation that stays cheap and pairs with Prometheus and Grafana, ideal for high-volume application logs with modest full-text needs. Elasticsearch: full inverted-index search with rich analytics and heavier ops cost, ideal for deep ad-hoc investigation and security analytics. Choose Loki for cost-efficient tail-and-filter flows and Elasticsearch for heavy search.

OpenTelemetry provides vendor-neutral SDKs, auto-instrumentation, and a collector pipeline that enriches, samples, and exports signals to any backend. Propagated trace context links requests across services so latency breakdowns and error attribution become visual. Standardize on semantic conventions early so dashboards stay portable across vendors.

SLIs: measured indicators like success ratio or p99 latency that quantify service behavior. SLOs: internal targets on those indicators, such as 99.9 percent success over 30 days, that drive alerting and prioritization. SLAs: contractual commitments to customers with remedies when missed, set looser than SLOs. Alert on SLO burn, not on every SLI wiggle.

Page only on user-facing symptoms with clear severity tiers, keep everything else as tickets, and attach runbooks with escalation paths to every alert. Follow-the-sun or capped weekly rotations with compensation and post-incident rest protect health, while alert reviews prune noisy rules. Track paging volume per person as a first-class reliability metric.

Blue-green keeps two identical environments: blue serves live traffic while green gets the new release, then the router flips over instantly with instant rollback by flipping back. It trades double capacity for safe zero-downtime deploys with full pre-cutover testing. Contrast with rolling (gradual) and canary (percentage-based) strategies.

Canary: shifts a small traffic percentage to the new version while comparing error and latency metrics, then ramps progressively with automatic rollback on regression. Blue-green: stands up a full parallel environment and flips all traffic at once for instant cutover and instant revert at double capacity cost. Use canary for metric-gated risk reduction and blue-green for fast deterministic swaps.

Rolling updates replace pods incrementally within surge and unavailable budgets so capacity stays serving while the new revision proves healthy. Readiness gates hold each new pod out of the Service until warm, and failed rollouts pause or roll back automatically. Tune maxSurge and maxUnavailable from load tests and watch rollout status during the change.

Flags wrap new behavior in runtime toggles served by a flag system, so code ships dark and product enables it for percentages or cohorts later. This enables trunk shipping, fast kill-switches, and A/B measurement with per-flag ownership and expiry to avoid debt. Mention testing both flag states and cleaning up stale flags.

TCP is connection-oriented with handshakes, ordering, retransmission and congestion control, guaranteeing reliable byte streams for web, files and APIs. UDP is connectionless datagrams with no guarantees, giving lower latency for gaming, VoIP and DNS where occasional loss beats delay. Mention QUIC as a modern hybrid running HTTP/3 over UDP.

Carve the CIDR into public subnets holding load balancers and NAT gateways plus private subnets for application and data tiers across at least two availability zones. Route public subnets to an internet gateway and private subnets through NAT, locking cross-tier traffic with security groups and NACLs. Reserve address space generously and peer or transit-gateway connect VPCs instead of overlapping ranges.

ALB: layer-7 load balancing with host and path routing, header rules, and content features, ideal for HTTP microservices. NLB: layer-4 TCP and UDP balancing with extreme throughput, static IPs, and low latency for non-HTTP or performance-critical traffic. Pick ALB for smart HTTP routing and NLB for raw speed and static addressing.

Start at the client with resolver config and cache, then query authoritatives with dig plus trace to separate propagation, record, and delegation errors from local misconfig. Check TTLs during migrations, security-group blocks on port 53, and Kubernetes CoreDNS or node-local cache health for cluster symptoms. Monitor resolution latency and failure ratios so issues page before users notice.

Client and server negotiate versions and ciphers, authenticate via the certificate chain, exchange key material, and switch to encrypted records, with mutual TLS adding client certificates. Debug with openssl s_client for chain and expiry, verify intermediate completeness and SAN coverage, and watch for clock skew or SNI mismatches. Automate renewal with short-lived certificates and expiry alerts.

Forward proxies: sit near clients and fetch on their behalf for filtering, caching, or egress control across many destinations. Reverse proxies: sit in front of servers terminating TLS, balancing load, caching, and shielding origins for one or few applications. Name Squid-style egress versus Nginx-style ingress as canonical examples.

Never bake secrets into images or repos: inject at runtime from a manager such as AWS Secrets Manager, Vault or GCP Secret Manager with rotation, least-privilege IAM, audit logging and short-lived credentials where possible. Separate config from secrets, scan for leaks in CI, and use workload identity (IRSA, Workload Identity) instead of long-lived keys.

Start FROM pinned minimal base images, install only required packages, run as a non-root user, and scan every build for OS and language vulnerabilities with admission gates on severity. Sign images, verify signatures at deploy time, and rebuild regularly to pick up patched bases instead of patching running containers. Keep a short list of approved bases so security review stays tractable.

RBAC binds subjects like users, groups, and service accounts to Roles or ClusterRoles through RoleBindings, granting only listed verbs on listed resources. Scope tightly with namespaced Roles by default, reserve cluster-wide grants for platform needs, and audit with access review commands. Prefer short-lived workload identities over static tokens wherever possible.

IAM policies grant explicit allow or deny actions on resources, attached to roles that humans or workloads assume temporarily instead of holding static keys. Scope policies to exact actions and resource ARNs, require MFA and conditions for sensitive paths, and audit with access advisors and credential reports. Prefer instance and pod identity roles so credentials rotate automatically.

Every file carries owner, group, and other bits for read, write, and execute, inspected with ls and set with chmod in symbolic or octal form. Directories need execute to traverse and write to create entries, while umask shapes defaults for new files. Prefer group-based access and sudo rules over broad 777 permissions that invite incidents.

Disable password logins and root SSH, require key or certificate-based auth with MFA where possible, and restrict source ranges through bastions or private connectivity. Harden the daemon with protocol defaults, idle timeouts, and fail2ban style throttling, then centralize auth logs and rotate keys on offboarding. Prefer short-lived certificates over permanent authorized keys.

NetworkPolicies whitelist allowed ingress and egress by pod selectors, namespace selectors, and port ranges so denied traffic drops by default under a deny-all baseline. Apply default-deny per namespace first, then open only required service paths including DNS, and test with connectivity probes in CI. Remember enforcement needs a supporting CNI plugin.

Generate SBOMs at build time to inventory every dependency, scan them continuously for new CVEs, and sign artifacts so admission controllers only deploy trusted provenance. Pin base images by digest, verify signatures with policy engines, and rebuild on advisory changes. Treat unsigned or unattested images as undeployable in regulated paths.

Issue new credentials alongside old ones, roll them through secret managers with versioned references, and restart or reload consumers in waves while both remain valid. Dynamic short-lived credentials from Vault or cloud IAM remove rotation pain entirely for supported datastores. Audit every consumer first since one missed client causes the outage rotation meant to prevent.

Horizontal scaling adds more instances behind a load balancer for near-unlimited, fault-tolerant capacity, but needs stateless design and shared state. Vertical scaling enlarges one machine, which is simple but capped and creates a single point of failure. In Kubernetes terms this maps to HPA for replicas versus VPA for pod size; prefer horizontal for elasticity.

HPA watches metrics such as CPU, memory, or custom application signals and adjusts replica counts between min and max to hold target utilization. It needs sane resource requests since scaling math divides observed use by requested values, plus cooldown behavior to avoid flapping. Pair it with Cluster Autoscaler for nodes and load-test scaling thresholds before peak events.

HPA: scales the replica count outward for stateless workloads under varying request load, the default elasticity answer. VPA: resizes individual pod requests and limits, even recreating pods, suited to right-sizing steady workloads that cannot simply replicate. They clash when both target the same metrics, so run VPA in recommendation mode alongside HPA or scope them to different resources.

Auto Scaling Groups maintain desired capacity across zones using launch templates, scaling policies on metrics like CPU or queue depth, and health checks that replace failed instances. Warm pools and lifecycle hooks smooth deploys, while scheduled and predictive scaling pre-position capacity for known peaks. Test termination policies so scale-in removes the right instances first.

A CDN caches static and cacheable dynamic content at edge points of presence so users fetch from nearby instead of the origin. Tiered caching, request collapsing, and shielding protect origins during spikes while TLS termination and WAF features add security. Size cache keys and TTLs deliberately and purge surgically on content updates.

Define target traffic from history plus growth, replay realistic mixes with gradual ramps using staged environments at production scale, and watch saturation, latency percentiles, and error budgets rather than averages. Pre-scale stateful tiers, verify autoscaling headroom, and rehearse failure injection so limits are known before the event. Record results as the capacity model for next time.

Producers enqueue work durably while consumers poll at their own pace, so traffic bursts buffer in the queue instead of overwhelming downstream systems. Visibility timeouts, dead-letter queues, and batch sizing give at-least-once processing with poison-message isolation. Scale consumers on queue depth and alert on age-of-oldest-message for real backpressure signal.

Follow a runbook: check pipeline logs and the exact failing step, inspect recent changes with diffs and image tags, verify health and readiness probes, resource limits and config or secret mounts, then look at app logs, metrics and traces around deploy time. Mitigate first with rollback or pinning last-good, reproduce in staging, and fix forward with a test that would have caught it.

Declare severity, page the commander, and open a single coordination channel with a live timeline to stop parallel confusion. Mitigate first by rolling back, shedding load, or failing over, then diagnose with recent deploys, metrics, and traces before attempting risky fixes. Close with a blameless review whose action items have owners and deadlines.

A good postmortem states impact, timeline, root causes, and contributing factors without naming or shaming, then converts findings into owned preventive actions. It distinguishes triggers from systemic causes and updates runbooks, tests, or automation rather than adding process theater. Share the write-up org-wide so every team learns from one outage.

Define units with restart policies, resource limits, and dependency ordering, then enable them for boot persistence across reboots. Inspect with status, filtered journalctl time ranges, and exit codes to separate app crashes from signal kills or OOM events. Mention drop-in overrides and log rotation so service config stays maintainable.

Read pod events and previous-container logs first since crash output usually names the missing env, bad flag, or failing probe. OOMKilled with exit 137 means the working set exceeded its memory limit, so profile heap, fix leaks, then raise limits and requests deliberately. Crash loops from config or image errors need manifest fixes, not bigger limits, so separate the two causes early.

Confirm with node conditions and kubelet eviction signals, then find culprits among container logs, image layers, and emptyDir volumes using du sorted scans. Evict or relocate offenders, add log rotation and image garbage-collection thresholds, and move durable data to networked volumes. Alert on disk fill rate rather than static thresholds to act before evictions.

Chaos engineering injects controlled failures like pod kills, latency, or zone loss to prove resilience hypotheses before real outages do. Run game days in staging first with explicit blast-radius limits, abort conditions, and observers, then graduate small experiments to production during business hours. Every finding must become an automated fix or it was theater.