Linux_Practical_Interview_501 750

Linux Practical Interview Questions (501-750)

Level 12: Logging and Observability

Monitoring Tools

Q501: How do you configure Grafana dashboards?

What it is: A Grafana dashboard is JSON defining panels laid out on a grid, each with PromQL targets and legend formats for CPU, memory, and network series.

Why it is used: To visualize key system metrics in one place for monitoring and incident response.

{
  "dashboard": {
    "title": "System Overview",
    "panels": [
      {
        "id": 1,
        "title": "CPU Usage",
        "type": "graph",
        "gridPos": {"x": 0, "y": 0, "w": 12, "h": 8},
        "targets": [
          {
            "expr": "100 - (avg by (mode) (irate(node_cpu_seconds_total{mode='idle'}[5m])) * 100)",
            "legendFormat": "{{mode}}"
          }
        ]
      },
      {
        "id": 2,
        "title": "Memory Usage",
        "type": "graph",
        "gridPos": {"x": 12, "y": 0, "w": 12, "h": 8},
        "targets": [
          {
            "expr": "node_memory_MemTotal_bytes - node_memory_MemAvailable_bytes",
            "legendFormat": "Used"
          },
          {
            "expr": "node_memory_MemAvailable_bytes",
            "legendFormat": "Available"
          }
        ]
      },
      {
        "id": 3,
        "title": "Network Traffic",
        "type": "graph",
        "gridPos": {"x": 0, "y": 8, "w": 12, "h": 8},
        "targets": [
          {
            "expr": "irate(node_network_receive_bytes_total{device!='lo'}[5m])",
            "legendFormat": "{{device}} RX"
          },
          {
            "expr": "irate(node_network_transmit_bytes_total{device!='lo'}[5m])",
            "legendFormat": "{{device}} TX"
          }
        ]
      }
    ]
  }
}

Interview talking point: “legendFormat turns raw series into readable labels such as per-mode CPU or device RX and TX, and irate over a 5m window shows the most recent per-second rate.”

Q502: How do you configure Grafana?

What it is: Grafana is a dashboard and visualization server that graphs data from configured data sources. It ships with a default admin and admin login and accepts dashboards defined as JSON panels.

Why it is used: To turn metric queries into interactive dashboards shared across teams for monitoring system health.

# Install
apt install grafana
# or
yum install grafana

# Start
systemctl start grafana-server
# Default: admin/admin

# Add data source
# HTTP URL: http://localhost:9090 (Prometheus)

# Create dashboard (JSON)
{
  "dashboard": {
    "title": "System Monitoring",
    "panels": [
      {
        "title": "CPU Usage",
        "type": "graph",
        "targets": [
          {
            "expr": "100 - (avg by (instance) (irate(node_cpu_seconds_total{mode='idle'}[5m])) * 100)"
          }
        ]
      }
    ]
  }
}

Interview talking point: “Grafana only visualizes; the PromQL expression in each panel target runs against the Prometheus data source, so the dashboard JSON can be exported and reimported to move a whole dashboard between environments.”

Q503: How do you configure Grafana?

What it is: Grafana is installed with apt, pointed at a Prometheus datasource on port 9090, and used to build dashboards from node_exporter metric names.

Why it is used: To visualize node health over time in panels engineers can read at a glance.

# Install
apt install grafana

# Configure datasource
# HTTP URL: http://localhost:9090

# Create dashboard
# Add panel with query
# node_exporter metrics:
# - node_cpu_seconds_total
# - node_memory_MemAvailable_bytes
# - node_filesystem_avail_bytes

Interview talking point: “Grafana stores no data itself; queries like node_cpu_seconds_total and node_memory_MemAvailable_bytes run against Prometheus, so verifying the datasource is the first debugging step.”

Q504: How do you configure Nagios?

What it is: Nagios runs custom check scripts plus service definitions; the sample parses df usage and exits with a code meaning OK or CRITICAL.

Why it is used: To alert operators when disk usage crosses a threshold before users notice.

# Install
apt install nagios4

# Create check script
# /usr/local/nagios/lib/check_disk.sh
#!/bin/bash
USAGE=$(df -h / | tail -1 | awk '{print $5}' | sed 's/%//')
if [ "$USAGE" -gt 90 ]; then
    echo "CRITICAL - Disk usage is ${USAGE}%"
    exit 2
fi
echo "OK - Disk usage is ${USAGE}%"
exit 0

# Configure service
# /etc/nagios4/conf.d/services.cfg
define service{
    host_name           localhost
    service_description Disk Usage
    check_command       check_disk
}

Interview talking point: “Nagios maps exit codes to states, so exit 0 is OK and exit 2 is CRITICAL, and the script strips the percent sign with sed so the numeric comparison works.”

Q505: How do you configure Prometheus alerting?

What it is: This alerting rule evaluates a CPU expression, waits for it to hold for 5 minutes, then fires with a severity label and a summary annotation using the instance label.

Why it is used: To turn a metric expression into a firing alert with severity and message.

groups:
- name: alerts
  rules:
  - alert: HighCPU
    expr: 100 - (avg by (instance) (rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100) > 80
    for: 5m
    labels:
      severity: warning
    annotations:
      summary: "High CPU usage on {{ $labels.instance }}"

Interview talking point: “The for clause demands the condition hold continuously before firing, which filters out brief spikes, while labels drive routing and annotations carry the human-readable text.”

Q506: How do you configure Zabbix?

What it is: Zabbix is an agent-based monitoring platform with a PHP frontend and a MySQL backend. Its schema is imported into the database, and the agent points at the server through Server, ServerActive, and Hostname settings.

Why it is used: To monitor hosts centrally through agents that report metrics and availability back to one server.

# Install Zabbix server
apt install zabbix-server-mysql zabbix-frontend-php

# Create database
mysql -u root -p
CREATE DATABASE zabbix CHARACTER SET utf8 COLLATE utf8_bin;
CREATE USER 'zabbix'@'localhost' IDENTIFIED BY 'password';
GRANT ALL PRIVILEGES ON zabbix.* TO 'zabbix'@'localhost';
FLUSH PRIVILEGES;
quit;

# Import schema
zcat /usr/share/zabbix-sql-scripts/mysql/server.sql.gz | mysql -u zabbix -p zabbix

# Configure server
# /etc/zabbix/zabbix_server.conf
DBPassword=password

# Install agent
apt install zabbix-agent

# Configure agent
# /etc/zabbix/zabbix_agentd.conf
Server=127.0.0.1
ServerActive=127.0.0.1
Hostname=Zabbix server

Interview talking point: “The agent Server and ServerActive settings both name the allowed server address, while Hostname must match the host name registered in Zabbix, otherwise the server silently discards incoming data.”

Q507: How do you configure Zabbix?

What it is: The Zabbix server runs on a MySQL backend: the database and user are created, the schema is imported, config gets the DB password, and services start.

Why it is used: To run a full monitoring platform with a web frontend and historical data.

# Install Zabbix server
apt install zabbix-server-mysql zabbix-frontend-php

# Configure database
mysql -u root -p
CREATE DATABASE zabbix CHARACTER SET utf8 COLLATE utf8_bin;
CREATE USER 'zabbix'@'localhost' IDENTIFIED BY 'password';
GRANT ALL PRIVILEGES ON zabbix.* TO 'zabbix'@'localhost';
FLUSH PRIVILEGES;
quit;

# Import schema
zcat /usr/share/zabbix-sql-scripts/mysql/server.sql.gz | mysql -u zabbix -p zabbix

# Configure Zabbix
# /etc/zabbix/zabbix_server.conf
DBPassword=password

# Start services
systemctl start zabbix-server
systemctl start apache2

Interview talking point: “The schema loads by piping zcat of server.sql.gz straight into mysql, and DBPassword in zabbix_server.conf must match the password given to the created zabbix database user.”

Q508: How do you set up Prometheus?

What it is: Prometheus scrapes metrics on a schedule; prometheus.yml sets scrape_interval 15s and a node job targeting localhost:9100, where node-exporter publishes host-level metrics.

Why it is used: Collecting time-series metrics for monitoring and alerting.

# Install
docker run -d -p 9090:9090 prom/prometheus

# prometheus.yml
global:
  scrape_interval: 15s
scrape_configs:
  - job_name: 'node'
    static_configs:
      - targets: ['localhost:9100']

# Add node exporter
docker run -d -p 9100:9100 prom/node-exporter

Interview talking point: “Prometheus pulls rather than pushes, so each target exposes an exporter: node-exporter on 9100 is the endpoint the node job scrapes, and dashboards or alerts then query that stored data.”

Q509: How do you set up Zabbix?

What it is: Zabbix is a monitoring platform with a MySQL database and PHP frontend, plus an agent installed on each host and pointed at the server through zabbix_agentd.conf.

Why it is used: Centralized metrics, thresholds, and alerting across many hosts.

# Install Zabbix server
apt install zabbix-server-mysql zabbix-frontend-php

# Configure database
mysql -e "create database zabbix character set utf8 collate utf8_bin;"
mysql zabbix < /usr/share/zabbix-sql.sql

# Add agent
apt install zabbix-agent
# Edit /etc/zabbix/zabbix_agentd.conf
Server=192.168.1.10

Interview talking point: “The Server line in the agent config both points the agent at the Zabbix server and restricts which address may query it, so Server=192.168.1.10 doubles as access control.”

Q510: How do you use Cockpit with Kubernetes?

What it is: Cockpit extends to containers with cockpit-podman for the /podman view and cockpit-kubernetes for Kubernetes resources.

Why it is used: To manage containers and clusters from the same web console as the host itself.

# Install
apt install cockpit-podman

# Enable
systemctl enable --now cockpit.socket

# Podman
# https://server:9090/podman

# Kubernetes
# Install kubernetes plugin
apt install cockpit-kubernetes

Interview talking point: “Installing cockpit-podman and cockpit-kubernetes is what adds those URLs, since Cockpit only ships panels for packages that are actually installed.”

Q511: How do you use Cockpit?

What it is: Cockpit is a web console for Linux servers, enabled through cockpit.socket and reached at https://server:9090.

Why it is used: To manage storage, networking, and services from a browser with limited shell access.

# Install
apt install cockpit

# Enable
systemctl enable --now cockpit.socket

# Access
# https://server:9090

# Modules
apt install cockpit-storaged
apt install cockpit-networkmanager
apt install cockpit-packagekit

Interview talking point: “Cockpit is modular, so packages like cockpit-storaged and cockpit-networkmanager add panels, meaning a minimal install only exposes what you explicitly installed.”

Q512: How do you use Glances?

What it is: Glances is a Python-based system monitoring tool that shows CPU, memory, disks, and processes in one terminal view. It can run as a server with -s and connect remotely with -c.

Why it is used: To get a quick cross-section of system health locally or on a remote machine.

# Install
pip install glances

# Run
glances

# Server mode
glances -s

# Remote
glances -c server-ip

Interview talking point: “glances -s starts server mode on one host and glances -c server-ip attaches a client to it, so you can inspect a remote box without an interactive login. The pip install option also means it is not tied to distro packages.”

Q513: How do you use Grafana Tempo?

What it is: Grafana Tempo is a distributed tracing backend deployed with Helm, storing traces in an S3 bucket and exposing its service as a LoadBalancer.

Why it is used: To keep large volumes of trace data at low cost while Grafana queries it during debugging.

# Helm
tempo:
  repository: grafana/tempo
  tag: latest
  service:
    type: LoadBalancer

  storage:
    trace:
      backend: s3
      s3:
        bucket: tempo-traces

Interview talking point: “The Helm values separate the two concerns: storage trace backend s3 points traces at bucket tempo-traces, while service type LoadBalancer exposes access outside the cluster, so storage and reachability are configured independently.”

Q514: How do you use Grafana with Prometheus?

What it is: In Grafana you add Prometheus as a data source using its URL, then import a dashboard by pasting its JSON or entering its numeric ID.

Why it is used: To graph Prometheus metrics on shareable dashboards without building your own frontend.

# Add datasource
# Configuration -> Data Sources -> Add Prometheus
# URL: http://prometheus:9090

# Add dashboard
# Dashboards -> Import
# JSON or ID

Interview talking point: “Grafana is only a rendering layer, so the datasource URL must point at Prometheus first, and importing by dashboard ID gives you ready-made panels without writing PromQL yourself.”

Q515: How do you use Grafana?

What it is: Grafana renders dashboards from data sources; it runs on port 3000, registers a Prometheus datasource through its API, and builds panels from Prometheus queries.

Why it is used: Giving teams visual dashboards over their metrics.

# Install
docker run -d -p 3000:3000 grafana/grafana

# Add data source
# POST /api/datasources
{
  "name": "Prometheus",
  "type": "prometheus",
  "url": "http://localhost:9090"
}

# Create dashboard
# Add panels with Prometheus queries

Interview talking point: “Grafana stores no metrics itself, it queries the datasource URL you configure, so pointing dashboards at a different Prometheus instance is a datasource change rather than a rebuild.”

Q516: How do you use Grafana?

What it is: Grafana is a dashboard and visualization server, run here on port 3000. A datasource is registered through its REST API with basic auth, and dashboards are imported as JSON via UI or API.

Why it is used: To graph metrics from Prometheus and other sources and build operational dashboards.

# Install
docker run -d -p 3000:3000 grafana/grafana

# Add datasource
curl -X POST http://localhost:3000/api/datasources \
  -H "Content-Type: application/json" \
  -u admin:admin \
  -d '{"name":"Prometheus","type":"prometheus","url":"http://localhost:9090","access":"proxy"}'

# Dashboard JSON
# Import via UI or API

Interview talking point: “The curl call posts to /api/datasources with basic authentication, showing Grafana can be configured entirely through its HTTP API. access proxy means Grafana itself queries Prometheus, so the browser never needs direct access to the datasource.”

Q517: How do you use Nagios?

What it is: Nagios monitors through defined commands; commands.cfg maps check_disk to a plugin taking warning and critical thresholds, and a service definition calls check_disk!20%!10% for a host.

Why it is used: Scheduled checks with threshold-based alerting on services.

# Install
apt install nagios4

# Add check to commands.cfg
define command{
    command_name check_disk
    command_line $USER1$/check_disk -w $ARG1$ -c $ARG2$
}

# Add service
define service{
    host_name localhost
    service_description Disk
    check_command check_disk!20%!10%
}

Interview talking point: “The exclamation marks pass arguments into the command, so check_disk!20%!10% fills $ARG1$ and $ARG2$, letting one command definition serve many services with different thresholds.”

Q518: How do you use Nagios?

What it is: Nagios monitors hosts and services through plugin commands. commands.cfg defines check_local_disk with warning and critical thresholds, services.cfg applies it to a host, and the service is then enabled and started.

Why it is used: To schedule checks and get alerted when disk, CPU, or services cross thresholds.

# Install
apt install nagios4

# Commands /etc/nagios4/conf.d/commands.cfg
define command{
    command_name check_local_disk
    command_line $USER1$/check_disk -w $ARG1$ -c $ARG2$ -p $ARG3$
}

# Services /etc/nagios4/conf.d/services.cfg
define service{
    host_name localhost
    service_description Disk Space
    check_command check_local_disk!20%!10%!/
}

# Enable checks
systemctl enable nagios4
systemctl start nagios4

Interview talking point: “The bang-separated arguments in check_local_disk map positionally onto $ARG1$, $ARG2$, and $ARG3$ in the command definition, so 20% is the warning level and / is the path checked. That separation lets one command serve many services with different thresholds.”

Q519: How do you use Nagios?

What it is: Nagios monitors hosts and services by running check scripts that exit with a status code. The example script parses df output and returns 0 for OK, 1 for warning, and 2 for critical.

Why it is used: To alert administrators when checks like disk usage cross defined thresholds before users are affected.

# Install
apt install nagios4
# or
yum install nagios

# Create check script
#!/bin/bash
# /usr/local/nagios/lib/check_disk.sh
USAGE=$(df -h / | tail -1 | awk '{print $5}' | sed 's/%//')
if [ "$USAGE" -gt 90 ]; then
    echo "CRITICAL - Disk usage is ${USAGE}%"
    exit 2
elif [ "$USAGE" -gt 80 ]; then
    echo "WARNING - Disk usage is ${USAGE}%"
    exit 1
else
    echo "OK - Disk usage is ${USAGE}%"
    exit 0
fi

# Define service
# /etc/nagios4/conf.d/services.cfg
define service{
    host_name           localhost
    service_description Disk Usage
    check_command       check_disk
    check_interval      5
}

Interview talking point: “Nagios relies on exit codes rather than printed text, so a check script is judged by 0, 1, or 2 and its message is just what operators read, while the service definition check interval drives how often it reruns.”

Q520: How do you use Netdata?

What it is: Netdata is a real-time monitoring agent installed with a kickstart script, running as the netdata user, and serving a live dashboard on port 19999.

Why it is used: To get instant per-second system metrics in a browser without setting up a monitoring server.

# Install
bash <(curl -Ss https://my-netdata.io/kickstart.sh)

# Configure
# /etc/netdata/netdata.conf
[global]
    run as user = netdata

# Access
# http://localhost:19999

Interview talking point: “Netdata is agent-first and self-contained: install it, open port 19999, and metrics are already charted, unlike Prometheus which needs exporters plus a separate UI. The config shows it drops privileges to the netdata user rather than running as root.”

Q521: How do you use Prometheus Alertmanager?

What it is: AlertmanagerConfig configures alert routing and receivers inside the cluster: this one groups by alertname, sends to an email receiver, and notifies when alerts resolve.

Why it is used: To route firing alerts to the right people without hand editing global Alertmanager configuration.

apiVersion: monitoring.coreos.com/v1beta1
kind: AlertmanagerConfig
metadata:
  name: email-config
spec:
  route:
    groupBy: [alertname]
    receiver: email
  receivers:
  - name: email
    emailConfigs:
    - to: admin@example.com
      sendResolved: true

Interview talking point: “groupBy alertname batches related firing alerts into one notification, and sendResolved true closes the loop by emailing when the alert clears — both settings matter when tuning alert noise.”

Q522: How do you use Prometheus Operator?

What it is: The Prometheus Operator is installed with kubectl apply of the bundle, and a ServiceMonitor selects services by label match and lists the endpoint port to scrape.

Why it is used: To manage Prometheus monitoring in Kubernetes through labelled custom objects.

# Install
kubectl apply -f https://raw.githubusercontent.com/prometheus-operator/prometheus-operator/master/bundle.yaml

# Create ServiceMonitor
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: myapp
spec:
  selector:
    matchLabels:
      app: myapp
  endpoints:
  - port: web

Interview talking point: “The ServiceMonitor decouples scraping from pod details because Prometheus discovers targets through label selectors, so exposing the named service port is enough to get scraped automatically.”

Q523: How do you use Prometheus Operator?

What it is: The Prometheus Operator manages monitoring through custom resources: a Prometheus resource selects ServiceMonitors by label, and each ServiceMonitor points at service endpoints by label and port name.

Why it is used: To manage Prometheus declaratively while teams opt their services into scraping by labeling a ServiceMonitor.

apiVersion: monitoring.coreos.com/v1
kind: Prometheus
metadata:
  name: prometheus
spec:
  serviceAccountName: prometheus
  serviceMonitorSelector:
    matchLabels:
      team: frontend
  resources:
    requests:
      memory: 400Mi

---
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: myapp
  labels:
    team: frontend
spec:
  selector:
    matchLabels:
      app: myapp
  endpoints:
  - port: web

Interview talking point: “Everything is label based: Prometheus has serviceMonitorSelector matching team frontend, and the ServiceMonitor carries that label while its selector matches app myapp, so a team scrapes its own targets just by labeling.”

Q524: How do you use Prometheus?

What it is: Prometheus is a metrics collection server, here run in Docker and configured by prometheus.yml. Scrape jobs pull endpoints every 15 seconds, including itself on 9090 and node-exporter on 9100.

Why it is used: To collect time-series metrics from hosts and services for alerting and dashboards.

# Install
docker run -d -p 9090:9090 prom/prometheus

global:
  scrape_interval: 15s

scrape_configs:
  - job_name: 'prometheus'
    static_configs:
      - targets: ['localhost:9090']
  - job_name: 'node'
    static_configs:
      - targets: ['localhost:9100']

# Node exporter
docker run -d -p 9100:9100 prom/node-exporter

Interview talking point: “Prometheus pulls metrics by scraping targets rather than agents pushing to it, so every exporter must be reachable from the server. Node exporter exposes host-level statistics on port 9100, and scrape_interval sets how often samples are collected.”

Q525: How do you use Webmin?

What it is: Webmin is a browser-based administration panel served on port 10000 with modules for system, servers, networking, and hardware.

Why it is used: To administer a server through a GUI when shell work is impractical.

# Install
apt install webmin

# Access
# https://server:10000

# Modules available
# System, Servers, Networking, Hardware, Clusters, Others

Interview talking point: “Webmin’s module categories such as Servers and Networking organize the panel, and it listens on https://server:10000 rather than Cockpit’s port 9090.”

Q526: How do you use Zabbix?

What it is: Zabbix is a monitoring platform with a server, a PHP web frontend, and agents on monitored hosts. The server config points at its MySQL database, and the agent config names the server to report to.

Why it is used: To collect metrics and trigger alerts across a fleet from a central server.

# Install
apt install zabbix-server-mysql zabbix-frontend-php

# Configure
# /etc/zabbix/zabbix_server.conf
DBHost=localhost
DBName=zabbix
DBUser=zabbix
DBPassword=password

# Web UI
# http://server/zabbix

# Agent
apt install zabbix-agent
# /etc/zabbix/zabbix_agentd.conf
Server=192.168.1.10
ServerActive=192.168.1.10
Hostname=localhost

Interview talking point: “Server and ServerActive in zabbix_agentd.conf differ: Server is for passive checks the server initiates, ServerActive is where the agent pushes active checks. Both must point at the Zabbix server IP or data never arrives.”

Q527: How do you configure system performance monitoring?

What it is: sysstat collects system performance history and sar reports it, offering CPU, memory, I/O and network views through sar -u, -r, -b and -n DEV.

Why it is used: To get historical CPU, memory, I/O and network numbers for past incidents instead of only live values from top.

# Use SAR for historical data
ENABLED="true"

# Enable collection
systemctl enable sysstat
systemctl start sysstat

# View reports
sar -u 1 5  # CPU usage
sar -r 1 5  # Memory
sar -b 1 5  # I/O
sar -n DEV 1 5  # Network

Interview talking point: “sar reads archived data rather than just current load, so enabling sysstat lets you answer questions about metrics from hours or days ago after the fact.”

Q528: How do you implement application performance monitoring?

What it is: Application performance monitoring uses an agent such as elastic-apm, installed with pip and created with a service name, to report traces from inside the running application.

Why it is used: To see request latency, errors and traces in application code rather than only host level CPU and memory.

# Use APM (Application Performance Monitoring)
# Options: New Relic, Datadog, Dynatrace, Elastic APM

# Example: Elastic APM
# Install agent
pip install elastic-apm

# Configure
# apm.py
from elasticapm import Client
client = Client(service_name='myapp')

Interview talking point: “APM agents instrument the app itself, giving per request traces and timings that infrastructure numbers like CPU percent can never reveal.”

Q529: How do you use performance monitoring tools?

What it is: Performance tools cover top or htop for overall load, pidstat for one process, iostat and iotop for disk I/O, nethogs and iftop for network, and perf for profiling.

Why it is used: To locate which resource and which process is the bottleneck before changing anything.

# System performance
top
htop
atop

# Process monitoring
pidstat -p 1234 1
prstat -p 1234

# I/O monitoring
iostat -xz 1
iotop

# Network monitoring
nethogs
iftop
sar -n DEV 1

# Full system analysis
perf record -g ./myapp
perf report

Interview talking point: “iostat -xz shows utilization and latency per device while iotop attributes I/O to processes, so a busy disk is traced to the writer. perf record -g plus perf report builds a call-graph profile of the binary.”

Q530: How do you use performance monitoring tools?

What it is: Performance monitoring uses top, htop and atop for overview, pidstat, mpstat, vmstat, iostat and sar for subsystems, iotop, nethogs, iftop for consumers, and perf for profiling.

Why it is used: To find which resource is saturated and which process is responsible during an incident.

# System monitoring
top
htop
atop

# Process monitoring
pidstat -p <pid> 1
ps aux --sort=-%cpu | head

# CPU monitoring
mpstat -P ALL 1
sar -u 1

# Memory monitoring
vmstat 1
sar -r 1

# I/O monitoring
iostat -xz 1
iotop

# Network monitoring
nethogs
iftop
sar -n DEV 1

# Full analysis
perf record -g ./myapp
perf report

# System resource usage
ss -s
netstat -s

Interview talking point: “iostat -xz shows per-device utilization and await, while nethogs and iftop attribute traffic per process and per connection, unlike sar -n DEV which only gives totals per interface.”

Metrics and Alerting

Q531: How do you configure alerting?

What it is: Prometheus alerts are defined as rules in a PrometheusRule object, with an expression, a duration, a severity label, and a summary annotation.

Why it is used: To get notified when metrics cross a threshold, such as memory above 80 percent for five minutes.

# Prometheus Alert
apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: myapp-alerts
spec:
  groups:
  - name: myapp
    rules:
    - alert: HighMemory
      expr: (container_memory_working_set_bytes / container_spec_memory_limit_bytes) > 0.8
      for: 5m
      labels:
        severity: warning
      annotations:
        summary: "High memory usage"

Interview talking point: “The for clause of 5m means the condition must hold for five minutes before the alert fires, which suppresses alerts caused by brief spikes.”

Q532: How do you configure alerting?

What it is: Prometheus alerting rules declare named expressions with a pending period, severity labels, and summary annotations, evaluated on a 30 second interval in the linux_alerts group.

Why it is used: To turn metric thresholds into alerts that fire when CPU, memory, or disk conditions persist.

# Prometheus alerting rules
groups:
  - name: linux_alerts
    interval: 30s
    rules:
      - alert: HighCPUUsage
        expr: 100 - (avg by (instance) (irate(node_cpu_seconds_total{mode="idle"}[5m])) * 100) > 80
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "High CPU usage on {{ $labels.instance }}"

      - alert: HighMemoryUsage
        expr: (node_memory_MemTotal_bytes - node_memory_MemAvailable_bytes) / node_memory_MemTotal_bytes * 100 > 90
        for: 5m
        labels:
          severity: critical

      - alert: DiskSpaceLow
        expr: (node_filesystem_avail_bytes{mountpoint="/"} / node_filesystem_size_bytes{mountpoint="/"}) < 0.1
        for: 10m
        labels:
          severity: warning

Interview talking point: “The for clause is what separates a firing alert from a flapping one: HighCPUUsage and HighMemoryUsage wait 5 minutes while DiskSpaceLow waits 10, so brief spikes never page anyone.”

Q533: How do you configure custom metrics collection?

What it is: node_exporter serves host metrics on port 9100, while the shell loop pushes a custom timestamped metric into port 9090 with netcat every 10 seconds.

Why it is used: To expose machine metrics to Prometheus and push application specific counters that stock exporters do not collect.

# Use node_exporter
docker run -d -p 9100:9100 prom/node-exporter

# Custom metrics
#!/bin/bash
while true; do
  echo "custom_metric $(date +%s)" | nc -q1 localhost 9090
  sleep 10
done

Interview talking point: “node_exporter waits to be scraped on 9100, while the loop pushes a value straight into 9090, a useful contrast between exporting for collection and pushing to a receiving endpoint.”

Q534: How do you configure custom metrics?

What it is: Two ways to expose custom metrics to Prometheus: a script pushing metric lines over netcat to node_exporter port 9100, and a textfile collector script that writes a .prom file on a schedule.

Why it is used: To publish application metrics like request counts and latency that node_exporter does not collect by itself.

# Using Prometheus node_exporter
# Custom collector script
#!/bin/bash
while true; do
  echo "custom_app_requests_total $(date +%s)"
  echo "custom_app_active_connections $(netstat -an | grep :8080 | wc -l)"
  sleep 10
done | nc -q1 localhost 9100

# Using textfile collector
cat > /usr/local/bin/app_metrics.sh << 'EOF'
#!/bin/bash
METRICS_DIR="/var/lib/node_exporter/textfile_collector"

# Application metrics
APP_REQUESTS=$(curl -s localhost:8080/metrics | grep requests | awk '{print $2}')
APP_LATENCY=$(curl -s localhost:8080/metrics | grep latency | awk '{print $2}')

cat > ${METRICS_DIR}/app.prom << EOM
app_requests_total ${APP_REQUESTS}
app_latency_seconds ${APP_LATENCY}
EOM
EOF

chmod +x /usr/local/bin/app_metrics.sh
crontab -e
# * * * * * /usr/local/bin/app_metrics.sh

Interview talking point: “The textfile collector writes app.prom into the textfile_collector directory on a cron schedule, while the first script pushes via netcat; textfile is the common pattern because node_exporter reads those files at scrape time.”

Q535: How do you configure network metrics?

What it is: Network metrics come from tools like Cilium’s Prometheus metrics, Weave Scope for topology, and kube-state-metrics for Kubernetes object state.

Why it is used: To feed dashboards and alerts about cluster networking and resource state.

# Use Prometheus with Cilium
cilium metrics enable

# Or use Weave Scope
kubectl apply -f https://cloud.weave.works/launch/k8s/weavescope.yaml

# Or use kube-state-metrics
kubectl apply -f https://github.com/kubernetes/kube-state-metrics/archive/master/examples/standard/deployment.yaml

Interview talking point: “kube-state-metrics reports object state rather than live traffic, while Cilium metrics expose dataplane counters, so the right source depends on whether the question is about objects or packets.”

Q536: How do you configure Thanos?

What it is: Thanos sidecar runs beside Prometheus with a TSDB path plus gRPC and HTTP addresses, and Prometheus supplies external labels for cluster and replica.

Why it is used: To extend Prometheus retention and query across replicas.

# Sidecar
# In Prometheus
external_labels:
  cluster: 'cluster1'
  replica: 'a'
thanos:
  sidecar:
    tsdb: /prometheus
    grpc-address: :10901
    http-address: :10902

Interview talking point: “external_labels are required because Thanos deduplicates across replicas, so without distinct cluster and replica values a query spanning two Prometheus instances returns duplicate series.”

Q537: How do you implement alerting?

What it is: Prometheus alerts route through AlertManager, which groups by alertname and sends to email, Slack, or PagerDuty receivers, guided by practices such as alerting on symptoms and keeping runbooks.

Why it is used: To reach the right person quickly without drowning the team in noise.

# Alerting
# Prometheus + AlertManager
# alertmanager.yaml
route:
  group_by: ['alertname']
  receiver: 'team'
  group_wait: 10s
  group_interval: 10s

receivers:
- name: 'team'
  email_configs:
  - to: 'team@example.com'
  slack_configs:
  - api_url: 'https://hooks.slack.com/...'
    channel: '#alerts'

# PagerDuty integration
- name: 'pagerduty'
  pagerduty_configs:
  - service_key: 'KEY'

# Best practices
# 1. Alert on symptoms, not causes
# 2. Set appropriate thresholds
# 3. Avoid alert fatigue
# 4. Have runbooks
# 5. Test alerts regularly

Interview talking point: “group_wait and group_interval batch related notifications into one message, which directly fights alert fatigue. Alerting on symptoms such as user-visible error rate rather than causes such as CPU ties the alert to impact.”

Q538: How do you implement DevOps metrics?

What it is: DevOps metrics implementation involves tracking DORA metrics, including deployment frequency, lead time, mean time to recovery (MTTR), and change failure rate.

Why it is used: It measures software delivery and operational performance objectively.

# DORA metrics
# Track

# Deployment frequency
# Measure

# Lead time
# Monitor

# MTTR
# Calculate

# Change failure rate
# Analyze

Interview talking point: “Tracking all four DORA metrics together gives a balanced view, as optimizing one alone can mask other issues.”

Q539: How do you use Alertmanager?

What it is: Alertmanager runs on 9093 and its YAML config routes alerts grouped by alertname to a receiver, here an email receiver with SMTP host, sender, and credentials.

Why it is used: To route firing alerts to the right people and collapse duplicates.

# Install
docker run -d -p 9093:9093 prom/alertmanager

# Configure
# /etc/alertmanager/alertmanager.yml
route:
  group_by: ['alertname']
  receiver: 'email'
receivers:
- name: 'email'
  email_configs:
  - to: 'admin@example.com'
    from: 'alertmanager@example.com'
    smarthost: 'smtp.example.com:587'
    auth_username: 'user'
    auth_password: 'pass'

Interview talking point: “Alertmanager routes rather than detects; group_by alertname batches related firing alerts into one email, so a flapping instance does not flood the inbox with repeats.”

Q540: How do you use blackbox exporter?

What it is: Blackbox exporter probes targets with named modules such as an HTTP check with a timeout, and Prometheus calls its /probe path using relabel configs to set the target.

Why it is used: To check whether websites and endpoints actually respond, not just whether hosts are up.

# Install
docker run -d -p 9115:9110 prom/blackbox-exporter

# Configure
# /etc/blackbox/blackbox.yml
modules:
  http_2xx:
    prober: http
    timeout: 5s

# Prometheus
- job_name: blackbox
  metrics_path: /probe
  params:
    module: [http_2xx]
  static_configs:
  - targets:
    - https://example.com
  relabel_configs:
  - source_labels: [__address__]
    target_label: __param_target
  - source_labels: [__param_target]
    target_label: instance

Interview talking point: “Blackbox exporter turns active probing into metrics, and the relabel configs are essential because they move the real URL into the probe parameter and back into the instance label.”

Q541: How do you use Cortex?

What it is: Cortex runs as a container on ports 9009 and 9091, started with a command that points at its YAML config file.

Why it is used: To run a horizontally scalable Prometheus-compatible metrics backend instead of one instance.

cortex:
  image: cortexproject/cortex:latest
  ports:
    - "9009:9009"
    - "9091:9091"
  command:
    - -config.file=/etc/cortex/cortex.yaml

Interview talking point: “Unlike a single Prometheus driven by flags, Cortex is a service whose entire behavior is set through the config file path passed in the command at startup.”

Q542: How do you use Cortex?

What it is: Cortex is a scalable Prometheus-compatible metrics system configured through Helm values, declaring a dated schema config with tsdb store, v11 schema, and S3 object storage.

Why it is used: To hold long-term metrics for many tenants in object storage instead of local disks.

# Helm values
cortex:
  config:
    schema:
      configs:
      - from: 2022-01-01
        store: tsdb
        object_store: s3
        schema: v11
    storage:
      s3:
        bucket: cortex-data

Interview talking point: “The schema config is versioned by date, with from 2022-01-01 selecting store tsdb, schema v11 and object_store s3, so a new schema version starts from a date while older data stays readable.”

Q543: How do you use Mimir?

What it is: Mimir runs on 9009 and 9091 with a config whose schema block maps a date to a store, object store, schema version, and index period.

Why it is used: To keep long-term metrics with versioned storage schemas on object storage.

# Install
docker run -d -p 9009:9009 -p 9091:9091 grafana/mimir

# Configure
# /etc/mimir/config.yaml
schema:
  configs:
  - from: 2023-01-01
    store: tsdb
    object_store: files
    schema: v11
    index:
      prefix: index_
      period: 168h

Interview talking point: “The from date in the schema block means a new schema version takes effect from that point while older blocks remain readable, so schema upgrades do not rewrite history.”

Q544: How do you use node exporter?

What it is: node exporter runs on 9100 and exposes collectors for CPU, meminfo, diskstats, and netdev, plus a textfile collector that reads .prom files from a directory you choose.

Why it is used: To expose CPU, memory, disk, and network metrics for Prometheus to scrape.

# Install
docker run -d -p 9100:9100 prom/node-exporter

# Collectors
# --collector.cpu
# --collector.meminfo
# --collector.diskstats
# --collector.netdev

# Textfile collector
# --collector.textfile.directory=/var/lib/node_exporter/textfile_collector

Interview talking point: “The textfile collector lets cron jobs drop metric files into a directory so one-off script metrics join the normal scrape instead of needing a separate exporter each time.”

Q545: How do you use Pushgateway?

What it is: Pushgateway runs on 9091 and accepts metrics piped to it with curl using a job path, and a DELETE to the same path removes them again.

Why it is used: To get metrics from batch or short-lived jobs into Prometheus.

# Install
docker run -d -p 9091:9091 prom/pushgateway

# Push metrics
echo "metric_name 3.14" | curl --data-binary @- http://pushgateway:9091/metrics/job/some_job

# Delete
curl -X DELETE http://pushgateway:9091/metrics/job/some_job

Interview talking point: “Pushgateway suits jobs that die before a scrape, but you must DELETE the job endpoint afterwards or the stale value lingers and keeps looking like a live reading.”

Q546: How do you use Thanos?

What it is: Thanos adds a sidecar container beside Prometheus that exposes the local TSDB over gRPC and HTTP, pointed at the Prometheus URL and data path shown.

Why it is used: To reach Prometheus data from long-term storage tooling without replacing the Prometheus instance.

# Sidecar
- args:
  - sidecar
  - --tsdb.path=/prometheus
  - --prometheus.url=http://localhost:9090
  - --grpc-address=:10901
  - --http-address=:10902
  image: thanosio/thanos:v0.30.0
  name: thanos-sidecar

Interview talking point: “The sidecar runs alongside Prometheus rather than replacing it, serving the same TSDB on separate gRPC and HTTP addresses, which is how a long-term backend reaches Prometheus data without changing scrape config.”

Q547: How do you use VictoriaMetrics?

What it is: VictoriaMetrics can run as a single container with a data volume, or as the vmcluster image for clustered deployment, both on port 8428.

Why it is used: To store and query metrics at scale starting from a simple single binary.

# Install
docker run -d -p 8428:8428 victoriametrics/victorialogs-datasource

# Single
docker run -d -p 8428:8428 -v /victoria-data:/victoria-data victoriametrics/victoria-metrics

# Cluster
docker run -d -p 8480:8480 victoriametrics/vmcluster

Interview talking point: “The single and cluster images share the same query API, so you can grow from one box to a cluster without rewriting the dashboards or queries that hit it.”

Observability Concepts

Q548: How do you implement observability platform?

What it is: Observability platform implementation involves using Prometheus for metrics, Loki or ELK for logs, Jaeger for traces, Grafana for dashboards, and PagerDuty for alerting.

Why it is used: It provides unified visibility into system health and performance.

# Metrics
# Prometheus

# Logs
# Loki/ELK

# Traces
# Jaeger

# Dashboards
# Grafana

# Alerting
# PagerDuty

Interview talking point: “Correlating metrics, logs, and traces in one platform speeds root cause analysis during incidents.”

Q549: How do you implement observability?

What it is: OpenTelemetry provides a collector that receives traces, metrics, and logs from instrumented applications, while Jaeger stores and displays distributed traces.

Why it is used: To follow a request across services and pinpoint latency or failures.

# Use OpenTelemetry
# Install collector
kubectl apply -f https://github.com/open-telemetry/opentelemetry-collector-releases/releases/download/v0.50.0/otelcol.yaml

# Instrument application
# Add OpenTelemetry SDK to application

# Use Jaeger for tracing
kubectl apply -f https://github.com/jaegertracing/jaeger-operator/releases/latest/download/jaeger.yaml

Interview talking point: “Traces require the OpenTelemetry SDK inside the application, unlike infrastructure-only metrics, so end-to-end visibility exists only once the application code emits spans.”

Q550: How do you implement observability?

What it is: Observability has three pillars: Jaeger traces one request across services via OpenTelemetry spans, Prometheus exposes counters at a /metrics endpoint, and structured logging records context fields as JSON.

Why it is used: To know what is slow, how much traffic there is, and what happened during an incident.

# Distributed tracing with Jaeger
# Client integration
# Python
from opentelemetry import trace
from opentelemetry.exporter.jaeger.thrift import JaegerExporter
from opentelemetry.sdk.trace import TracerProvider

trace.set_tracer_provider(TracerProvider())
jaeger_exporter = JaegerExporter(
    agent_host_name="jaeger",
    agent_port=6831,
)
trace.get_tracer_provider().add_span_processor(
    BatchSpanProcessor(jaeger_exporter)
)

tracer = trace.get_tracer(__name__)

with tracer.start_as_current_span("operation") as span:
    span.set_attribute("key", "value")
    # Do work

# Metrics with Prometheus
from prometheus_client import Counter, generate_latest

requests_total = Counter('requests_total', 'Total requests')

@app.route('/')
def hello():
    requests_total.inc()
    return 'Hello'

# Export metrics
@app.route('/metrics')
def metrics():
    return generate_latest()

# Logging structured
import logging
import json

logger = logging.getLogger(__name__)
logger.info("Request processed", extra={
    "user_id": user.id,
    "duration_ms": duration
})

Interview talking point: “Traces follow a single request end to end, metrics like requests_total aggregate across all traffic, and logs give detail for one event. Prometheus only sees a service after that /metrics route is exposed.”

Q551: How do you implement observability?

What it is: Observability combines Prometheus metrics, structured logs in ELK or Loki, distributed traces, correlation through trace and request IDs, SLO-based alerts, dashboards, and OpenTelemetry instrumentation.

Why it is used: To answer why a system misbehaves without shipping new debug code.

# Observability
# 1. Metrics
# Prometheus
# Custom metrics

# 2. Logs
# Structured logging
# ELK/Loki

# 3. Traces
# Distributed tracing

# 4. Correlation
# Trace IDs
# Request IDs

# 5. Alerting
# Based on SLOs

# 6. Dashboards
# Service overview
# Troubleshooting

# 7. Post-mortems
# Blameless analysis

# Implementation
# OpenTelemetry
# Many tools

Interview talking point: “Correlation is the glue, a trace ID carried across logs and metrics lets you jump from an alert straight to the slow request path instead of guessing from dashboards.”

Q552: How do you implement observability?

What it is: Observability here means Prometheus for metrics, ELK for logs, Jaeger for traces, correlation through trace IDs, and alerting tied to SLOs.

Why it is used: To detect and diagnose production issues from three complementary signal types.

# Observability
# 1. Metrics
# Prometheus

# 2. Logs
# ELK

# 3. Traces
# Jaeger

# 4. Correlation
# Trace IDs

# 5. Alerting
# SLO-based

Interview talking point: “Alerts keyed to SLOs page on real user impact rather than raw threshold breaches, which keeps alert volume low and ties every page to error budget consumption.”

Q553: How do you implement observability?

What it is: Observability combines metrics, structured logs, and distributed traces, correlated by trace and request IDs, with SLO-based alerts and purposeful dashboards.

Why it is used: To answer why something broke, not just that it broke.

# Metrics
# Prometheus
# Custom metrics

# Logs
# Structured logging
# ELK/Loki

# Traces
# Distributed tracing

# Correlation
# Trace IDs
# Request IDs

# Alerting
# Based on SLOs

# Dashboards
# Service overview
# Troubleshooting

# Post-mortems
# Blameless analysis

Interview talking point: “Correlation is the value: a trace ID ties a metric spike to the exact request path in the logs, which is what makes incident debugging fast.”

Q554: How do you implement observability?

What it is: Observability provides visibility into system behavior through metrics, logs, and traces.

Why it is used: It helps detect, diagnose, and resolve issues in complex distributed environments.

# Metrics
# Prometheus

# Logs
# ELK

# Traces
# Jaeger

# Correlation
# Trace IDs

# Alerting
# SLO-based

Interview talking point: “Correlating trace IDs across metrics and logs allows teams to pinpoint root causes faster.”

Q555: How do you implement service mesh observability?

What it is: Istio telemetry enables tracing sampled at 10 percent to a Jaeger collector, and kubectl port-forward exposes Jaeger, Kiali and Grafana dashboards on local ports.

Why it is used: Follow request paths, latencies and failures across service boundaries.

# Istio telemetry
# Enable tracing
istioctl install --set values.telemetry.enabled=true

# Configure tracing
# istio configmap
apiVersion: v1
kind: ConfigMap
metadata:
  name: istio
data:
  meshConfig: |
    enableTracing: true
    defaultConfig:
      tracing:
        sampling: 10
        zipkin:
          address: jaeger-collector.observability:9411

# Access dashboards
# Jaeger
kubectl port-forward -n istio-system svc/jaeger-query 16686:16686

# Kiali
kubectl port-forward -n istio-system svc/kiali 20001:20001

# Grafana
kubectl port-forward -n istio-system svc/grafana 3000:3000

Interview talking point: “Port-forward maps a local port to the in-cluster service so dashboards need no public exposure, and a sampling rate of 10 means only a fraction of requests show up in traces.”

Q556: How do you use OpenTelemetry?

What it is: The OpenTelemetry Collector is configured by a ConfigMap: it receives OTLP over gRPC and HTTP and exports traces to a Tempo endpoint plus a logging exporter in one pipeline.

Why it is used: To collect telemetry once and route it to backends without changing application instrumentation.

# Collector
apiVersion: v1
kind: ConfigMap
metadata:
  name: otel-collector-config
data:
  otel-collector-config: |
    receivers:
      otlp:
        protocols:
          grpc:
          http:
    exporters:
      otlp:
        endpoint: "tempo:4317"
      logging:
        loglevel: debug
    service:
      pipelines:
        traces:
          receivers: [otlp]
          exporters: [otlp, logging]

Interview talking point: “The traces pipeline lists receiver otlp and exporters otlp plus logging, so the same spans go to Tempo at tempo:4317 and to debug logs, which is a handy way to confirm data flow when a backend shows nothing.”

Level 13: Performance and Resource Management

CPU Utilization

Q557: How do you monitor CPU usage?

What it is: top, htop, and glances give live CPU views, ps sorted by %cpu ranks processes, pidstat samples per-process CPU, and mpstat or sar report system-wide and per-processor CPU.

Why it is used: To decide whether load comes from one runaway process or from the machine as a whole.

# Real-time
top
htop
# or
glances

# Per-process
ps aux --sort=-%cpu | head
# or
pidstat 1

# System-wide
mpstat -P ALL 1
sar -u 1

Interview talking point: “mpstat -P ALL 1 breaks usage down per core, which exposes a single saturated core that an overall system average would completely hide.”

Q558: How do you use mpstat?

What it is: mpstat from sysstat reports processor statistics. It shows all CPUs by default, mpstat 2 5 repeats on an interval, -P ALL breaks the numbers out per processor, and -P 0 watches a single CPU.

Why it is used: To spot one saturated core or uneven CPU load across processors during a performance investigation.

# Install
apt install sysstat

# All CPUs
mpstat

# Interval
mpstat 2 5

# Per processor
mpstat -P ALL 2 3

# Specific CPU
mpstat -P 0 1 5

# Fields
mpstat -A

Interview talking point: “mpstat is the per-CPU view that sar -P ALL also gives, and -P ALL is what reveals a single busy core that an overall average would hide.”

Q559: How do you use mpstat?

What it is: mpstat reports processor statistics, summarizing all CPUs or breaking them out per CPU with -P ALL at an interval and count. -A selects all available fields.

Why it is used: To spot load or interrupt imbalance concentrated on a single CPU core.

# All CPUs
mpstat

# Interval
mpstat 2 5

# Per processor
mpstat -P ALL 1 3

# Specific CPU
mpstat -P 0 1 5

# Fields
mpstat -A

Interview talking point: “mpstat -P ALL 1 3 shows per-core user and sys percentages, which is how you catch a single hot core from a process that is not spreading its threads.”

Q560: How do you use pidstat?

What it is: pidstat from sysstat reports statistics per process. pidstat -p PID 1 tracks one process, -r shows memory, -d shows I/O, pidstat 1 lists all processes, and -hl gives interval output line by line.

Why it is used: To attribute CPU, memory or disk usage to a specific PID instead of looking at whole-system numbers.

# Install
apt install sysstat

# CPU per process
pidstat -p PID 1

# Memory
pidstat -r PID 1

# I/O
pidstat -d PID 1

# All processes
pidstat 1

# User-specific
pidstat -u -p ALL 1

# Interval with lines
pidstat -hl 1

Interview talking point: “The other sysstat tools answer what is busy, pidstat answers who is busy: -r for memory per process, -d for I/O per process, and -p narrows the report to one PID.”

Memory and Swap Monitoring

Q561: How do you monitor memory usage?

What it is: free -h and /proc/meminfo show totals and free memory, ps sorted by %mem ranks consumers, pmap -X maps one process’s memory, and vmstat or sar sample trends over time.

Why it is used: To detect high memory consumption and early swapping.

# Check memory
free -h
# or
cat /proc/meminfo

# Per-process
ps aux --sort=-%mem | head
# or
pmap -X PID

# Trends
vmstat 1
sar -r 1

Interview talking point: “free gives a snapshot while vmstat 1 and sar -r 1 show the trend, and pmap -X on a PID explains exactly where one process’s memory is allocated.”

Q562: How do you view system memory usage?

What it is: free -h reports total, used, and free memory in human units, /proc/meminfo holds the raw counters, and vmstat 1 prints memory and system figures every second.

Why it is used: To spot memory pressure, heavy cache use, or swapping on a host.

# Check memory
free -h
# or
cat /proc/meminfo
# or
vmstat 1

Interview talking point: “free gives a single snapshot while vmstat 1 streams samples each second, so use free for a quick reading and vmstat to see whether the pressure is steady or spiking.”

Q563: How do you use earlyoom?

What it is: earlyoom is a daemon that kills processes when free memory drops below a threshold, before the kernel OOM killer takes over.

Why it is used: To keep a desktop or SSH session responsive when memory runs out.

# Install
apt install earlyoom

# Configure
# /etc/default/earlyoom
EARLYOOM_ARGS="-m 10 -r 60"

# Start
systemctl enable earlyoom
systemctl start earlyoom

Interview talking point: “EARLYOOM_ARGS of -m 10 -r 60 triggers at ten percent free memory, so earlyoom decides first and avoids the system-wide stall the kernel path can cause.”

Q564: How do you use vmstat?

What it is: vmstat reports virtual memory statistics at intervals. It can show slab memory with -m, disk statistics with -d, a cumulative summary with -s, and extended fields with -e.

Why it is used: To get a fast overview of CPU, memory, I/O, and swap activity on a loaded machine.

# Basic
vmstat 1

# Memory
vmstat -m

# Disk
vmstat -d

# Summary
vmstat -s

# Extended
vmstat -e

Interview talking point: “vmstat 1 prints a line per second, but the first line summarizes activity since boot rather than that one-second interval. The -s switch swaps the usual columns for cumulative counters instead.”

Q565: How do you use vmstat?

What it is: vmstat reports virtual memory statistics at a chosen interval, with modes for kernel memory tables, disk activity, a summary of counters, and active versus inactive memory.

Why it is used: To get a one-screen view of CPU, memory, swap, and I/O trends on a Linux host.

# Basic
vmstat 1

# Memory details
vmstat -m

# Disk
vmstat -d

# Summary
vmstat -s

# Active/inactive memory
vmstat -a

Interview talking point: “Running vmstat 1 gives repeating rows where the bo, wa, and id columns quickly show whether pressure is coming from disk writes, I/O wait, or idle CPU.”

Q566: How do you tune memory usage?

What it is: Memory tuning via sysctl controls how eagerly the kernel swaps and reclaims caches. The record sets vm.swappiness, vfs_cache_pressure, dirty page ratios, and reserves huge pages, with /proc/meminfo showing the result.

Why it is used: To keep latency-sensitive services off swap and tune how fast dirty pages hit the disk.

# Swappiness
vm.swappiness = 10
vm.vfs_cache_pressure = 50
vm.dirty_ratio = 60
vm.dirty_background_ratio = 5

# Huge pages
vm.nr_hugepages = 128

# Apply
sysctl -p

# Check memory
cat /proc/meminfo

Interview talking point: “A low vm.swappiness like 10 keeps processes in RAM rather than swapping, and vm.dirty_ratio versus vm.dirty_background_ratio split when the kernel starts writing dirty pages back.”

Disk I/O

Q567: How do you monitor I/O?

What it is: iostat -x reports per-device I/O statistics, iotop shows which processes generate I/O, vmstat exposes the wa column for I/O wait, and lsblk and blktrace examine block devices.

Why it is used: To diagnose slow storage and find which device or process is causing the wait.

# I/O stats
iostat -x 1

# Per-process I/O
iotop

# Check I/O wait
vmstat 1
# Look at 'wa' column

# Block devices
lsblk
blktrace

Interview talking point: “a high wa in vmstat means the CPU is idle waiting on I/O, and iostat -x 1 then pinpoints which device is saturated while iotop names the process responsible.”

Q568: How do you test disk I/O?

What it is: Disk I/O testing with fio driving direct asynchronous reads and writes at set block sizes and depths, plus hdparm and dd quick checks.

Why it is used: To benchmark storage throughput and latency under realistic workload shapes.

# fio installation
apt install fio

# Sequential read
fio --name=seqread --readonly --direct=1 --ioengine=libaio \
    --bs=4k --iodepth=32 --numjobs=4 --rw=readread \
    --filename=/dev/sda1 --size=1G

# Random write
fio --name=randwrite --direct=1 --ioengine=libaio \
    --bs=4k --iodepth=32 --numjobs=4 --rw=randwrite \
    --filename=/tmp/test --size=1G

# Mixed workload
fio --name=mixed --direct=1 --ioengine=libaio \
    --bs=4k --iodepth=1 --numjobs=1 --rw=randrw \
    --rwmixread=70 --filename=/tmp/test

# Using hdparm
hdparm -t /dev/sda
hdparm -T /dev/sda

# Using dd
# Write speed
dd if=/dev/zero of=/tmp/test bs=1M count=1024 oflag=direct

# Read speed
dd if=/tmp/test of=/dev/null bs=1M count=1024 iflag=direct

Interview talking point: “direct=1 bypasses the page cache so numbers reflect the device, and hdparm -T measures cache reads while -t measures the device itself, which is why both are reported.”

Q569: How do you use iostat?

What it is: iostat reports CPU and per-device I/O statistics from sysstat. iostat 2 5 repeats every two seconds five times, -x gives the detailed per-device view, -t adds timestamps, and -d or -c selects devices or CPU.

Why it is used: To measure per-device I/O load and see whether a disk is saturated.

# Install
apt install sysstat

# Basic
iostat

# Interval
iostat 2 5

# Per device
iostat -x

# Detailed
iostat -x -t

# Specific device
iostat -d sda

# Report CPU and device utilization
iostat -c -d sda

Interview talking point: “iostat -x is the mode to reach for because it breaks utilization down per device, and combining -d with a device name like sda plus -c shows device and CPU figures in one run.”

Q570: How do you use iotop?

What it is: iotop is a top-like display of per-process disk I/O. -o shows only processes actually doing I/O, -b -n 3 runs batch output, -P limits to processes, and -t shows threads.

Why it is used: To identify which process causes disk pressure when overall I/O wait is high.

# Basic
iotop

# Only I/O
iotop -o

# Batch mode
iotop -b -n 3

# Only processes
iotop -P

# By thread
iotop -t

Interview talking point: “Plain iotop fills the screen with idle processes, so -o isolates the real offenders. Choosing -P versus -t sets granularity, and -b -n 3 is the form to capture in scripts or log files.”

Q571: How do you analyze I/O issues?

What it is: iostat -x reports per-device utilization and latency, iotop shows which processes issue I/O, blktrace and blkparse trace block requests, and ioping probes device latency.

Why it is used: To find which disk is saturated and which process is driving the load.

# iostat
iostat -x 1
iostat -x -d sda 1

# iotop
iotop
iotop -o  # only active
iotop -P  # processes only

# blktrace
blktrace -d /dev/sda -o /tmp/blk
blkparse -i /tmp/blk

# Latency
ioping -c 10 /dev/sda

Interview talking point: “iostat -x gives await and utilization for the device as a whole, while iotop -o lists only processes actually doing I/O, which keeps output short on a busy host.”

Q572: How do you analyze I/O performance?

What it is: I/O analysis uses iostat for device throughput and latency, iotop and pidstat for per-process readers, and blktrace with blkparse for request-level tracing on a disk.

Why it is used: To find the device or process causing latency and to tune the scheduler.

# I/O statistics
iostat -xz 1
sar -d 1

# Per-process I/O
iotop
pidstat -d 1

# Block device info
lsblk
blkid

# I/O scheduler
cat /sys/block/sda/queue/scheduler
echo deadline > /sys/block/sda/queue/scheduler

# Queue depth
cat /sys/block/sda/queue/nr_requests

# Check for I/O waits
vmstat 1

# File system performance
# Read-ahead
cat /sys/block/sda/queue/read_ahead_kb

# Trace I/O
blktrace -d /dev/sda -o trace
blkparse -i trace

Interview talking point: “cat on the queue scheduler file lists available schedulers and writing a name to that file switches them live, so tuning happens without downtime.”

Network Utilization

Q573: How do you monitor network?

What it is: netstat -s and ss -s summarize protocol statistics, netstat -ant and ss -ant list TCP connections, and nethogs, iftop, and bmon display bandwidth usage.

Why it is used: To watch connections and find which host or process is consuming bandwidth.

# Network stats
netstat -s
ss -s

# Per-connection
netstat -ant
ss -ant

# Bandwidth
nethogs
iftop
bmon

Interview talking point: “ss is the modern replacement for netstat with the same -ant style flags, and nethogs groups bandwidth by process while iftop groups it by remote host.”

Q574: How do you use iftop?

What it is: iftop is a live per-connection bandwidth display; -i picks the interface, -n skips DNS lookups, and -B reports bytes instead of bits.

Why it is used: Shows which remote hosts are consuming bandwidth right now.

# Install
apt install iftop

# Usage
iftop
iftop -i eth0
iftop -i eth0 -n  # no DNS
iftop -i eth0 -B  # bytes

Interview talking point: “iftop groups traffic by connection pairs, and -n matters because reverse DNS can stall the display, while -B switches units to bytes so the numbers line up with other byte-based tools.”

Q575: How do you use iperf3?

What it is: iperf3 runs a server with -s and a client with -c; -P 4 opens parallel streams and -u with -b 1G switches to a UDP test at a set rate.

Why it is used: To measure achievable throughput between two hosts and find bottlenecks.

# Server
iperf3 -s

# Client
iperf3 -c server-ip

# Test bandwidth
iperf3 -c server-ip -P 4  # parallel

# UDP test
iperf3 -c server-ip -u -b 1G

Interview talking point: “The default run measures TCP, so you add -u with an explicit bandwidth to test UDP, and -P 4 with parallel streams usually reports higher realistic throughput.”

Q576: How do you use nethogs?

What it is: nethogs attributes bandwidth to the process owning each socket; running it with an interface shows per-program usage, -d sets the refresh interval, and -p runs promiscuous.

Why it is used: Finds which application, not merely which connection, is saturating the network.

# Install
apt install nethogs

# Usage
nethogs
nethogs eth0
nethogs -d 2  # refresh
nethogs -p  # promiscuous

Interview talking point: “The contrast with iftop is the grouping: iftop shows who you are talking to while nethogs shows which process is doing the talking, which is what you need when one daemon looks guilty.”

Q577: How do you use vnStat?

What it is: vnStat is a daemon configured for an interface that accumulates traffic totals, viewed with the plain command, -h for hourly, -l for live, and -m for monthly.

Why it is used: To track long-term interface traffic totals without storing packets.

# Install
apt install vnstat

# Configure
# /etc/vnstat.conf
Interface "eth0"

# Start
systemctl start vnstat

# View
vnstat
vnstat -h
vnstat -l
vnstat -m

Interview talking point: “vnStat is a historical database of interface totals rather than a packet sniffer, and the view flags matter: -l shows live rates while -m and -h show monthly and hourly history.”

Q578: How do you use vnstat?

What it is: vnstat is a long-term traffic logger that records interface totals in a database; -u initializes a database for eth0, and the views show hourly, daily, monthly, and live statistics.

Why it is used: Keeps historical bandwidth records for capacity planning and reporting.

# Install
apt install vnstat

# Configure
vnstat -u -i eth0

# View
vnstat
vnstat -h
vnstat -d
vnstat -m
vnstat -l

Interview talking point: “Unlike iftop or bmon, vnstat does not sniff packets, it polls the kernel interface counters, so the daemon keeps accumulating history across reboots and months of totals stay queryable on demand.”

Q579: How do you optimize network latency?

What it is: Network latency optimization uses a geographic CDN, multi-layer caching, gz or brotli compression, HTTP/2 multiplexing, and Anycast DNS.

Why it is used: To reduce round-trip time for users far from the origin.

# Network latency optimization
# 1. CDN
# Geographic

# 2. Caching
# Multi-layer

# 3. Compression
# gz/brotli

# 4. HTTP/2
# Multiplexing

# 5. DNS
# Anycast

Interview talking point: “HTTP/2 multiplexing removes connection-level queueing, so many requests share one connection instead of waiting behind each other like they would on separate older connections.”

Q580: How do you optimize network latency?

What it is: Network latency optimization uses geographic CDNs, multi-layer caching, compression (gz/brotli), HTTP/2 multiplexing, and DNS anycast.

Why it is used: To deliver faster responses to users across geographic locations.

# CDN
# Geographic

# Caching
# Multi-layer

# Compression
# gz/brotli

# HTTP/2
# Multiplexing

# DNS
# Anycast

Interview talking point: “Caching and compression reduce payloads, while anycast and HTTP/2 improve routing and connection efficiency for lower latency.”

Q581: How do you optimize network throughput?

What it is: NIC throughput tuning with ethtool: enabling TSO, GSO, GRO, rx, and tx offloads, enlarging ring buffers with -G, setting interrupt coalescing with -C, enabling flow control with -A, and steering receive queues through the rps_cpus sysfs file.

Why it is used: To get more bits per second and fewer interrupts from high-speed interfaces.

# Network card offloading
ethtool -K eth0 tso on
ethtool -K eth0 gso on
ethtool -K eth0 gro on
ethtool -K eth0 rx on
ethtool -K eth0 tx on

# Ring buffer
ethtool -G eth0 rx 4096 tx 4096

# Interrupt coalescing
ethtool -C eth0 rx-usecs 100 tx-usecs 100

# Flow control
ethtool -A eth0 rx on tx on

# Bonding for throughput
# See earlier section on network bonding

# Multi-queue
# Check
cat /sys/class/net/eth0/queues/rx-0/rps_cpus
# Set
echo ffffff > /sys/class/net/eth0/queues/rx-0/rps_cpus

Interview talking point: “Offloads like TSO and GRO let the NIC handle large segments, reducing CPU work, and writing a hex mask to rps_cpus spreads softirq processing across cores.”

Q582: How do you tune network for latency?

What it is: Latency tuning sets tcp_low_latency to 1, grows the netdev backlog, and enlarges tcp_rmem and tcp_wmem socket buffers.

Why it is used: To cut queuing delay and keep fast links fed under load.

# Kernel parameters
net.ipv4.tcp_low_latency = 1
net.core.netdev_max_backlog = 1000

# Increase socket buffer
net.ipv4.tcp_rmem = 4096 87380 16777216
net.ipv4.tcp_wmem = 4096 65536 16777216

Interview talking point: “tcp_rmem and tcp_wmem carry default, and maximum values, so raising the third number lets a high-bandwidth connection grow its window instead of being capped.”

System Activity Reporting

Q583: How do you use atop?

What it is: atop is an interactive system monitor that also logs to disk. atop shows CPU, network and disk views, atop -w records samples to a file, atop -r replays them, and atopsar prints daily summaries.

Why it is used: To review a past incident by replaying recorded samples instead of guessing what happened while nobody watched.

# Install
apt install atop

# Interactive
atop

# Interval
atop 1

# Specific process
atop -p PID

# Network
atop -n

# Disk
atop -d

# Write to file
atop -w /tmp/atop.log 30 120
# Read
atop -r /tmp/atop.log

# Daily summaries
atopsar -c
atopsar -m

Interview talking point: “atop -w /tmp/atop.log 30 120 captures a sample every 30 seconds for 120 intervals, so the box keeps evidence for later, and atop -r reads that log back for review.”

Q584: How do you use atop?

What it is: atop is a system monitor that records resource usage to a log and replays it. atop -w writes samples at an interval for a set count, atop -r reads them back, and -n shows network.

Why it is used: To reconstruct what happened historically instead of only watching live.

# Install
apt install atop

# Run
atop

# Interval
atop 1

# Specific process
atop -p PID

# Network
atop -n

# Write/read log
atop -w /tmp/atop.log 30 120
atop -r /tmp/atop.log

Interview talking point: “atop -w /tmp/atop.log 30 120 logs 120 samples at 30-second intervals, and atop -r replays that file afterwards, so you can review a past incident window. That recording and replay is the main difference between atop and live-only tools like htop.”

Q585: How do you use sar - advanced?

What it is: sar collects and reports system activity. Sample switches include -u for CPU, -r for memory, -S for swap, -b for I/O, and -n DEV for network, each taking an interval and count, while -A reports everything.

Why it is used: To measure performance over time and to review historic resource usage recorded on the host.

# CPU
sar -u 1 5

# Memory
sar -r 1 5

# Swap
sar -S 1 5

# I/O
sar -b 1 5

# Network
sar -n DEV 1 5

# All
sar -A

Interview talking point: “Each letter selects a subsystem, so sar -u 1 5 gives five one-second CPU samples while sar -n DEV 1 5 covers interfaces. sar -A is the convenient catch-all when you do not yet know which metric matters.”

Q586: How do you use sar?

What it is: sar is part of sysstat and reports system activity. sar -u samples CPU, -r memory, -b I/O, and -n DEV network devices, each over an interval and count; -A dumps a full report.

Why it is used: To see current or historical performance trends for CPU, memory, I/O, and network.

# Enable
systemctl enable sysstat
systemctl start sysstat

# CPU
sar -u 1 5

# Memory
sar -r 1 5

# I/O
sar -b 1 5

# Network
sar -n DEV 1 5

# Generate report
sar -A > /tmp/sar_report.txt

Interview talking point: “Enabling the sysstat service is what makes sar keep historical data, so without it sar only reports live samples. Each flag targets one resource class, and sar -A aggregates everything into a single report file for sharing.”

Tracing Tools (strace, lsof)

Q587: How do you use ltrace?

What it is: ltrace traces library function calls of a process. It can attach to a PID, count calls with -c, restrict to one library with -l, time calls with -T, and follow forks with -f.

Why it is used: To see which shared library functions a binary calls when debugging linkage or surprising behavior.

# Basic
ltrace -p PID
ltrace -c command

# Library calls
ltrace -l library.so command

# Time
ltrace -T command

# Follow forks
ltrace -f command

Interview talking point: “strace shows kernel system calls while ltrace shows user-space library calls, so the choice depends on where the fault sits. -f follows child processes and -T reports per-call timing for spotting slow library routines.”

Q588: How do you use ltrace?

What it is: ltrace traces library calls rather than system calls, attaching to a PID or wrapping a command. It can filter by library or pattern, show time with -T, and follow children.

Why it is used: To see which shared library functions a program calls and where it spends time there.

# Basic
ltrace -p PID
ltrace -c command

# Library calls
ltrace -l lib.so command
ltrace -L command

# Time spent
ltrace -T command

# Follow child processes
ltrace -f command

# Filter
ltrace -e "*malloc*+*free*" command

# Attach to process
ltrace -p PID

Interview talking point: “The core distinction is that strace records system calls while ltrace records library calls, so an application problem that never appears at the syscall layer may show up here.”

Q589: How do you use strace effectively?

What it is: strace traces system calls of a process, attached by PID or wrapping a command. Filters select call groups, -t and -T add timestamps and durations, and -f follows forks.

Why it is used: To see what a stuck or failing program is actually asking the kernel to do.

# Basic
strace -p PID
strace -c command

# Trace specific calls
strace -e trace=read,write -p PID
strace -e trace=network -p PID
strace -e trace=file -p PID

# Timestamps
strace -t -p PID
strace -tt -p PID
strace -T -p PID

# Follow forks
strace -f -p PID

# Output to file
strace -o output.txt -p PID

# Attach to running process
strace -p PID

Interview talking point: “strace -c gives a per-call summary of counts and times, which spots an odd syscall faster than reading a full trace, and -f is essential or child processes are missed.”

Q590: How do you use strace?

What it is: strace traces a process system calls, attaching to a PID or wrapping a command. It offers summary counts with -c, output to a file, timestamps with -t or -tt, and filtering with -e.

Why it is used: To discover why a program fails, hangs, or touches files by revealing the syscalls it issues.

# Basic
strace -p PID
strace -c command

# Output to file
strace -o output.txt command

# Timestamps
strace -t command
strace -tt command

# Filter
strace -e trace=network,file command
strace -e openat,write command

Interview talking point: “-c produces an aggregate call summary instead of a stream, and -e trace=network,file narrows output to those classes while -e openat,write names specific calls. -tt prints microsecond timestamps where single -t gives only seconds.”

Performance Profiling

Q591: How do you use perf tools?

What it is: perf profiles kernel and user code. perf record captures samples with call graphs for perf report, perf stat counts events like cycles and instructions, perf top streams hotspots, and perf sched shows scheduler latency.

Why it is used: To find where CPU time or cycles go in a program, or to check scheduler delay on a busy host.

# Install
apt install linux-tools-common linux-tools-generic

# CPU sampling
perf record -g ./myprogram
perf report

# Specific events
perf stat -e cycles,instructions ./myprogram

# Top
perf top

# Scheduler analysis
perf sched latency

Interview talking point: “perf record plus perf report answers where time is spent, while perf stat -e cycles,instructions gives counts and ratios with no profile at all. perf sched latency focuses on runqueue wait rather than on the application code itself.”

Q592: How do you use perf?

What it is: perf records CPU samples with call graphs from a program, a PID, or the whole system, then perf report displays them. perf stat counts hardware events and perf top live-samples.

Why it is used: To find which functions burn CPU time or where cache misses and context switches occur.

# CPU profiling
perf record -g ./program
perf record -p PID
perf record -a -g  # system-wide

# Report
perf report
perf report --stdio

# Top
perf top
perf top -p PID

# Specific events
perf stat -e cycles,instructions,cache-misses ./program
perf stat -e 'syscalls:sys_enter_*' -a

# Trace
perf record -e sched:sched_switch -a
perf script

Interview talking point: “perf stat gives quick counters such as cycles, instructions, and cache-misses across a whole run, while perf record plus perf report shows where the time went function by function.”

Q593: How do you profile applications?

What it is: Application profiling combines tools like gprof, valgrind, perf, and strace to analyze runtime behavior.

Why it is used: Identify CPU, memory, and syscall bottlenecks to optimize application performance.

# Using gprof
gcc -pg -g program.c -o program
./program
gprof program gmon.out > analysis.txt

# Using valgrind
valgrind --tool=callgrind ./program
# View with kcachegrind
kcachegrind callgrind.out.*

# Using perf
perf record -g ./program
perf report
perf annotate

# Using strace
strace -c ./program
strace -T -tt -e trace=write ./program

# Using flamegraph
perf record -F 99 -g ./program
perf script | stackcollapse-perf.pl > out.folded
flamegraph.pl out.folded > flamegraph.svg

Interview talking point: “Flamegraphs visualize aggregated call stacks over time, making hot paths easy to identify.”

Q594: How do you use performance profiling tools?

What it is: perf samples stacks for a process or system-wide and reports hot code paths, which can be rendered as flame graphs. Valgrind, gprof, and strace cover cache, call counts, and syscalls.

Why it is used: To find where CPU time actually goes before optimizing anything.

# perf
perf record -g -p <pid>
perf report
perf top

# Flame graph
# Install
git clone https://github.com/brendangregg/FlameGraph.git

# Generate
perf record -F 99 -g -p <pid>
perf script | ./stackcollapse-perf.pl | ./flamegraph.pl > flame.svg

# System-wide profiling
perf record -a -g -- sleep 10
perf report

# Valgrind
valgrind --tool=cachegrind ./program
cg_annotate cachegrind.out.*

# gprof
gcc -pg -g program.c -o program
./program
gprof program gmon.out > analysis.txt

# strace
strace -c -p <pid>
strace -T -tt -p <pid>

Interview talking point: “strace -c gives a summary of count and time per syscall, so you can tell whether time goes into the kernel or into userspace compute.”

Bottleneck Analysis

Q595: How do you analyze memory issues?

What it is: Memory analysis combines free -h and /proc/meminfo for totals, pmap -X for one process, slabtop for kernel slab caches, and dmesg for out-of-memory events.

Why it is used: To find what consumes RAM, detect leaks, and confirm whether the OOM killer ran.

# Check memory usage
free -h
cat /proc/meminfo

# Per-process memory
ps aux --sort=-%mem | head
pmap -X PID

# Check for memory leaks
valgrind --leak-check=full ./program

# Slab info
slabtop
cat /proc/slabinfo

# Huge pages
cat /proc/meminfo | grep -i huge

# OOM killer
dmesg | grep -i "out of memory"
cat /var/log/messages | grep -i oom

Interview talking point: “Plenty of free output alongside a heavy slab cache points at kernel memory rather than user processes, and grepping dmesg for out of memory reveals which process the OOM killer chose.”

Q596: How do you analyze CPU performance?

What it is: CPU analysis combines lscpu for hardware layout, mpstat and sar for usage over time, and top or pidstat for per-process consumption, plus affinity and frequency tools.

Why it is used: To find saturated cores, the process burning CPU, and whether virtualization steal time is high.

# CPU info
lscpu
cat /proc/cpuinfo

# CPU usage over time
mpstat -P ALL 1
sar -u 1

# Per-CPU usage
mpstat -P ALL 1

# Process CPU usage
top
ps aux --sort=-%cpu
pidstat -p <pid> 1

# CPU steal (virtualization)
vmstat 1

# Scheduler
# View process priority
ps -eo pid,ni,pri,comm

# CPU affinity
taskset -c 0-3 program
taskset -p 0xF <pid>

# Check CPU frequency
cpupower frequency-info
cpupower frequency-set -g performance

Interview talking point: “mpstat -P ALL breaks usage down per core to spot a single busy CPU, and vmstat reveals steal time indicating the hypervisor is starving the guest.”

Q597: How do you analyze memory performance?

What it is: Memory analysis uses free and /proc/meminfo for totals, vmstat and sar for trends, pmap and ps per process, plus slabtop, huge page flags, and pressure stats.

Why it is used: To tell leak, cache, and swap problems apart and to confirm an OOM kill.

# Memory info
free -h
cat /proc/meminfo

# Memory usage over time
vmstat 1
sar -B 1

# Per-process memory
ps aux --sort=-%mem
pmap -x <pid>
cat /proc/<pid>/status | grep -i vm

# Memory allocation issues
# Check for OOM
dmesg | grep -i "out of memory"
cat /var/log/syslog | grep -i oom

# Slab info
slabtop

# Huge pages
cat /proc/meminfo | grep -i huge

# Transparent huge pages
cat /sys/kernel/mm/transparent_hugepage/enabled

# Memory pressure
cat /proc/pressure/memory

Interview talking point: “/proc/pressure/memory reports stall time from memory pressure, which separates a genuinely RAM-starved host from one that is simply using a lot of page cache.”

Q598: How do you analyze network performance?

What it is: Network performance tools report socket and connection statistics with ss and netstat, interface counters with ip -s link, bandwidth with iftop and nethogs, and captures with tcpdump.

Why it is used: To locate retransmits, connection buildup, or the host eating bandwidth during an incident.

# Network statistics
netstat -s
ss -s

# Per-interface statistics
ip -s link
netstat -i

# Connection states
ss -tan state established
ss -tan state time-wait

# Bandwidth monitoring
iftop
nethogs

# Packet capture
tcpdump -i eth0
tcpdump -i eth0 -w capture.pcap

# Network latency
ping -c 4 host
traceroute host
mtr host

# TCP analysis
# TCP retransmits
netstat -s | grep -i retrans

# Connection tracking
conntrack -L

# Socket statistics
ss -tulpn
lsof -i

Interview talking point: “ss -tan state time-wait counts sockets sitting in TIME_WAIT, and netstat -s filtered for retrans points to packet loss rather than application slowness.”

Q599: How do you find performance bottlenecks?

What it is: sar -A samples all collected metrics each second, uptime or /proc/loadavg show load averages, strace traces a running PID’s system calls, ltrace traces library calls, and perf top profiles hot code.

Why it is used: To narrow a slowdown down to CPU, memory, I/O, or a specific blocking call.

# System resource summary
sar -A 1

# Load average
uptime
# or
cat /proc/loadavg

# Process analysis
strace -p PID  # system calls
ltrace -p PID  # library calls
perf top

Interview talking point: “a load average above the core count signals contention, strace -p PID shows in real time what a stuck process is waiting on, and perf top points at the hottest functions.”

Q600: How do you handle performance degradation?

What it is: Bottleneck hunting: top and ps aux sorted by CPU for compute, free -h and vmstat for memory, iostat -xz for disk, iftop and nethogs for network, then fix and re-measure.

Why it is used: To identify which resource is the constraint before spending on fixes.

# Performance troubleshooting
# 1. Identify symptoms
# Check metrics
# top
# iostat 1

# 2. Locate bottleneck
# CPU bound?
top
ps aux --sort=-%cpu

# Memory bound?
free -h
vmstat 1

# I/O bound?
iostat -xz 1

# Network bound?
iftop
nethogs

# 3. Fix
# CPU: Scale, optimize code
# Memory: Add RAM, fix leaks
# I/O: Use faster storage
# Network: Optimize queries

# 4. Verify
# Monitor metrics
# Compare before/after

Interview talking point: “The order matters: measure first, apply the matching fix, then compare before and after. iostat -xz gives extended per-device stats, which is what distinguishes a saturated disk from a slow CPU.”

Q601: How do you optimize database connections?

What it is: A checklist for tuning database connections covering pooling, pool sizing, timeouts, monitoring active connections, and database configuration tuning.

Why it is used: Stops the app from exhausting database connection limits and adds headroom under load.

# Database connection optimization
# 1. Pooling
# Connection pool

# 2. Sizing
# Pool size

# 3. Timeouts
# Configure

# 4. Monitoring
# Active connections

# 5. Tuning
# Database config

Interview talking point: “Connection pools trade memory for latency, so sizing them is a balancing act: too few leaves requests waiting, too many overwhelms the database. The outline pairs sizing with timeouts and monitoring, because a pool without timeout and active-connection visibility just shifts the failure elsewhere.”

Q602: How do you optimize database connections?

What it is: Database connection optimization uses connection pooling, proper pool sizing, configured timeouts, active connection monitoring, and database configuration tuning.

Why it is used: To reduce connection overhead and prevent resource exhaustion.

# Pooling
# Connection pool

# Sizing
# Pool size

# Timeouts
# Configure

# Monitoring
# Active connections

# Tuning
# Database config

Interview talking point: “Pool sizing and timeouts prevent bottlenecks and leaks, while monitoring allows tuning under changing workload conditions.”

Q603: How do you optimize database indexing?

What it is: Database indexing optimization: find slow queries, analyze them with EXPLAIN, add appropriate indexes, order composite indexes, and rebuild periodically.

Why it is used: Speeds up queries by giving the database a path to rows instead of a scan.

# Database indexing
# 1. Identify
# Slow queries

# 2. Analyze
# EXPLAIN

# 3. Create
# Appropriate

# 4. Composite
# Order

# 5. Maintenance
# Rebuild

Interview talking point: “EXPLAIN is the decision tool here: it shows whether an index will actually be used before you create one. Composite index order matters because a leading-column mismatch makes the index unusable for that query.”

Q604: How do you optimize database indexing?

What it is: Database indexing optimization involves identifying slow queries, analyzing with EXPLAIN, creating appropriate indexes, ordering composite indexes, and performing maintenance rebuilds.

Why it is used: It improves query performance and reduces load on database systems.

# Identify
# Slow queries

# Analyze
# EXPLAIN

# Create
# Appropriate

# Composite
# Order

# Maintenance
# Rebuild

Interview talking point: “The order of columns in a composite index matters greatly, and even good indexes require periodic maintenance to stay efficient.”

Q605: How do you optimize database performance?

What it is: Database performance optimization involves profiling to identify bottlenecks, optimizing indexing, configuring caching, rewriting queries, and planning for scaling.

Why it is used: It keeps databases responsive as workloads grow.

# Profiling
# Identify bottleneck

# Indexing
# Optimize

# Caching
# Configure

# Query
# Rewrite

# Scaling
# Plan

Interview talking point: “Profiling first prevents premature scaling, as query rewrites and indexing often resolve bottlenecks with less cost.”

Q606: How do you optimize database queries?

What it is: Query optimization analyzes plans with EXPLAIN, adds strategic indexes, rewrites queries in equivalent forms, uses a query cache, and profiles slow queries.

Why it is used: To make database queries return faster under production load.

# Query optimization
# 1. EXPLAIN
# Analyze

# 2. Indexing
# Strategic

# 3. Rewriting
# Equivalent

# 4. Caching
# Query cache

# 5. Profiling
# Slow queries

Interview talking point: “EXPLAIN comes first because an index added without checking the plan may never be chosen by the optimizer and just sits unused.”

Q607: How do you optimize database queries?

What it is: Database query optimization uses EXPLAIN/ANALYZE to understand plans, strategic indexing, equivalent query rewriting, query caching, and profiling slow queries.

Why it is used: To reduce response time, lower load, and improve application performance.

# EXPLAIN
# Analyze

# Indexing
# Strategic

# Rewriting
# Equivalent

# Caching
# Query cache

# Profiling
# Slow queries

Interview talking point: “Indexing decisions depend on query plans; rewriting and caching help, but profiling slow queries identifies where to focus effort.”

Q608: How do you optimize databases?

What it is: Database optimization adds proper indexes, uses EXPLAIN to review query plans, caches results, pools connections, scales with read replicas or sharding, and tunes configuration parameters.

Why it is used: To make queries faster and let the database handle more concurrent load.

# Database optimization
# 1. Indexing
# Proper indexes

# 2. Query optimization
# EXPLAIN

# 3. Caching
# Use cache

# 4. Connection pooling
# Pool

# 5. Scaling
# Read replicas
# Sharding

# 6. Configuration
# Tune parameters

Interview talking point: “Connection pooling matters because each connection costs memory, so a shared pool beats every application instance opening its own unbounded set of connections.”

Q609: How do you optimize databases?

What it is: Database optimization improves query performance and resource usage through tuning and structure changes.

Why it is used: It reduces latency and prevents databases from becoming bottlenecks under load.

# Indexing
# Proper indexes

# Query optimization
# EXPLAIN

# Caching
# Use cache

# Connection pooling
# Pool

# Scaling
# Read replicas
# Sharding

# Configuration
# Tune parameters

Interview talking point: “Using EXPLAIN to analyze queries alongside proper indexing often yields bigger gains than hardware upgrades.”

Q610: How do you optimize disk I/O?

What it is: Disk I/O optimization: the scheduler is read and set through the sysfs queue scheduler file, made permanent with a udev rule, read-ahead and rotational flags adjusted, and noatime and nodiratime added to fstab mount options.

Why it is used: To raise storage throughput and reduce latency under load.

# I/O Scheduler
# Check current
cat /sys/block/sda/queue/scheduler

# Set deadline scheduler
echo deadline > /sys/block/sda/queue/scheduler
echo cfq > /sys/block/sda/queue/scheduler

# Make permanent
# /etc/udev/rules.d/60-ioschedulers.rules
ACTION=="add|change", KERNEL=="sda", SUBSYSTEM=="block", ATTR{queue/scheduler}="deadline"

# Block device settings
echo 4096 > /sys/block/sda/queue/read_ahead_kb
echo 0 > /sys/block/sda/queue/rotational
echo 2 > /sys/block/sda/queue/rq_affinity

# Filesystem options
# /etc/fstab
/dev/sda1 / ext4 noatime,nodiratime,errors=remount-ro 0 1

Interview talking point: “Writing to the sysfs scheduler file changes it until reboot, and a udev rule matching add or change is what makes that choice survive a restart.”

Q611: How do you optimize Linux for databases?

What it is: Database tuning sets sysctl values like vm.swappiness, vm.dirty_ratio, and shared memory limits, plus nofile entries in limits.conf.

Why it is used: To reduce write stalls and give databases enough memory and file descriptors.

# Memory
vm.swappiness = 10
vm.dirty_ratio = 15
vm.dirty_background_ratio = 5

# Network
net.core.rmem_max = 134217728
net.core.wmem_max = 134217728

# File descriptors
fs.file-max = 65535

# Kernel
kernel.shmmax = 68719476736
kernel.shmall = 4294967296

# Limits.conf
* soft nofile 65535
* hard nofile 65535

Interview talking point: “vm.dirty_ratio caps how much dirty data may accumulate before writeback forces itself, keeping bursts from freezing the database, while limits.conf raises open files above the default.”

Q612: How do you optimize Linux for databases?

What it is: Linux database tuning selects XFS or ext4, chooses the deadline or noop I/O scheduler, enables huge pages, tunes network buffer sizes, and uses SSD or NVMe disks.

Why it is used: To reduce database latency and raise throughput on the host itself.

# Linux database optimization
# 1. Filesystem
# XFS/ext4

# 2. I/O scheduler
# Deadline/noop

# 3. Memory
# Huge pages

# 4. Network
# Buffer sizes

# 5. Disk
# SSD/NVMe

Interview talking point: “The noop scheduler suits fast SSD and NVMe where reordering gains little, while the deadline scheduler helps spinning disks meet read latency targets.”

Q613: How do you optimize Linux for databases?

What it is: Optimizing Linux for databases involves choosing filesystems (XFS/ext4), tuning I/O scheduler (deadline/noop), configuring huge pages, adjusting network buffers, and using SSD/NVMe.

Why it is used: To maximize database throughput and reduce I/O latency.

# Filesystem
# XFS/ext4

# I/O scheduler
# Deadline/noop

# Memory
# Huge pages

# Network
# Buffer sizes

# Disk
# SSD/NVMe

Interview talking point: “I/O scheduler and storage choice strongly affect performance, while huge pages and tuned buffers improve memory and network efficiency.”

Q614: How do you optimize Linux performance?

What it is: Linux performance tuning covers scheduler and CPU affinity, swappiness and huge pages, I/O schedulers and mount options, TCP buffer sizes, and profiling with perf, sysbench, fio, and iperf.

Why it is used: To remove bottlenecks and extract more throughput from existing hardware.

# Linux optimization
# 1. CPU
# Tune scheduler
# Process affinity
# Priority adjustment

# 2. Memory
# Swappiness
# Cache tuning
# Huge pages

# 3. I/O
# I/O scheduler
# Filesystem choice
# Mount options
# SSD optimization

# 4. Network
# Buffer sizes
# TCP tuning
# Offloading

# 5. Kernel
# Update regularly
# Tune parameters

# 6. Applications
# Profiling
# Optimization

# Tools
# perf
# sysbench
# fio
# iperf

Interview talking point: “Each tool targets one subsystem, perf for CPU profiling, fio for disk I/O, iperf for network throughput, so measure the suspect layer before touching any kernel parameter.”

Q615: How do you optimize Linux performance?

What it is: Linux performance tuning spans CPU scheduling and affinity, memory swappiness and huge pages, I/O scheduler and mount options, TCP tuning, and kernel updates.

Why it is used: To remove bottlenecks so the hardware delivers the throughput it promises.

# 1. CPU
# Tune scheduler
# Process priority
# CPU affinity

# 2. Memory
# Swappiness
# Huge pages
# Cache tuning

# 3. I/O
# I/O scheduler
# Filesystem choice
# Mount options

# 4. Network
# Buffer sizes
# TCP tuning
# Offloading

# 5. Kernel
# Update regularly
# Tune parameters

Interview talking point: “Each layer has a lever worth naming: swappiness under memory pressure, the I/O scheduler for disks, and TCP buffer sizes for network, and you tune them only after measuring.”

Q616: How do you optimize Linux storage?

What it is: Linux storage tuning covering filesystem choice, mount options, LVM flexibility, RAID configuration, and I/O monitoring.

Why it is used: Improves storage performance, capacity management, and resilience on Linux hosts.

# Linux storage optimization
# 1. Filesystem
# Choice

# 2. Mount options
# Tuning

# 3. LVM
# Flexible

# 4. RAID
# Configuration

# 5. Monitoring
# I/O

Interview talking point: “The layers stack: RAID and LVM sit under the filesystem, and mount options tune how that filesystem uses them. Watching I/O tells you whether the bottleneck sits in the disk layer or above it.”

Q617: How do you optimize Linux storage?

What it is: Linux storage optimization involves choosing filesystems, tuning mount options, using LVM for flexibility, configuring RAID, and monitoring I/O.

Why it is used: To improve performance, resilience, and manageability of storage systems.

# Filesystem
# Choice

# Mount options
# Tuning

# LVM
# Flexible

# RAID
# Configuration

# Monitoring
# I/O

Interview talking point: “RAID and LVM provide protection and flexibility, while mount option tuning and I/O monitoring align storage behavior to workload needs.”

Q618: How do you optimize memory management?

What it is: Kernel memory sysctl parameters control virtual memory behavior such as swapping, writeback, and huge pages.

Why it is used: Improve performance and stability under varying memory pressure.

# Virtual memory
vm.swappiness=10
vm.dirty_ratio=15
vm.dirty_background_ratio=5
vm.dirty_expire_centisecs=3000
vm.dirty_writeback_centisecs=500
vm.vfs_cache_pressure=50
vm.min_free_kbytes=65536

# Memory overcommit
vm.overcommit_memory=0
vm.overcommit_ratio=50
vm.oom_dump_tasks=1
vm.oom_kill_allocating_task=1

# Huge pages
vm.nr_hugepages=1024

# Transparent huge pages
echo never > /sys/kernel/mm/transparent_hugepage/enabled
echo never > /sys/kernel/mm/transparent_hugepage/defrag

Interview talking point: “Lower swappiness favors keeping pages in RAM, while dirty ratios determine when background writeback flushes modified data.”

Q619: How do you optimize MySQL?

What it is: MySQL tuning in my.cnf sets innodb_buffer_pool_size to 4G, log file size, max_connections, table_open_cache, and turns on the slow query log with long_query_time 2.

Why it is used: Reducing latency under load and finding expensive queries.

[mysqld]
innodb_buffer_pool_size = 4G
innodb_log_file_size = 1G
max_connections = 500
query_cache_size = 0
table_open_cache = 4000
slow_query_log = 1
slow_query_log_file = /var/log/mysql/slow.log
long_query_time = 2

# Key parameters
# innodb_buffer_pool - 70-80% of RAM
# max_connections - based on expected concurrent connections
# query_cache - deprecated in MySQL 8.0

Interview talking point: “The notes point to the buffer pool at 70 to 80 percent of RAM as the main InnoDB lever, flag query_cache_size as deprecated in MySQL 8.0, and use slow_query_log to catch offenders.”

Q620: How do you optimize MySQL?

What it is: my.cnf [mysqld] tuning sets buffer pool size, log file size, connection limits, and table cache, disables the query cache, and enables the slow query log with a two-second threshold.

Why it is used: To reduce latency and disk reads under load, and to find slow queries worth indexing.

[mysqld]
innodb_buffer_pool_size = 4G
innodb_log_file_size = 1G
max_connections = 500
table_open_cache = 4000
query_cache_size = 0
slow_query_log = 1
slow_query_log_file = /var/log/mysql/slow.log
long_query_time = 2
max_connect_errors = 100000
wait_timeout = 600

Interview talking point: “slow_query_log = 1 with long_query_time = 2 records any query taking over two seconds, which is where tuning effort should start. query_cache_size = 0 turns the query cache off, and max_connect_errors relaxes the host blocked counter.”

Q621: How do you optimize network performance?

What it is: Network optimization covers hardware offloading, TCP buffer tuning, compression via accept encoding, a CDN for static content, and HTTP keepalive.

Why it is used: To raise throughput and cut latency on web-facing services.

# Network optimization
# 1. Offloading
# Hardware

# 2. Buffer tuning
# TCP

# 3. Compression
# Accept encoding

# 4. CDN
# Static

# 5. Keepalive
# HTTP

Interview talking point: “Keepalive reuses existing connections while offloading moves work to NIC hardware, so both attack connection overhead from opposite ends of the stack.”

Q622: How do you optimize network performance?

What it is: Network performance optimization improves throughput and latency through tuning and architectural choices.

Why it is used: It reduces bottlenecks for applications with high network traffic.

# Offloading
# Hardware

# Buffer tuning
# TCP

# Compression
# Accept encoding

# CDN
# Static

# Keepalive
# HTTP

Interview talking point: “TCP buffer tuning, compression, keepalive connections, and CDN usage each target different sources of delay.”

Q623: How do you optimize PostgreSQL?

What it is: postgresql.conf sizing covers shared buffers, cache estimates, maintenance and work memory, WAL sizes, and planner cost settings such as random_page_cost and effective_io_concurrency.

Why it is used: To match PostgreSQL memory and I/O behavior to the server’s RAM and storage type.

shared_buffers = 4GB
effective_cache_size = 12GB
maintenance_work_mem = 1GB
checkpoint_completion_target = 0.9
wal_buffers = 64MB
default_statistics_target = 100
random_page_cost = 1.1
effective_io_concurrency = 200
work_mem = 26214kB
min_wal_size = 1GB
max_wal_size = 4GB

Interview talking point: “shared_buffers is the actual cache while effective_cache_size is only a planner estimate of how much OS cache is available, so the two serve different purposes. random_page_cost = 1.1 tells the planner storage behaves like SSD, changing chosen index plans.”

Q624: How do you optimize storage performance?

What it is: Storage performance optimization involves using SSDs, configuring RAID, choosing an appropriate filesystem, enabling caching, and monitoring I/O metrics.

Why it is used: It improves data throughput and reduces latency for applications.

# SSD
# Use

# RAID
# Configuration

# Filesystem
# Choice

# Caching
# Enable

# Monitoring
# I/O metrics

Interview talking point: “Choosing the right filesystem and RAID configuration has a larger impact than caching alone on sustained I/O performance.”

Q625: How do you optimize storage?

What it is: Storage optimization tiers data between hot and cold, applies compression and deduplication, uses lifecycle policies, monitors usage, and performs regular cleanup.

Why it is used: To cut storage cost while keeping needed data fast to reach.

# Storage optimization
# 1. Tiering
# Hot/cold

# 2. Compression
# Deduplication

# 3. Lifecycle
# Policies

# 4. Monitoring
# Usage

# 5. Cleanup
# Regular

Interview talking point: “Lifecycle policies automate movement between tiers, so cost keeps falling continuously instead of relying on someone remembering to archive old data by hand.”

Q626: How do you optimize storage?

What it is: Storage optimization balances performance, cost, and capacity through tiering and lifecycle policies.

Why it is used: It reduces storage costs while keeping frequently accessed data fast.

# Tiering
# Hot/cold

# Compression
# Deduplication

# Lifecycle
# Policies

# Monitoring
# Usage

# Cleanup
# Regular

Interview talking point: “Automating lifecycle policies to move data to colder tiers and cleaning up unused data prevents uncontrolled growth.”

Q627: How do you optimize web performance?

What it is: Web performance work pushes static assets to a CDN, compresses with gz or brotli, sets caching headers, minifies CSS and JS, and optimizes images.

Why it is used: To cut page load time for end users.

# Web performance
# 1. CDN
# Static assets

# 2. Compression
# gz/brotli

# 3. Caching
# Headers

# 4. Minification
# CSS/JS

# 5. Images
# Optimization

Interview talking point: “Caching headers decide whether the CDN and browser can reuse assets, so without them compression and minification still pay the network cost on every single visit.”

Q628: How do you optimize web performance?

What it is: Web performance optimization reduces page load times through caching, compression, and asset optimization.

Why it is used: It improves user experience and can reduce infrastructure costs.

# CDN
# Static assets

# Compression
# gz/brotli

# Caching
# Headers

# Minification
# CSS/JS

# Images
# Optimization

Interview talking point: “Using a CDN for static assets, compression, and image optimization together deliver significant load time improvements.”

Q629: How do you tune CPU performance?

What it is: CPU tuning sets the frequency scaling governor and huge page behavior. The record lists available governors, writes performance to each CPU scaling_governor, sets GOVERNOR in cpufrequtils, and disables transparent huge pages.

Why it is used: To pin clock speed at maximum for low jitter workloads, and to stop transparent huge pages surprising databases.

# CPU governor
# Check available governors
cat /sys/devices/system/cpu/cpu0/cpufreq/scaling_available_governors

# Set performance
echo performance > /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor

# Make persistent
# /etc/default/cpufrequtils
GOVERNOR="performance"

# Disable transparent huge pages
echo never > /sys/kernel/mm/transparent_hugepage/enabled

Interview talking point: “The performance governor holds the CPU at full frequency instead of scaling down, and echoing never into transparent_hugepage/enabled stops automatic huge page defragmentation that can stall latency-sensitive processes.”

Q630: How do you tune CPU performance?

What it is: CPU tuning covers reading topology with lscpu, setting the frequency governor to performance, binding processes with taskset, and adjusting nice values.

Why it is used: To keep latency-sensitive work on specific cores and stop it competing with everything else.

# View CPU info
lscpu
cat /proc/cpuinfo

# CPU frequency
cpupower frequency-info
cpupower frequency-set -g performance

# CPU affinity
taskset -c 0-3 myapp
taskset -p 0xF myapp

# Process priority
nice -n 10 myapp
renice 5 -p 1234

# View scheduler
cat /proc/sched_debug

Interview talking point: “taskset -c 0-3 pins a process to cores zero through three, while nice only changes scheduling weight across all cores, so affinity controls placement and priority controls share. cpupower frequency-set -g performance keeps the governor at full speed.”

Q631: How do you tune CPU performance?

What it is: CPU tuning covers lscpu for topology, cpupower governors, taskset affinity, nice priority, disabling turbo, and CFS scheduler sysctls.

Why it is used: To make latency-sensitive or throughput-bound workloads run predictably on busy hosts.

# View CPU info
lscpu
cat /proc/cpuinfo

# CPU frequency scaling
cpupower frequency-info
cpupower frequency-set -g performance

# Set CPU affinity
taskset -c 0-3 myapp

# Process priority
nice -n 10 myapp
renice 5 -p $(pgrep myapp)

# Disable turbo boost (if needed)
echo 1 > /sys/devices/system/cpu/intel_pstate/no_turbo

# CFS scheduler tuning
# /etc/sysctl.conf
kernel.sched_latency_ns = 10000000
kernel.sched_min_granularity_ns = 1000000
kernel.sched_wakeup_granularity_ns = 2000000

Interview talking point: “nice adjusts how much CPU a process competes for while taskset pins it to cores 0 through 3, and the performance governor holds frequency at maximum instead of scaling down.”

Q632: How do you tune disk I/O?

What it is: Disk tuning changes the per-device I/O scheduler and queue settings under /sys/block. The record reads the current scheduler, sets deadline, raises nr_requests and read_ahead_kb, and adds a udev rule so the setting persists.

Why it is used: To reduce latency or raise throughput on database and logging hosts by choosing how requests are ordered.

# Scheduler
# Check current
cat /sys/block/sda/queue/scheduler
# Set deadline
echo deadline > /sys/block/sda/queue/scheduler

# I/O scheduler options
echo 1024 > /sys/block/sda/queue/nr_requests
echo 256 > /sys/block/sda/queue/read_ahead_kb

# Make persistent (CentOS)
# /etc/udev/rules.d/60-scheduler.rules
ACTION=="add|change", KERNEL=="sda", ATTR{queue/scheduler}="deadline"

Interview talking point: “Writes to /sys/block/sda/queue/scheduler apply only until reboot, which is why the udev rule with ACTION and KERNEL matching is added to keep deadline selected on every boot.”

Q633: How do you tune I/O performance?

What it is: I/O tuning picks a block scheduler such as cfq or deadline, sets per-process I/O class with ionice, adjusts read-ahead, and mounts with noatime.

Why it is used: To cut latency or boost throughput on disks that are saturated.

# I/O scheduler
cat /sys/block/sda/queue/scheduler
echo cfq > /sys/block/sda/queue/scheduler
# or
echo deadline > /sys/block/sda/queue/scheduler

# I/O priority
ionice -c 2 -n 0 -p 1234

# Block device settings
echo 4096 > /sys/block/sda/queue/read_ahead_kb
echo 0 > /sys/block/sda/queue/rotational

# Filesystem options
mount -o noatime,nodiratime /dev/sda1 /mnt

Interview talking point: “The scheduler name written to queue/scheduler takes effect at once but resets on reboot, and noatime stops the kernel updating access times on every read, removing a write per file read.”

Q634: How do you tune I/O performance?

What it is: I/O tuning picks a scheduler like deadline in sysfs, persists it with a udev rule, sets ionice classes, read ahead, queue depth and noatime mount options.

Why it is used: To reduce disk latency and keep throughput high under heavy write load.

# Check I/O scheduler
cat /sys/block/sda/queue/scheduler

# Set scheduler
echo deadline > /sys/block/sda/queue/scheduler

# Make permanent
# /etc/udev/rules.d/60-ioschedulers.rules
ACTION=="add|change", KERNEL=="sda", SUBSYSTEM=="block", ATTR{queue/scheduler}="deadline"

# I/O priorities
ionice -c 2 -n 0 -p $(pgrep myapp)

# Read ahead
cat /sys/block/sda/queue/read_ahead_kb
echo 4096 > /sys/block/sda/queue/read_ahead_kb

# Queue depth
cat /sys/block/sda/queue/nr_requests
echo 1024 > /sys/block/sda/queue/nr_requests

# Filesystem options
# /etc/fstab
/dev/sda1 / ext4 noatime,nodiratime,errors=remount-ro 0 1

Interview talking point: “writing to /sys/block/sda/queue/scheduler lasts only until reboot, whereas the udev rule reapplies deadline each time the device is added, and noatime in fstab skips access-time writes.”

Q635: How do you tune memory performance?

What it is: Memory tuning reads free and meminfo, drops page cache, sets vm.swappiness to control how eagerly the kernel swaps, and reserves huge pages.

Why it is used: To reduce swap activity and give databases or VMs predictable large-page memory.

# View memory
free -h
cat /proc/meminfo

# Clear cache
sync && echo 3 > /proc/sys/vm/drop_caches

# Swappiness
cat /proc/sys/vm/swappiness
sysctl vm.swappiness=10
# Add to /etc/sysctl.conf
# vm.swappiness=10

# Huge pages
echo 1024 > /sys/kernel/mm/hugepages/hugepages-2048kB/nr_hugepages

Interview talking point: “vm.swappiness=10 keeps the kernel from reaching for swap quickly, and the sysctl value reverts on reboot unless added to /etc/sysctl.conf. Writing 3 to drop_caches clears page cache immediately, which is mainly useful before a measurement.”

Q636: How do you tune memory performance?

What it is: Memory tuning sets vm.swappiness and vfs_cache_pressure, drops caches, reserves hugepages, controls overcommit, and disables transparent huge pages, applied with sysctl -p.

Why it is used: To keep databases and latency-sensitive apps from swapping or stalling on reclaim.

# View memory
free -h
cat /proc/meminfo

# Swappiness
# /etc/sysctl.conf
vm.swappiness=10
vm.vfs_cache_pressure=50

# Drop caches
sync && echo 3 > /proc/sys/vm/drop_caches

# Huge pages
# /etc/sysctl.conf
vm.nr_hugepages=1024

# Memory overcommit
# /etc/sysctl.conf
vm.overcommit_memory=1
vm.overcommit_ratio=50

# Transparent huge pages
echo never > /sys/kernel/mm/transparent_hugepage/enabled
echo never > /sys/kernel/mm/transparent_hugepage/defrag

# Enable
sysctl -p

Interview talking point: “swappiness 10 favors keeping processes resident over swapping, and echo 3 to drop_caches clears pagecache, dentries and inodes, a diagnostic step that costs performance until the cache warms again.”

Q637: How do you tune network performance?

What it is: sysctl network settings adjust TCP buffers, connection tracking capacity, and the congestion controller. The record raises rmem and wmem limits, sets nf_conntrack_max, and enables BBR with the fq queue discipline, applied via sysctl -p.

Why it is used: To improve throughput and latency on hosts moving heavy traffic or serving many connections.

# Increase TCP buffer sizes
net.core.rmem_max = 16777216
net.core.wmem_max = 16777216
net.ipv4.tcp_rmem = 4096 87380 16777216
net.ipv4.tcp_wmem = 4096 65536 16777216

# Increase connection tracking
net.netfilter.nf_conntrack_max = 1048576

# Enable TCP BBR
net.core.default_qdisc = fq
net.ipv4.tcp_congestion_control = bbr

# Apply
sysctl -p

Interview talking point: “tcp_rmem and tcp_wmem list minimum, default and maximum buffer sizes in one line, while the fq qdisc must be set alongside tcp_congestion_control = bbr for BBR to take effect.”

Q638: How do you tune network performance?

What it is: Network tuning raises socket buffer limits with sysctl, enables TCP window scaling, chooses a congestion control, and sizes NIC ring buffers with ethtool.

Why it is used: To fix slow transfers and packet loss on high-bandwidth or long-distance links.

# Network buffer sizes
sysctl -w net.core.rmem_max=16777216
sysctl -w net.core.wmem_max=16777216

# TCP tuning
sysctl -w net.ipv4.tcp_window_scaling=1
sysctl -w net.ipv4.tcp_congestion_control=cubic

# Make persistent
# /etc/sysctl.conf
net.core.rmem_max=16777216
net.core.wmem_max=16777216

# Ring buffer (NIC)
ethtool -G eth0 rx 4096 tx 4096
ethtool -g eth0

Interview talking point: “sysctl settings are per-boot, so they belong in /etc/sysctl.conf to survive restarts, while ethtool -G resizes the NIC ring buffer and ethtool -g reads the current size back for comparison.”

Q639: How do you tune network performance?

What it is: Network tuning raises socket buffer sysctls, sets cubic congestion control, fastopen, shorter fin timeout and keepalive, and enables ethtool offloads and ring buffers.

Why it is used: To improve throughput and connection handling on high-traffic servers.

# Network buffer sizes
net.core.rmem_max=16777216
net.core.wmem_max=16777216
net.ipv4.tcp_rmem=4096 87380 16777216
net.ipv4.tcp_wmem=4096 65536 16777216

# TCP tuning
net.ipv4.tcp_congestion_control=cubic
net.ipv4.tcp_fastopen=3
net.ipv4.tcp_max_syn_backlog=8192
net.ipv4.tcp_fin_timeout=15
net.ipv4.tcp_keepalive_time=600
net.ipv4.tcp_keepalive_intvl=60

# Enable offloading
ethtool -K eth0 tso on
ethtool -K eth0 gso on
ethtool -K eth0 gro on

# Ring buffer
ethtool -G eth0 rx 4096 tx 4096

# Apply
sysctl -p

Interview talking point: “ethtool -K tso, gso and gro shifts segmentation to the NIC to cut CPU cost, and sysctl -p reloads sysctl.conf so the TCP buffer and timeout values take effect.”

Capacity Planning

Q640: How do you capacity planning?

What it is: Capacity planning samples CPU with sar, memory with free and vmstat, disk with df and iostat, and bandwidth with iftop and nethogs, then projects needs with growth formulas.

Why it is used: Forecast hardware or cloud spend before saturation turns into an outage.

# CPU capacity
# Current usage
sar -u 1
# Projected
# (current_avg * growth_factor * days) / capacity

# Memory capacity
free -h
vmstat 1

# Disk capacity
df -h
# Monitor trends
iostat -x 1

# Network capacity
iftop
nethogs

# Calculate requirements
# CPU: (peak_cpu / cores) * growth_factor
# Memory: (peak_mem * growth_factor) + headroom
# Disk: (current_disk * (1 + growth_rate)^years)
# Network: peak_bandwidth * redundancy

# Tools
# Prometheus + node_exporter
# Grafana dashboards
# Custom scripts

Interview talking point: “iftop attributes bandwidth per connection while nethogs attributes it per process, and the disk formula compounds growth so current usage scales by one plus growth rate raised to the years needed.”

Q641: How do you handle capacity planning?

What it is: Capacity planning measures current CPU, memory, disk, and network utilization, analyzes growth trends, forecasts future needs, and adds or right-sizes capacity proactively using Prometheus and Grafana.

Why it is used: To prevent saturation outages while avoiding waste from over-provisioning.

# Capacity planning
# 1. Current state
# Measure utilization

# 2. Trends
# Analyze growth

# 3. Forecasting
# Predict future needs

# 4. Planning
# Add capacity proactively

# 5. Optimization
# Right-size resources
# Use automation

# Metrics
# CPU
# Memory
# Disk
# Network
# Application-specific

# Tools
# Prometheus
# Grafana
# AWS Compute Optimizer
# Azure Advisor

Interview talking point: “The order matters, measure before forecasting and forecast before buying, and tools like AWS Compute Optimizer turn the same utilization data into right-sizing recommendations.”

Q642: How do you handle capacity planning?

What it is: Capacity planning measures current utilization with sar, projects growth trends, forecasts future needs, and adds or right-sizes resources ahead of demand.

Why it is used: To add capacity before saturation instead of during an incident.

# Current state
# Measure utilization
sar -u 1
sar -r 1
sar -d 1

# Trends
# Analyze growth rate

# Forecasting
# Predict future needs

# Planning
# Add capacity proactively

# Optimization
# Right-size resources
# Use automation

Interview talking point: “sar -u reports CPU, -r memory, and -d disk at the chosen interval, giving the historical baseline you forecast from rather than a single snapshot.”

Q643: How do you implement capacity planning?

What it is: Capacity planning: track CPU, memory, disk, and network trends, review growth monthly, add capacity at 70 percent utilization, and scale with Cluster Autoscaler plus HPA and VPA.

Why it is used: Prevents outages by adding resources before utilization becomes critical.

# Monitor trends
# CPU, Memory, Disk, Network

# Growth rate
# Monthly review

# Planning
# Add capacity at 70% utilization

# Auto-scaling
# Cloud native: Cluster Autoscaler
# HPA/VPA

Interview talking point: “The 70 percent trigger buys lead time because provisioning lags demand, and note HPA scales pods while Cluster Autoscaler adds nodes, so they solve different bottlenecks.”

Q644: How do you implement capacity planning?

What it is: Capacity planning samples usage with sar for CPU, memory, I/O, and network, applies growth formulas, then right-sizes using kubectl autoscale and a VerticalPodAutoscaler.

Why it is used: To allocate resources ahead of demand instead of reacting to outages.

# Metrics collection
# CPU
sar -u 1 60 > cpu_usage.csv
# Memory
sar -r 1 60 > memory_usage.csv
# I/O
sar -d 1 60 > io_usage.csv
# Network
sar -n DEV 1 60 > network_usage.csv

# Analysis
# Growth rate
# (current_value - past_value) / days_between

# Capacity planning formula
# CPU: (peak_usage * growth_factor * buffer) / cores
# Memory: (peak_usage * growth_factor * buffer) / available
# Disk: (current_usage * (1 + growth_rate)^years)
# Network: peak_bandwidth * redundancy_factor

# Tools
# Google SRE capacity planning
# Horizontal Pod Autoscaler metrics
kubectl autoscale deployment myapp --cpu-percent=80 --min=2 --max=10

# Vertical Pod Autoscaler
kubectl apply -f - <<EOF
apiVersion: autoscaling.k8s.io/v1
kind: VerticalPodAutoscaler
metadata:
  name: myapp-vpa
spec:
  targetRef:
    apiVersion: "apps/v1"
    kind: Deployment
    name: myapp
  updatePolicy:
    updateMode: "Auto"
EOF

Interview talking point: “The HPA scales replica count against a target such as 80 percent CPU, while the VPA changes each pod CPU and memory requests. You generally avoid running both against the same deployment.”

Level 14: Troubleshooting

General Diagnostic Approach

Q645: How do you debug application issues?

What it is: Application debugging captures core dumps with ulimit and gcore, analyzes them in gdb, checks leaks with valgrind, and profiles with perf, pdb, or jstack.

Why it is used: To find crashes, leaks, and thread stalls inside a failing program.

# Core dumps
# Enable
ulimit -c unlimited

# /etc/security/limits.conf
* soft core unlimited

# Generate core
gcore <pid>

# Analyze
gdb program core
(gdb) bt
(gdb) info threads

# Memory leaks
valgrind --leak-check=full program

# Performance profiling
perf record -g program
perf report

# Python debugging
python -m pdb program.py
python -m cProfile program.py

# Java debugging
jstack <pid>
jmap -heap <pid>
jmap -dump:format=b,file=heap.bin <pid>

# Node.js debugging
node --inspect program.js
chrome://inspect

Interview talking point: “ulimit -c unlimited raises the shell core size limit so a crashing process actually writes a core, which gdb then opens with bt for the stack.”

Q646: How do you debug disk issues?

What it is: Disk debugging spans usage and inode checks with df and du, large file searches, I/O stats, filesystem and SMART checks, and finding deleted files still held open.

Why it is used: To tell apart a full filesystem, a hung process, or failing hardware.

# Disk usage
df -h
df -i
du -sh /*

# Find large files
find / -type f -size +100M -exec ls -lh {} \; 2>/dev/null | sort -k5 -h

# I/O stats
iostat -xz 1
sar -d 1

# Check filesystem
fsck -n /dev/sda1
xfs_repair -n /dev/sda1

# Check SMART
smartctl -a /dev/sda
smartctl -H /dev/sda

# Mount options
mount | grep sda1
cat /proc/mounts

# Lsof for deleted files
lsof +L1
ls -l /proc/*/fd/* | grep deleted

Interview talking point: “df -i catches the case where space is free but inodes are exhausted, and lsof +L1 reveals deleted files that keep consuming space until the process holding them exits.”

Q647: How do you debug disk issues?

What it is: Disk debugging combines df -h and df -i for space and inodes, du and find for big consumers, iostat for I/O, fsck and smartctl for health, and lvs, pvs, vgs for LVM.

Why it is used: To find why a filesystem filled, an inode ran out, or a disk is slow or failing.

# Disk usage
df -h
df -i
du -sh /*

# Find large files
find / -type f -size +100M -exec ls -lh {} \; 2>/dev/null | sort -k5 -h

# I/O stats
iostat -xz 1
sar -d 1

# Mount status
mount
cat /proc/mounts

# Filesystem check
fsck -n /dev/sda1

# SMART status
smartctl -a /dev/sda

# LVM status
lvs
pvs
vgs

# Deleted files
lsof +L1

Interview talking point: “df -i exposes inode exhaustion that df -h hides since a volume can run out of inodes while blocks still look free, and lsof +L1 lists deleted files still held open.”

Q648: How do you debug I/O issues?

What it is: iostat -xz reports per-device utilization and queue depth, iotop shows writing processes, pidstat -d tracks per-process I/O, strace traces syscalls, and ioping samples latency.

Why it is used: Determine whether slowness comes from the disk, one device, or a single noisy process.

# iostat
iostat -xz 1

# iotop
iotop -o

# pidstat
pidstat -d 1
pidstat -D sda 1

# blockdev
blockdev --getsize64 /dev/sda
blockdev --report /dev/sda

# hdparm
hdparm -tT /dev/sda

# ftrace
# Block I/O tracing
echo nop > /sys/kernel/debug/tracing/current_tracer
echo 1 > /sys/kernel/debug/tracing/events/block/block_rq_issue/enable
cat /sys/kernel/debug/tracing/trace_pipe

# strace for I/O
strace -e trace=openat,read,write -p $(pgrep -f myapp)

# ioping
ioping -c 10 /dev/sda
ioping -c 10 /mnt/data

Interview talking point: “iostat attributes load to the device while iotop and pidstat attribute it to the process, and ioping -c 10 compares raw device latency against the same filesystem mounted at a path.”

Q649: How do you debug Kubernetes issues?

What it is: Debugging Kubernetes uses events sorted by timestamp, container logs including the previous container, describe for detail, and exec for an interactive shell.

Why it is used: To find why a pod is failing or behaving oddly.

# Check events
kubectl get events --sort-by='.lastTimestamp'

# Check logs
kubectl logs -f deployment/myapp
kubectl logs -f deployment/myapp --previous

# Describe
kubectl describe pod myapp-pod

# Exec into container
kubectl exec -it myapp-pod -- /bin/bash

Interview talking point: “kubectl logs with —previous shows the log of the last crashed container, which is often the only evidence left after the kubelet restarts it.”

Q650: How do you debug service failures?

What it is: A service debug workflow using systemctl status and failed units, journalctl logs, process inspection, config tests, dependency views, strace, and cgroup information.

Why it is used: To find why a service will not start or is misbehaving without guessing.

# Service status
systemctl status service-name
systemctl list-units --failed

# Service logs
journalctl -u service-name -n 50
journalctl -u service-name --since "1 hour ago"
journalctl -xe

# Process info
ps auxf | grep service-name
lsof -p $(pgrep -f service-name)

# Configuration test
service-name -t
nginx -t
apache2ctl configtest

# Dependencies
systemctl list-dependencies service-name

# Resource limits
cat /proc/$(pgrep -f service-name)/limits

# Strace
strace -f -p $(pgrep -f service-name)
strace -c service-name

# Cgroups
systemd-cgls | grep service-name
systemctl show service-name

Interview talking point: “Running the config test commands such as nginx -t before restarting catches a broken config first, and strace -c gives a syscall summary while strace -f follows child processes.”

Q651: How do you debug service failures?

What it is: Service failure debugging reads systemctl status and failed units, journalctl for logs, config tests like nginx -t, dependency listings, strace for syscalls and the process limits file.

Why it is used: To find why a service will not start, crashes or hangs.

# Service status
systemctl status service
systemctl list-units --failed

# Service logs
journalctl -u service -n 50
journalctl -u service --since "1 hour ago"
journalctl -xe

# Process info
ps auxf | grep service
lsof -p $(pgrep -f service)

# Configuration test
nginx -t
apache2ctl configtest
mysqladmin ping

# Dependencies
systemctl list-dependencies service
systemctl is-active service

# Strace
strace -f -p $(pgrep -f service)
strace -c service

# Limits
cat /proc/$(pgrep -f service)/limits

Interview talking point: “nginx -t and apache2ctl configtest validate configuration before you restart, and strace -c summarizes syscalls per service so a hang can be pinned to file, network or signal waits.”

Q652: How do you debug service issues?

What it is: Service debugging starts with systemctl status and list-failed, then journalctl for logs, config tests, dependency listings, and the process limits, environment, and cgroup.

Why it is used: To find why a unit will not start or keeps restarting.

# Service status
systemctl status service
systemctl list-failed

# Logs
journalctl -u service -n 50
journalctl -u service --since "1 hour ago"
journalctl -xe

# Process
ps auxf | grep service
lsof -p $(pgrep -f service)

# Configuration
service configtest
nginx -t

# Dependencies
systemctl list-dependencies service
systemctl is-active service

# Resources
cat /proc/$(pgrep -f service)/limits

# Network
netstat -tulpn | grep service

# Environment
cat /proc/$(pgrep -f service)/environ | tr '\0' '\n'

# Cgroups
systemd-cgls | grep service

Interview talking point: “journalctl -u with —since pins logs to a time window, while nginx -t validates configuration before a reload so a bad file is caught early.”

Q653: How do you debug storage issues?

What it is: Storage debugging checks space with df -h, inodes with df -i, finds large files, reads iostat, verifies mounts, runs fsck, and checks disks with smartctl.

Why it is used: To tell full-disk, inode exhaustion, failing hardware, and slow I/O apart.

# Disk usage
df -h
df -i

# Find large files
find / -type f -size +100M 2>/dev/null | head -20

# I/O stats
iostat -xz 1
sar -d 1

# Mount issues
mount
cat /proc/mounts

# Filesystem check
fsck -n /dev/sda1

# LVM issues
lvs
pvs
vgs
lvdisplay

# NFS issues
showmount -e server
mount -v server:/share /mnt

# SMART status
smartctl -a /dev/sda
smartctl -H /dev/sda

# Lsof for deleted files
lsof +L1

Interview talking point: “df -i catches inode exhaustion that df -h misses, and lsof +L1 lists deleted files still held open, keeping apparent usage high until the handle closes.”

Q654: How do you debug system issues?

What it is: System debugging gathers identity from uname and os-release, hardware from lshw and lspci, logs from dmesg and journalctl, processes with ps auxf, and resources with df, free, vmstat and iostat.

Why it is used: To narrow an unknown failure down to hardware, resource, process or service cause.

# System information
uname -a
cat /etc/os-release
lsb_release -a

# Hardware info
lshw
lspci
lsblk

# System logs
dmesg | tail
journalctl -xe
tail -f /var/log/syslog

# Process status
ps auxf
top
htop

# Resource usage
df -h
free -h
vmstat 1
iostat -xz 1

# Network status
ip addr
ip route
netstat -tulpn
ss -tulpn

# Service status
systemctl status service
systemctl list-failed

Interview talking point: “ps auxf prints the process tree so parent-child relationships are visible, and systemctl list-failed immediately names units that did not reach a clean state.”

Q655: How do you diagnose kernel issues?

What it is: Kernel diagnosis uses dmesg for ring buffer messages, kern.log for persistent records, proc config.gz or the boot config for the built configuration, and strace to count and filter a program system calls.

Why it is used: To pinpoint whether a crash, hang, or memory kill originated in the kernel or an application.

# Kernel messages
dmesg
dmesg | tail
dmesg -T | grep -i error

# Kernel panic
# Check /var/log/kern.log
tail -f /var/log/kern.log

# Kernel configuration
zcat /proc/config.gz
# or
cat /boot/config-$(uname -r)

# System calls
strace -c ./program
strace -e openat ls

# Kernel debugging
echo 1 > /proc/sys/kernel/debug/earlyprintk
echo "debug" > /sys/power/state

# OOM killer
dmesg | grep -i "out of memory"
cat /var/log/syslog | grep -i oom

Interview talking point: “dmesg -T adds human-readable timestamps so you can line kernel events up with log entries, and grepping for out of memory reveals which process the OOM killer chose.”

Q656: How do you handle pod failures?

What it is: Pod failure handling combines a restart policy of Always, OnFailure, or Never, liveness probes to restart hung containers, and resource requests and limits.

Why it is used: To recover from crashes and stop one pod from starving the others.

# Restart policy
# Always, OnFailure, Never

# Liveness probe
livenessProbe:
  httpGet:
    path: /healthz
    port: 8080
  failureThreshold: 3
  periodSeconds: 10

# Resources
resources:
  requests:
    memory: "256Mi"
    cpu: "200m"
  limits:
    memory: "512Mi"
    cpu: "500m"

Interview talking point: “The restart policy governs container restarts while the liveness probe decides when a restart is needed, and limits keep a misbehaving pod from exhausting node memory.”

Q657: How do you perform root cause analysis?

What it is: Root cause analysis digs past symptoms using the 5 Whys to reach a process gap, supported by evidence from journalctl, sar, traces, core dumps, and a minute-by-minute timeline.

Why it is used: To fix the underlying cause so the same incident does not recur.

# 5 Whys Analysis
# Problem: API response time increased
# Why 1: Database queries slow
# Why 2: Missing index
# Why 3: New feature added without proper schema review
# Why 4: Code review didn't catch it
# Why 5: Process doesn't require schema review

# Corrective Action: Implement schema review in CI/CD

# Tools for RCA
# Logs
journalctl -u service -n 100

# Metrics
# Compare before/after
sar -q

# Traces
# Jaeger, Zipkin

# Dumps
# Core files, heap dumps

# Timeline
# Create incident timeline
# 14:00 - Alert triggered
# 14:05 - On-call acknowledged
# 14:10 - Root cause identified
# 14:15 - Fix deployed
# 14:20 - Service recovered

Interview talking point: “The example keeps asking why until it lands on missing schema review in CI, not on the missing index. A timeline with entries from alert to recovery keeps the post-mortem factual instead of speculative.”

Q658: How do you troubleshoot authentication?

What it is: Authentication troubleshooting tails auth.log or the secure log, tests SSH verbosely with ssh -v, inspects PAM-related entries, and checks SSSD through its journal.

Why it is used: To determine whether a login failure is credentials, PAM, directory service, or SSH configuration.

# Check logs
tail -f /var/log/auth.log
tail -f /var/log/secure

# PAM
auth.log

# SSH
ssh -v user@host
tail -f /var/log/auth.log

# SSSD
journalctl -u sssd

Interview talking point: “ssh -v prints negotiation step by step so you can see whether failure happens before authentication, during key exchange, or at password check. On Debian-family systems failures land in auth.log while on RHEL-family systems they land in /var/log/secure.”

Q659: How do you troubleshoot container issues?

What it is: Container debugging uses docker logs for output, docker inspect for configuration, docker exec to run commands inside, and docker stats for resource use.

Why it is used: To find why a container exits, hangs, or underperforms without rebuilding it.

# View logs
docker logs container_id
docker logs -f container_id
docker logs --tail 100 container_id

# Inspect container
docker inspect container_id
docker inspect --format='{{.NetworkSettings.IPAddress}}' container_id

# Execute commands
docker exec -it container_id /bin/bash
docker exec container_id ps aux

# Resource usage
docker stats container_id
docker stats --no-stream container_id

# Network debugging
docker run --rm -it --network container:name nicolaka/netshoot

Interview talking point: “docker logs —tail 100 keeps output short while -f follows live, and the inspect format flag pulls a single field such as the container IP. —no-stream gives one stats snapshot suitable for scripting.”

Q660: How do you troubleshoot disk issues?

What it is: Disk troubleshooting covers space with df -h, inodes with df -i, large directories with du -sh sorted, SMART health with smartctl, bad blocks, and fsck for filesystem repair.

Why it is used: Diagnosing full filesystems, exhausted inodes, and failing drives.

# Check disk space
df -h

# Check inodes
df -i

# Find large files
du -sh /* 2>/dev/null | sort -rh | head

# Check disk health
smartctl -a /dev/sda

# Check for bad blocks
badblocks -sv /dev/sda

# Check filesystem
fsck /dev/sda1

Interview talking point: “A filesystem can show free space yet refuse new files because inodes ran out, so df -i is checked beside df -h, and du -sh piped to sort -rh names the guilty directory.”

Q661: How do you troubleshoot disk issues?

What it is: Disk troubleshooting uses df -h for space and df -i for inodes, du to find large directories, smartctl for drive health, fsck -n for a read-only filesystem check, and iostat or iotop for I/O load.

Why it is used: To find what fills a disk, whether it is failing, and which process hammers it.

# Check space
df -h
df -i

# Check inodes
df -i

# Find large files
du -sh /*
du -sh /var/* | sort -rh | head

# Check disk health
smartctl -a /dev/sda
smartctl -H /dev/sda

# Check filesystem
fsck -n /dev/sda1

# Check I/O
iostat -x 1
iotop

Interview talking point: “A filesystem can have free blocks but no free inodes, so df -i matters as much as df -h when small files proliferate. fsck -n only reports and does not repair, making it safe to run on a suspect filesystem.”

Q662: How do you troubleshoot disk space issues?

What it is: Commands for chasing disk usage: df -h for filesystem space, df -i for inodes, du with sorting for big directories, find for large files, journalctl for journal size, and lsof +L1 for deleted-but-open files.

Why it is used: To locate what fills a filesystem and reclaim space quickly.

# Disk usage
df -h
df -i

# Largest directories
du -sh /*
du -sh /var/*
du -shx /var/* | sort -rh | head

# Largest files
find / -type f -size +100M -exec ls -lh {} \; 2>/dev/null | sort -k5 -h

# Log files
journalctl --disk-usage
du -sh /var/log/*
find /var/log -type f -mtime +7 -delete

# Deleted but open files
lsof +L1
ls -l /proc/*/fd/* | grep deleted

Interview talking point: “df -i matters because a filesystem can show free bytes yet be full of inodes, and lsof +L1 finds space held by deleted files still open by a process.”

Q663: How do you troubleshoot DNS resolution?

What it is: DNS troubleshooting uses nslookup and dig +trace, checks /etc/resolv.conf, flushes caches with systemd-resolve —flush-caches, and queries a chosen nameserver directly.

Why it is used: To find whether failures come from resolver configuration, a stale cache or the upstream server.

# Test DNS
nslookup host
dig host +trace

# Check resolv.conf
cat /etc/resolv.conf

# Flush DNS
systemd-resolve --flush-caches

# Test specific server
nslookup host 8.8.8.8

Interview talking point: “nslookup host 8.8.8.8 bypasses the local resolver, so a difference between that and a plain lookup isolates the fault to your own caching layer.”

Q664: How do you troubleshoot DNS?

What it is: DNS troubleshooting tests resolution with nslookup, dig, and host, checks /etc/resolv.conf and systemd-resolve status for the resolver, flushes caches, and queries a specific server with dig at an address.

Why it is used: To tell whether a name fails at the resolver, the cache, or the authoritative side.

# Test resolution
nslookup google.com
dig google.com
host google.com

# Check resolver
cat /etc/resolv.conf
systemd-resolve --status

# Flush DNS
systemd-resolve --flush-caches
resolvectl flush-caches

# Test specific server
dig @8.8.8.8 google.com
nslookup google.com 8.8.8.8

Interview talking point: “dig at 8.8.8.8 bypasses the local resolver entirely, so if that works while a plain lookup fails, the problem is local configuration or cache. systemd-resolve flush-caches clears cached answers before you retest.”

Q665: How do you troubleshoot filesystem errors?

What it is: Filesystem checking uses fsck for read-only or auto-repair passes, xfs_repair for XFS, and df -i or tune2fs to inspect inode usage and detect exhaustion.

Why it is used: To repair a filesystem after a crash or find why writes fail even with free space.

# Check filesystem
fsck -n /dev/sda1  # dry run
fsck -y /dev/sda1  # auto repair

# XFS specific
xfs_repair /dev/sda1
xfs_info /dev/sda1

# View inode usage
df -i
tune2fs -l /dev/sda1 | grep -i inode

# Recover deleted files (ext3/4)
extundelete /dev/sda1 --inode 2

Interview talking point: “fsck -n reports without changing anything, so it comes first on a live system, while fsck -y answers yes to every repair prompt. df -i catches inode exhaustion that df -h completely misses.”

Q666: How do you troubleshoot mail issues?

What it is: Mail troubleshooting checks the queue with mailq or postqueue, follows logs, flushes stuck mail, and inspects a message by queue id with postcat. postsuper removes messages, and telnet to port 25 tests the SMTP dialogue directly.

Why it is used: To find where in the delivery chain a message is stuck and clear or resend it.

# Check mail queue
mailq
postqueue -p

# View mail log
tail -f /var/log/mail.log
journalctl -u postfix -f

# Flush queue
postfix flush
postqueue -f

# Check specific mail
postcat -q <queue_id>

# Remove stuck mail
postsuper -d <queue_id>
postsuper -d ALL deferred

# Test SMTP
telnet mail.example.com 25
nc -C mail.example.com 25

# Check delivery
echo "Test" | mail -v user@example.com

Interview talking point: “postcat shows a message headers and recipients for a queue id while postsuper -d ALL deferred purges the retry backlog, and a raw connection on port 25 confirms the remote server response code.”

Q667: How do you troubleshoot service failures?

What it is: Service troubleshooting uses systemctl status, journalctl -u with -n 50 for recent lines, a manual config test such as nginx -t, list-dependencies, and ss or netstat for listening ports.

Why it is used: Diagnosing a failed service with evidence instead of restarting blindly.

# Check status
systemctl status service

# Check logs
journalctl -u service -n 50
tail -f /var/log/messages

# Test manually
/usr/sbin/nginx -t

# Check dependencies
systemctl list-dependencies nginx

# Check ports
netstat -tulpn | grep LISTEN
ss -tulpn | grep LISTEN

Interview talking point: “Running nginx -t by hand separates a bad configuration from a runtime failure, and ss -tulpn filtered for LISTEN confirms the port actually bound after the restart.”

Q668: How do you troubleshoot service failures?

What it is: Service troubleshooting checks systemctl status and journalctl for the unit, tests the daemon manually with a config check, reviews dependencies and listening ports, then kills and restarts the service.

Why it is used: To get a failing daemon running again with a clear root cause.

# Check status
systemctl status service
journalctl -u service -n 50

# Check logs
journalctl -xe
tail -f /var/log/messages

# Test manually
/usr/sbin/nginx -t

# Dependencies
systemctl list-dependencies service

# Ports
netstat -tulpn | grep LISTEN

# Kill and restart
systemctl kill service
systemctl restart service

Interview talking point: “nginx -t validates configuration before a restart, which prevents bouncing a service only to have it fail again on the same bad config. journalctl -xe shows recent log entries with context, and list-dependencies reveals units that must be active first.”

Q669: How do you troubleshoot service failures?

What it is: A systemd troubleshooting sequence: status and journalctl for logs, daemons own config tests like nginx -t, permission checks, systemctl list-dependencies, daemon-reload, and process inspection with ps, lsof, and strace.

Why it is used: To find why a service will not start or is misbehaving.

# Service status
systemctl status service-name
journalctl -u service-name -n 50
journalctl -u service-name --since "1 hour ago"

# Service logs
cat /var/log/service/name.log
tail -f /var/log/syslog | grep service

# Configuration test
apache2ctl configtest
nginx -t
named-checkconf

# Check permissions
ls -la /etc/service/
ls -l /var/run/service/

# Dependencies
systemctl list-dependencies service-name
systemctl daemon-reload

# Process issues
ps aux | grep service
lsof -p <pid>
strace -p <pid>

Interview talking point: “Running nginx -t or apache2ctl configtest before restarting catches a bad config early, and daemon-reload is what makes systemd re-read unit files after you edit them.”

Q670: How do you troubleshoot SSL/TLS?

What it is: SSL/TLS checks use openssl s_client to open a TLS connection, optionally with SNI via -servername, then pipe the session into openssl x509 to print certificate text, dates, or verify the chain against a CA file.

Why it is used: To diagnose certificate errors, expiry, and chain problems on a live service.

# Test SSL
openssl s_client -connect example.com:443
openssl s_client -connect example.com:443 -servername example.com

# Check certificate
echo | openssl s_client -connect example.com:443 2>/dev/null | openssl x509 -noout -text

# Verify chain
openssl verify -CAfile ca.crt server.crt

# Check expiry
echo | openssl s_client -connect example.com:443 2>/dev/null | openssl x509 -noout -dates

Interview talking point: “Passing -servername sends SNI, which is required when one host serves multiple certificates. Supplying echo as input makes s_client exit immediately, and piping into x509 with the dates option is the quickest expiry check without a browser.”

Q671: How do you troubleshoot virtualization issues?

What it is: Troubleshooting virtualization starts with confirming the kvm modules are loaded and CPU virtualization flags are present, then reading libvirtd logs. virsh console and dump provide guest-level debugging, while bridge commands inspect networking.

Why it is used: To find whether a VM problem comes from missing hardware support, service logs, or the virtual network.

# Check KVM
lsmod | grep kvm
cat /proc/cpuinfo | grep vmx
virt-host-validate

# View logs
journalctl -u libvirtd
dmesg | grep -i kvm

# Debug VM
virsh console webserver
virsh dump webserver /tmp/core.qemu

# Network issues
brctl show
ip link show virbr0

# Performance
virsh dominfo webserver
virsh cpu-stats webserver

Interview talking point: “virt-host-validate gives a quick pass or fail on host readiness, and dmesg filtered for kvm often reveals immediately whether hardware virtualization was refused at boot.”

Q672: How do you debug performance issues?

What it is: A performance triage toolkit covering CPU, memory, disk I/O, and network, plus perf and Python profiling for finding per-process hotspots.

Why it is used: To identify which resource and which process is causing a slowdown.

# CPU analysis
top -c
htop
mpstat -P ALL 1
pidstat -p <pid> 1

# Memory analysis
free -h
vmstat 1
pmap -x <pid>
cat /proc/<pid>/status

# I/O analysis
iostat -xz 1
iotop
pidstat -d 1

# Network analysis
nethogs
iftop
ss -s

# System resources
sar -A 1 5
atop

# Process analysis
perf top
flamegraph.pl < perf.data

# Application profiling
python -m cProfile -o profile.out app.py
pyprof2calltree -i profile.out

Interview talking point: “pidstat attributes CPU and disk activity to a single PID while mpstat spreads CPU across all cores, so you narrow from a busy box to a specific offending process.”

Q673: How do you debug performance issues?

What it is: Performance debugging samples CPU with top and mpstat, memory with free and vmstat, I/O with iostat and iotop, network with nethogs and iftop, and profiles with perf.

Why it is used: To identify the saturated resource and the code paths driving it.

# CPU
top
htop
mpstat -P ALL 1

# Memory
free -h
vmstat 1

# I/O
iostat -xz 1
iotop

# Network
nethogs
iftop

# System-wide
sar -A 1 5

# Process
perf top
perf record -g -p <pid>
perf report

# Flame graph
git clone https://github.com/brendangregg/FlameGraph.git
perf record -F 99 -g -p <pid>
perf script | ./stackcollapse-perf.pl | ./flamegraph.pl > flame.svg

Interview talking point: “perf record -F 99 -g samples at 99 Hz, and piping perf script through stackcollapse-perf.pl and flamegraph.pl renders a flame.svg that shows where CPU time actually goes.”

Q674: How do you debug performance regressions?

What it is: sar records system metrics into a data file for baseline and regression windows, sadf converts them for diffing, perf counts and profiles, and bpftrace probes kernel functions.

Why it is used: Compare a slow build against a known-good baseline instead of guessing at causes.

# Baseline comparison
# Save baseline
sar -A -o baseline.data 1 60

# Compare with regression
sar -A -o regression.data 1 60

# Using sadf
sadf -d baseline.data | head
sadf -d regression.data | head

# Using SAR for comparison
sar -q baseline.data > baseline_q.txt
sar -q regression.data > regression_q.txt
diff baseline_q.txt regression_q.txt

# Using perf
perf stat -a --repeat=3 ./benchmark
perf record -g ./regression
perf report

# Using bpfb
# Baseline
bpftrace -e 'kprobe:do_nanosleep { @start = nsecs(); }' > baseline.txt

# Regression
bpftrace -e 'kprobe:do_nanosleep { @start = nsecs(); }' > regression.txt

Interview talking point: “Diffing sar -q output line by line isolates load average and run queue shifts, while perf record with -g captures call graphs that show where cycles are actually being burned.”

Q675: How do you troubleshoot a slow database?

What it is: Database troubleshooting checks system CPU, memory, and I/O, then inspects pg_stat_activity or SHOW PROCESSLIST, profiles slow queries with EXPLAIN ANALYZE, and reviews indexes.

Why it is used: To find whether slowness comes from host resources, expensive queries, or missing indexes.

# Database troubleshooting
# 1. Check system resources
# CPU, Memory, I/O

# 2. Check database stats
# PostgreSQL
# pg_stat_activity
# pg_stat_statements

# MySQL
# SHOW PROCESSLIST;
# SHOW STATUS;

# 3. Check slow queries
# PostgreSQL
# pg_stat_statements
# EXPLAIN ANALYZE

# MySQL
# SHOW PROCESSLIST
# EXPLAIN

# 4. Check indexes
# PostgreSQL
# \d table_name

# MySQL
# SHOW INDEX FROM table

# 5. Fixes
# Add indexes
# Optimize queries
# Tune configuration
# Scale horizontally
# Add read replicas

Interview talking point: “EXPLAIN ANALYZE actually runs the query and reports real timing, whereas plain EXPLAIN only estimates the plan, so compare analyzed output before and after adding an index.”

Q676: How do you troubleshoot database performance?

What it is: Database performance work runs EXPLAIN ANALYZE in PostgreSQL and EXPLAIN FORMAT=JSON in MySQL, checks sessions with pg_stat_activity, and enables the slow query log.

Why it is used: To find slow queries and poor execution plans before paying for more hardware.

# PostgreSQL
EXPLAIN ANALYZE SELECT * FROM table;

# MySQL
EXPLAIN FORMAT=JSON SELECT * FROM table;

# Check connections
SELECT * FROM pg_stat_activity;

# Check slow queries
SHOW VARIABLES LIKE 'slow_query_log';

Interview talking point: “EXPLAIN shows the estimated plan while ANALYZE actually runs the query and reports real timings, so results reflect actual rows instead of the planner’s estimates.”

Q677: How do you troubleshoot disk I/O bottlenecks?

What it is: iostat -x shows extended device metrics including await latency, iotop ranks processes by I/O, and /proc/diskstats exposes raw per device queue counters.

Why it is used: To tell whether a slowdown is the disk itself or one process hammering it.

# Check I/O
iostat -x 1

# Check processes
iotop

# Check latency
iostat -x 1 | grep await

# Check queue
cat /proc/diskstats

Interview talking point: “await is the average time per I/O request, so a high value with low throughput indicts the device while iotop names the offending process.”

Q678: How do you troubleshoot network latency?

What it is: Latency troubleshooting uses ping for round trip time, mtr for per hop latency, ip route and traceroute for path, dig for DNS, and iptables -L -n for rules.

Why it is used: To locate where along the path or in name resolution the delay originates.

# Check latency
ping -c 10 host
mtr host

# Check routes
ip route
traceroute host

# Check DNS
dig host
nslookup host

# Check firewall
iptables -L -n

Interview talking point: “mtr merges ping and traceroute so you see which hop’s latency jumps, and iptables -L -n prints rules numerically without slow reverse DNS lookups.”

Q679: How do you troubleshoot performance bottlenecks?

What it is: Bottleneck analysis tools per subsystem: atop and sar for overall load, mpstat and pidstat for CPU, pmap and proc status for memory, iostat for disk, nethogs and iftop for network, plus perf and strace for the app.

Why it is used: To move from a vague slowness report to the specific resource and process causing it.

# Overall system
top
htop
atop
sar -A 1 5

# CPU
mpstat -P ALL 1
pidstat -p <pid> 1

# Memory
pmap -x <pid>
cat /proc/<pid>/status

# I/O
iostat -xz 1
pidstat -d 1

# Network
nethogs
iftop
sar -n DEV 1

# Application
perf record -g -p <pid>
perf report

# System calls
strace -c -p <pid>
strace -tt -p <pid>

Interview talking point: “pidstat and strace attach to a single PID, so after top identifies a suspect process you drill into that one PID instead of watching the whole machine.”

Q680: How do you troubleshoot performance?

What it is: Performance triage collects a system overview with sar, per-core CPU with mpstat, memory and swap with vmstat, disk latency with iostat -x, network flows with iftop or nethogs, and per-process cost with strace and perf.

Why it is used: To find which resource is the bottleneck before digging into a single process.

# System overview
sar -A 1 5

# CPU
mpstat -P ALL 1

# Memory
vmstat 1

# I/O
iostat -x 1

# Network
iftop
nethogs

# Processes
strace -c -p PID
perf top

Interview talking point: “iostat -x adds await and utilization metrics that show whether disks are the constraint, while vmstat 1 reveals swap activity and run-queue pressure. iftop reports bandwidth per connection and nethogs attributes traffic to specific processes.”

Q681: How do you troubleshoot slow database queries?

What it is: Slow query troubleshooting uses pg_stat_statements and EXPLAIN ANALYZE on PostgreSQL, SHOW PROCESSLIST and EXPLAIN on MySQL, then index and configuration fixes.

Why it is used: To find why a query is slow and fix it at the execution plan level.

# Check slow queries
# PostgreSQL
# pg_stat_statements
# EXPLAIN ANALYZE

# MySQL
# SHOW PROCESSLIST
# EXPLAIN

# Check indexes
# Missing indexes
# Outdated statistics

# Fixes
# Add indexes
# Rewrite queries
# Tune configuration
# Scale horizontally

Interview talking point: “EXPLAIN ANALYZE actually runs the query and reports real timing, while plain EXPLAIN shows only the estimated plan, so comparing them exposes wrong row estimates.”

Q682: How do you troubleshoot slow systems?

What it is: A layered troubleshooting routine for slow hosts: top, htop, and sorted ps for CPU, free, vmstat and pmap for memory, iostat, iotop and sar for I/O, plus strace, perf, uptime, and load averages.

Why it is used: To find which resource, CPU, memory, disk, or network, is the actual bottleneck.

# CPU usage
top
htop
ps aux --sort=-%cpu | head

# Memory
free -h
vmstat 1
pmap -x <pid>

# I/O
iostat -xz 1
iotop
sar -b 1

# Network
netstat -i
ss -s

# Process analysis
strace -c <command>
perf top

# System resources
uptime
cat /proc/loadavg

Interview talking point: “ps aux sorted by CPU with head ranks consumers instantly, while strace -c gives a per-command syscall count summary instead of live output, which is what you want first.”

CPU/Memory Scenarios

Q683: How do you configure OOM killer?

What it is: The OOM killer reclaims memory by killing a process; overcommit_memory controls allocation policy and oom_score_adj tilts which PID is chosen.

Why it is used: To protect critical daemons and avoid surprises when memory runs out.

# Check
cat /proc/sys/vm/overcommit_memory

# Disable
sysctl -w vm.overcommit_memory=2

# Adjust OOM score
echo -15 > /proc/PID/oom_score_adj

# Never kill root
sysctl -w vm.oom_dump_tasks=1

Interview talking point: “Setting vm.overcommit_memory=2 refuses allocations beyond the configured ratio so they fail cleanly, while a negative oom_score_adj makes a given PID less likely to be selected.”

Q684: How do you debug memory leaks?

What it is: Leak hunting tools include valgrind with full leak checking, pmap and /proc status for mappings, smem for accounting, gdb for inspection, and jmap for Java heaps.

Why it is used: Identify which process or allocation path is growing before the host runs out of memory.

# Using valgrind
valgrind --leak-check=full --show-leak-kinds=all --track-origins=yes ./program

# Using memleax
memleax -p $(pgrep -f myapp)

# System memory analysis
ps aux --sort=-rss | head
pmap -x $(pgrep -f myapp)

# Using /proc
cat /proc/$(pgrep -f myapp)/status | grep -i vm

# Using smem
smem -m
smem -r -k

# gdb debugging
gdb -p $(pgrep -f myapp)
(gdb) info proc mappings
(gdb) info registers

# For Java
jmap -heap $(pgrep -f java)
jmap -histo $(pgrep -f java) | head -20
jmap -dump:format=b,file=heap.bin $(pgrep -f java)

# For Python
python -m memory_profiler program.py

Interview talking point: “ps aux sorted by rss shows who is largest right now, while jmap histo and a heap dump reveal which objects keep accumulating inside a long-running JVM.”

Q685: How do you recover from memory issues?

What it is: Memory diagnosis by grepping out of memory messages, ranking processes with ps, dumping maps with pmap, dropping caches, and adjusting per-process OOM scores.

Why it is used: To find which process exhausted RAM and to free or protect memory quickly.

# Check OOM killer
dmesg | grep -i "out of memory"
journalctl -k | grep -i "killed process"

# Check process memory
ps aux --sort=-%mem | head
pmap -X PID

# Clear cache
sync
echo 3 > /proc/sys/vm/drop_caches

# Kill process manually
kill -15 PID
kill -9 PID

# Adjust OOM settings
echo -15 > /proc/PID/oom_score_adj

Interview talking point: “Writing 3 to drop_caches reclaims pagecache, dentries and inodes but not process memory, while a negative oom_score_adj makes a process less likely to be chosen by the OOM killer.”

Q686: How do you troubleshoot high CPU usage?

What it is: High CPU debugging uses top or htop for live load, ps sorted by CPU for offenders, ps -eLf for threads per PID, and mpstat -P ALL per core.

Why it is used: To decide whether one process, one thread or one saturated core is causing the load.

# Top processes
top
htop

# Per-process
ps aux --sort=-%cpu | head

# Threads
ps -eLf -p PID

# System-wide
mpstat -P ALL 1

Interview talking point: “mpstat -P ALL splits usage per CPU, which separates a single pegged core from true saturation, since top alone can look alarming when only one core is maxed.”

Q687: How do you troubleshoot high CPU?

What it is: High CPU diagnosis ranks processes with top, htop, or ps sorted by CPU, breaks usage down per core with mpstat, lists a process’s threads, and profiles it with jstack or strace.

Why it is used: To identify which process or thread is consuming CPU and why.

# Top processes
top
htop
ps aux --sort=-%cpu | head

# Per-core
mpstat -P ALL 1

# Threads
ps -eLf -p PID

# Java
jstack PID

# strace
strace -c -p PID

Interview talking point: “mpstat -P ALL separates per-core load, which reveals whether the spike is one saturated core or the whole box. strace -c counts syscalls by type for a PID, pointing to whether time goes to I/O or computation.”

Q688: How do you troubleshoot high load?

What it is: Load troubleshooting reads uptime and top, sorts processes by CPU with ps aux, checks memory with free -h, profiles I/O with iostat -x and iotop, and looks for zombies.

Why it is used: Identifying whether CPU, memory, I/O, or stuck processes cause the load.

# Check load
uptime
top
htop

# Check processes
ps aux --sort=-%cpu | head

# Check memory
free -h

# Check I/O
iostat -x 1
iotop

# Check for zombie processes
ps aux | grep zombie

Interview talking point: “Load averages include processes waiting on I/O, so high load with an idle CPU points at disks; iostat -x and iotop confirm that before you start killing CPU-heavy processes.”

Q689: How do you troubleshoot high memory?

What it is: Memory troubleshooting reads free -h and /proc/meminfo, ranks processes by memory, inspects kernel slab with slabtop, checks dmesg and journalctl for out-of-memory kills, and can drop page cache.

Why it is used: To find what consumes RAM, confirm OOM kills, and free reclaimable cache.

# Check usage
free -h
cat /proc/meminfo

# Top processes
ps aux --sort=-%mem | head

# Slab
slabtop

# OOM killer
dmesg | grep -i "out of memory"
journalctl -k | grep -i oom

# Clear cache
sync
echo 3 > /proc/sys/vm/drop_caches

Interview talking point: “The out-of-memory lines in dmesg or journalctl show which process was sacrificed when RAM ran out. Dropping caches via the vm drop_caches sysctl reclaims page cache but does not fix a genuine leak, and cache itself is not a problem.”

Q690: How do you troubleshoot memory leaks?

What it is: Leak investigation uses free -h for totals, pmap -x for one process’s mapped memory, valgrind —leak-check=full on a program, and systemd-cgtop for per cgroup usage.

Why it is used: To watch a process grow over time and pinpoint code that allocates memory it never frees.

# Check memory
free -h

# Monitor process
pmap -x PID

# Use valgrind
valgrind --leak-check=full ./program

# Use systemd-cgtop
systemd-cgtop

Interview talking point: “pmap shows one process’s current mappings while systemd-cgtop ranks all cgroups, so the first narrows the suspect and the second finds it across the fleet.”

Q691: How do you troubleshoot OOM issues?

What it is: OOM diagnosis greps dmesg and the kernel journal for out of memory messages, ranks processes with ps aux sorted by memory, and tunes overcommit and swappiness in sysctl.

Why it is used: To identify which process was killed and whether kernel memory settings caused the pressure.

# Check OOM killer
dmesg | grep -i "out of memory"
journalctl -k | grep -i "killed process"

# Check process
ps aux --sort=-%mem | head

# Tune kernel
# /etc/sysctl.conf
vm.overcommit_memory = 1
vm.swappiness = 10

Interview talking point: “The OOM killer logs the process it killed and the sorted ps output names the consumer, while swappiness and overcommit change when that pressure builds at all.”

Disk Scenarios

Q692: How do you handle data corruption?

What it is: Corruption response: confirm with checksums such as md5sum, remount the filesystem read-only to halt writes, restore the last good backup, attempt point-in-time recovery, then verify integrity.

Why it is used: To stop the damage spreading and recover to a known good state.

# Data corruption response
# 1. Identify corruption
# Check logs
# Verify checksums
# md5sum

# 2. Stop writes
# Read-only mount
# mount -o remount,ro /data

# 3. Restore from backup
# Find last good backup
# Restore
# mysql -u root -p mydb < backup.sql

# 4. Point-in-time recovery
# PostgreSQL
# Find transaction ID
# pg_restore -P "2024-01-01 12:00:00" backup.dump

# 5. Verify integrity
# Check application data
# Run database checks

# 6. Prevention
# Enable checksums
# Regular backups
# Monitoring

Interview talking point: “mount -o remount,ro freezes further damage without unmounting and risking dirty state. Point-in-time recovery with pg_restore to a timestamp can land before the corruption, which a plain full backup cannot do.”

Q693: How do you handle storage full?

What it is: Disk-full recovery: locate big consumers with du and sort, trim journals with journalctl —vacuum-size, clear tmp and package caches, prune Docker, and check deleted-but-open files with lsof +L1.

Why it is used: To reclaim space fast and find what keeps filling the filesystem.

# Storage full response
# 1. Find large files
du -sh /*
du -sh /var/*
du -sh /var/log/*

# 2. Find large directories
du -ah / | sort -rh | head -20

# 3. Clean logs
journalctl --vacuum-size=100M
find /var/log -type f -mtime +30 -delete

# 4. Clean tmp
rm -rf /tmp/*
rm -rf /var/tmp/*

# 5. Clean package cache
apt clean
yum clean all

# 6. Docker cleanup
docker system prune -a

# 7. Find deleted files still open
lsof +L1

# 8. Extend storage
# Add volume
# Add to LVM

Interview talking point: “lsof +L1 exposes files deleted by an application that still holds them open, so df keeps counting them until the process exits. docker system prune -a removes unused images too, not just dangling ones.”

Q694: How do you recover from disk failure?

What it is: A storage recovery flow using smartctl for drive health, fsck for filesystem checks, remounting read-only, and replacing the disk in a RAID array.

Why it is used: To assess failing storage, limit further damage, and restore redundancy by swapping the disk.

# Check disk health
smartctl -a /dev/sda

# Check filesystem
fsck -n /dev/sda1

# Remount read-only
mount -o ro,remount /dev/sda1

# Try to fix
fsck -y /dev/sda1

# Replace disk
# 1. Partition new disk (sfdisk -d /dev/sda | sfdisk /dev/sdb)
# 2. Copy boot sector (dd if=/dev/sda of=/dev/sdb bs=512 count=1)
# 3. Rebuild RAID (mdadm --add /dev/md0 /dev/sdb)
# 4. Rebuild GRUB

Interview talking point: “fsck -n checks without modifying anything while fsck -y answers yes to prompts, and remounting read-only first stops writes to a suspect filesystem before mdadm —add rebuilds the array.”

Q695: How do you recover from RAID failure?

What it is: Recovery steps are mdadm —detail to see state, —examine to read a disk’s own metadata, then —remove the failed member and —add a replacement so the array rebuilds.

Why it is used: Restore redundancy after a disk dies, before a second failure costs data.

# Check
mdadm --detail /dev/md0

# Identify failed
mdadm --examine /dev/sdb1

# Remove failed
mdadm /dev/md0 --remove /dev/sdb1

# Add new
mdadm /dev/md0 --add /dev/sdf1

# Rebuild
# Automatic

Interview talking point: “—examine reads metadata straight from the device rather than from the array, which is how you identify which disk failed when device names are unclear, and the rebuild then starts automatically.”

Service Failure Scenarios

Q696: How do you troubleshoot service crashes?

What it is: Crash triage checks systemctl status, reads journalctl -u and journalctl -xe, enables unlimited cores in limits.conf, and opens saved cores with coredumpctl gdb.

Why it is used: To get the failure reason and, when needed, a core dump showing the crashing code.

# Check status
systemctl status service

# Check logs
journalctl -u service -n 50
journalctl -xe

# Core dumps
# Enable
# /etc/security/limits.conf
* soft core unlimited

# Analyze
coredumpctl
coredumpctl gdb PID

Interview talking point: “journalctl -u targets one unit while journalctl -xe shows recent system wide events with annotations, and coredumpctl gdb attaches the saved core directly to the failed PID.”

Network Scenarios

Q697: How do you handle a production outage?

What it is: The outage playbook runs detection, assessment, communication, mitigation, resolution, then a post-incident review with a timeline and action items.

Why it is used: To restore service quickly and turn the incident into lasting fixes.

# 1. Detection
# Monitor alerts
# User reports

# 2. Assessment
# Check severity
# Determine impact

# 3. Communication
# Create incident channel
# Update status page

# 4. Mitigation
# Stop bleeding
# Restore service

# 5. Resolution
# Fix root cause

# 6. Post-incident
# Document timeline
# Identify root cause
# Action items

Interview talking point: “Mitigation comes before root cause: stop the bleeding and restore service first, then investigate, and the post-incident review documents the timeline and action items without blame.”

Q698: How do you handle network outages?

What it is: Network outage triage: confirm reachability with ping, inspect interfaces with ip link and ip addr, check DNS in /etc/resolv.conf, review ip route, then restart networking or bounce the interface.

Why it is used: To isolate whether a failure is physical, addressing, routing, or name resolution.

# Network outage response
# 1. Verify outage
# ping gateway
# ping 8.8.8.8

# 2. Check interfaces
# ip link
# ip addr

# 3. Check DNS
# cat /etc/resolv.conf
# nslookup example.com

# 4. Check routes
# ip route

# 5. Recovery steps
# Reset network
systemctl restart networking

# Or
# ip link set eth0 down
# ip link set eth0 up

# For DNS issues
# systemd-resolve --flush-caches

# For cloud
# AWS
aws ec2 describe-instance-status --instance-id i-xxx

# 6. Contact provider
# If not resolvable internally

Interview talking point: “ping 8.8.8.8 succeeding while a hostname fails points at DNS rather than the network. Flushing with systemd-resolve —flush-caches clears a stale cache without touching interfaces or routes.”

Q699: How do you analyze network issues?

What it is: This is a network troubleshooting toolkit: netstat -s and ss -s for protocol stats, ss -tunapl for sockets, nethogs and iftop for bandwidth, and tcpdump for capture.

Why it is used: To localize a problem as latency, packet loss, saturation, or a misbehaving connection.

# Network stats
netstat -s
ss -s

# Per-socket stats
ss -tunapl

# Bandwidth
nethogs eth0
iftop

# Latency
ping -c 10 8.8.8.8
traceroute 8.8.8.8
mtr 8.8.8.8

# Capture
tcpdump -i eth0 host 192.168.1.1
tcpdump -i eth0 port 80

Interview talking point: “nethogs groups bandwidth by process while iftop groups it by connection, and mtr combines ping with traceroute so one command shows both latency and the hop where loss starts.”

Q700: How do you debug DNS issues?

What it is: dig, nslookup and host query nameservers, dig can trace delegation or test an AXFR zone transfer, and resolv.conf plus nsswitch.conf define the resolver and lookup order.

Why it is used: Isolate whether resolution breaks at the client, the cache, or the authoritative server.

# Query tests
dig @8.8.8.8 example.com
dig +trace example.com
nslookup example.com 8.8.8.8
host -v example.com

# Check resolver
cat /etc/resolv.conf
systemd-resolve --status

# Check DNS cache
# systemd-resolved
systemd-resolve --flush-caches

# nscd
nscd -i hosts

# Check server
dig @ns1.example.com example.com AXFR

# tcpdump
tcpdump -i eth0 -n port 53

# Check resolution order
getent hosts example.com

# Debug nsswitch
# /etc/nsswitch.conf
hosts: files dns

# Test with strace
strace -e openat,read nslookup example.com

Interview talking point: “getent hosts follows the nsswitch hosts files dns order while dig bypasses libc entirely, so a name answering in dig but not getent points at local files or a caching daemon.”

Q701: How do you debug network issues?

What it is: A layered network debug routine: check interfaces and routes, verify DNS, test connectivity hop by hop, probe ports, then capture packets with tcpdump.

Why it is used: To localize a failure to cabling, routing, name resolution, a service, or the firewall.

# Check interface status
ip link show
ip addr show
ethtool eth0
miitool eth0

# Check routing
ip route show
ip route get 8.8.8.8
ip neighbor show

# Check DNS
dig +short example.com
getent hosts example.com
cat /etc/resolv.conf

# Connectivity tests
ping -c 4 8.8.8.8
traceroute -I 8.8.8.8
mtr -n 8.8.8.8

# Port tests
nc -zv host port
telnet host port
ss -tulpn | grep :port

# Packet capture
tcpdump -i eth0 host 192.168.1.1
tcpdump -i eth0 port 80
tcpdump -i eth0 -w capture.pcap

Interview talking point: “ip route get 8.8.8.8 shows the exact route the kernel would pick, while mtr merges ping and traceroute into one report, giving a faster path to the fault than either alone.”

Q702: How do you debug network issues?

What it is: Network debugging walks from link status with ip and ethtool, through routes, DNS, connectivity, listening sockets, packet capture, firewall counters, and ARP neighbors.

Why it is used: To isolate whether a failure is physical, routing, name resolution, or filtering.

# Interface status
ip link show
ip addr show
ethtool eth0

# Routing
ip route
ip route get 8.8.8.8

# DNS
dig example.com
getent hosts example.com

# Connectivity
ping -c 4 8.8.8.8
traceroute 8.8.8.8

# Port status
netstat -tulpn
ss -tulpn

# Capture
tcpdump -i eth0 host 192.168.1.1
tcpdump -i eth0 port 80

# Firewall
iptables -L -n -v
iptables -t nat -L -n -v

# TCP issues
# Retransmits
netstat -s | grep -i retrans
# Connection states
ss -tan state time-wait

# ARP
ip neigh show
arp -a

Interview talking point: “ip route get 8.8.8.8 shows exactly which route and source address a packet would take, which answers routing questions faster than reading the whole table.”

Q703: How do you debug network issues?

What it is: Network debugging checks links and addresses with ip, routing with ip route, name resolution with dig and getent, reachability with ping, traceroute and mtr, ports with nc, and packets with tcpdump.

Why it is used: To separate DNS, routing, firewall and application faults step by step.

# Interface status
ip link
ip addr
ethtool eth0

# Routing
ip route
ip route get 8.8.8.8
ip neighbor show

# DNS
dig example.com
getent hosts example.com
cat /etc/resolv.conf

# Connectivity
ping -c 4 8.8.8.8
traceroute 8.8.8.8
mtr -n 8.8.8.8

# Ports
nc -zv host port
telnet host port

# Capture
tcpdump -i eth0 host 192.168.1.1
tcpdump -i eth0 port 80

# Firewall
iptables -L -n -v
firewall-cmd --list-all

Interview talking point: “ip route get 8.8.8.8 shows the exact path the kernel would take for that destination, which tells you in one line whether the problem is routing or something above it.”

Q704: How do you recover from network issues?

What it is: Network recovery steps that bounce the interface, renew the DHCP lease, restart the network service, review journal logs, and tune TCP buffer sysctls.

Why it is used: To restore connectivity on a host with link, addressing, or TCP stack problems.

# Reset network
ip link set eth0 down
ip link set eth0 up
dhclient -r eth0
dhclient eth0

# Restart network service
systemctl restart NetworkManager
systemctl restart networking

# Check logs
journalctl -xe

# Reset TCP/IP stack
sysctl -w net.ipv4.tcp_rmem="4096 87380 6291456"
sysctl -w net.ipv4.tcp_wmem="4096 65536 6291456"
sysctl -w net.core.rmem_max="12582912"
sysctl -w net.core.wmem_max="12582912"

Interview talking point: “ip link set down and up resets layer one, dhclient -r releases the lease before renewing, and the sysctl -w buffer writes only hold until reboot unless persisted.”

Q705: How do you troubleshoot DNS issues?

What it is: DNS troubleshooting uses dig, nslookup, and host to query resolvers, plus a specific server argument, axfr for zone transfer, minus x for reverse lookup, and a trace to follow delegation. Caches are flushed with resolvectl.

Why it is used: To isolate whether a name problem sits with the resolver, the authoritative server, or a stale cache.

# Query DNS
dig example.com
dig @8.8.8.8 example.com
nslookup example.com
host example.com

# DNS zone transfer
dig axfr example.com @ns1.example.com

# Reverse lookup
dig -x 192.168.1.10

# Trace DNS resolution
dig +trace example.com

# Check DNS server
dig +short myip.opendns.com @resolver1.opendns.com

# Flush DNS cache
systemd-resolve --flush-caches
# or
/etc/init.d/nscd restart
# or
resolvectl flush-caches

Interview talking point: “dig pointing at 8.8.8.8 bypasses the local resolver so you can tell whether the system or upstream is at fault, and +trace walks delegation from the roots step by step.”

Q706: How do you troubleshoot network connectivity issues?

What it is: Connectivity troubleshooting walks from link and address state through ping, traceroute, and mtr, then DNS with dig, listening ports with ss, and packets with tcpdump.

Why it is used: To find where a connection breaks instead of guessing at the symptom.

# Check interface status
ip link show
ip addr show
ethtool eth0

# Test connectivity
ping -c 4 8.8.8.8
traceroute 8.8.8.8
mtr 8.8.8.8

# Check DNS
dig example.com
nslookup example.com
cat /etc/resolv.conf

# Check ports
netstat -tulpn
ss -tulpn
lsof -i :80

# Capture packets
tcpdump -i eth0 port 80
tcpdump -i eth0 host 192.168.1.1

Interview talking point: “ss -tulpn replaces netstat for listing sockets with process names, and the order matters: confirm the interface and IP first, then reachability, then DNS, then ports. tcpdump proves whether packets actually arrive.”

Q707: How do you troubleshoot network issues?

What it is: Network troubleshooting steps through layers: ping for reachability, nslookup and dig for DNS, ip route and traceroute for paths, netstat or ss for ports, and iptables or firewall-cmd for rules.

Why it is used: Isolating whether a fault is DNS, routing, ports, or firewall.

# Check connectivity
ping 8.8.8.8
ping google.com

# Check DNS
nslookup google.com
dig google.com

# Check routes
ip route
traceroute google.com

# Check ports
netstat -tulpn
ss -tulpn

# Check firewall
iptables -L -n
firewall-cmd --list-all

# Check DNS resolution
cat /etc/resolv.conf

Interview talking point: “ping 8.8.8.8 succeeding while ping google.com fails isolates DNS immediately, because raw IP reachability works and only name resolution breaks, and that split is the first branch in triage.”

Q708: How do you troubleshoot network issues?

What it is: Network troubleshooting walks the stack: ip addr and ip route for addressing and routing, ping for reachability, nslookup or dig for DNS, and netstat or ss for listening ports, plus firewall rules.

Why it is used: To localize a connectivity fault to interface, route, name resolution, port, or firewall.

# Check IP
ip addr show
ifconfig -a

# Check connectivity
ping -c 4 8.8.8.8
ping -c 4 google.com

# Check DNS
nslookup google.com
dig google.com

# Check routes
ip route
route -n

# Check ports
netstat -tulpn
ss -tulpn

# Check firewall
iptables -L -n
firewall-cmd --list-all

# Restart network
systemctl restart NetworkManager

Interview talking point: “ss -tulpn is the modern replacement for netstat -tulpn and both list sockets by protocol and state, so you can see what is actually listening. Testing 8.8.8.8 before a hostname separates raw connectivity from DNS failure.”

Q709: How do you troubleshoot network issues?

What it is: A network triage toolkit: ip link, addr, and route for interfaces and paths, getent and dig for DNS, ping, traceroute, and mtr for reachability, ss or netstat for ports, iptables for firewall rules, tcpdump for capture.

Why it is used: To isolate whether a problem is DNS, routing, firewall, or the service itself.

# Interface status
ip link
ip addr
ethtool eth0

# Routing
ip route
ip route get 8.8.8.8

# DNS
getent hosts example.com
dig +trace example.com

# Connectivity
ping -c 4 8.8.8.8
traceroute 8.8.8.8
mtr 8.8.8.8

# Ports
ss -tulpn
netstat -tulpn

# Firewall
iptables -L -n -v
firewall-cmd --list-all

# Traffic capture
tcpdump -i eth0 host 192.168.1.1
tcpdump -i eth0 port 80

Interview talking point: “ip route get 8.8.8.8 shows the exact route the kernel would pick, and ss -tulpn lists listening sockets with their owning processes, which netstat also reports but ss delivers faster.”

SSH Scenarios

Q710: How do you troubleshoot SSH issues?

What it is: SSH troubleshooting runs ssh -vvv for verbose negotiation, tails auth.log, tests the port with nc -zv host 22, and lists key files under .ssh.

Why it is used: To separate network blocks, wrong credentials and bad key permissions within minutes.

# Verbose SSH
ssh -vvv user@host

# Check logs
tail -f /var/log/auth.log

# Test connectivity
nc -zv host 22

# Check keys
ls -la ~/.ssh/

Interview talking point: “Triple verbose output shows which authentication methods the server offers and which the client tries, so it distinguishes a firewall problem from a rejected key quickly.”

Authentication Scenarios

Q711: How do you debug authentication issues?

What it is: Authentication debugging follows SSH debug output, PAM logs and tests, sudo rules, LDAP directory queries, and Kerberos ticket checks.

Why it is used: To find whether a login failure comes from SSH, PAM, directory lookup, or credentials.

# Check SSH logs
tail -f /var/log/auth.log
journalctl -u sshd -f

# Check PAM
tail -f /var/log/syslog | grep -i pam

# Test authentication
# SSH with debug
ssh -vvv user@host

# Test PAM
pamtester login username authenticate

# Check sudo
sudo -l
tail -f /var/log/auth.log | grep sudo

# LDAP issues
ldapsearch -x -D "cn=admin,dc=example,dc=com" -W
getent passwd username
id username

# Kerberos
kinit -f user@REALM
klist

Interview talking point: “ssh -vvv prints the full negotiation so you can see which auth methods the server actually offers, while getent passwd confirms the account is visible through NSS.”

Q712: How do you handle authentication failures?

What it is: Login troubleshooting: read journalctl and auth.log, confirm the account exists with getent and id, review sshd_config and PAM, debug with ssh -vvv, and for LDAP check connectivity and sssd.

Why it is used: To find whether the failure is the account, the server config, or the directory.

# Authentication troubleshooting
# 1. Check logs
journalctl -u sshd | tail -50
tail -f /var/log/auth.log

# 2. Verify user exists
getent passwd username
id username

# 3. Check SSH configuration
# /etc/ssh/sshd_config
# PasswordAuthentication yes
# PubkeyAuthentication yes
# AllowUsers username

# 4. Test authentication
# SSH with debug
ssh -vvv user@host

# 5. Reset password
passwd username

# 6. Check PAM
# /etc/pam.d/sshd

# 7. For LDAP
# Check connectivity
ldapsearch -x -D "cn=admin,dc=example,dc=com" -W

# Check sssd
sssd -i -d 10

Interview talking point: “ssh -vvv walks the negotiation step by step, showing whether the server rejects the key or never offers it. getent passwd resolves through NSS, so it also confirms LDAP or sssd users, not just local ones.”

Database Scenarios

Q713: How do you handle database failover?

What it is: Database failover detects failure with health checks, promotes the replica, updates DNS, and relies on application connection retries and circuit breakers while alerting on the event.

Why it is used: To restore database service quickly when the primary goes down.

# Database failover
# 1. Automatic detection
# Health checks

# 2. Failover process
# Promote replica
# Update DNS

# 3. Application handling
# Connection retry
# Circuit breakers

# 4. Monitoring
# Alert on failover

# 5. Testing
# Regular drills

Interview talking point: “Failover is not purely infrastructure work, without connection retry and circuit breakers in the application, a promoted replica still leaves clients holding dead connections until timeouts expire.”

Q714: How do you handle database failover?

What it is: Database failover is the process of switching from a failed primary database to a standby or replica.

Why it is used: It helps maintain availability when the primary database becomes unreachable or unhealthy.

# Automatic detection
# Health checks

# Failover process
# Promote replica
# Update DNS

# Application handling
# Connection retry
# Circuit breakers

# Monitoring
# Alert on failover

# Testing
# Regular drills

Interview talking point: “Health checks and DNS updates work together to redirect traffic, but applications need retry logic to handle connections during the transition.”

Boot and Kernel Scenarios

Q715: How do you handle kernel panic?

What it is: A kernel panic is a fatal internal error that halts the system. kdump captures a vmcore into /var/crash, and the crash tool analyzes it against vmlinux debug symbols.

Why it is used: To diagnose why a machine died instead of guessing after a reboot.

# Kernel panic response
# 1. Verify panic
# Check logs
dmesg | tail -100

# 2. Configure kdump
apt install kdump-tools

# 3. Analyze crash
# /var/crash/
crash /var/crash/202401011200/vmcore /usr/lib/debug/boot/vmlinux-$(uname -r)

# 4. Common causes
# - Hardware failure (RAM, disk)
# - Driver issues
# - OOM
# - Kernel bugs

# 5. Fixes
# Update kernel
# Disable problematic driver
# Add RAM
# Fix OOM settings

# 6. Prevention
# Monitor resources
# Keep kernel updated
# Use hardware from compatibility list

Interview talking point: “dmesg after reboot shows the panic only if logs persisted; real forensics come from the kdump vmcore. Causes split into hardware, driver, out-of-memory, and kernel bug, and each points to a different fix.”

Q716: How do you recover from boot failure?

What it is: A troubleshooting sequence covering previous boot logs, initramfs rebuild, GRUB reinstallation, and an fstab check to get a machine booting again.

Why it is used: To diagnose and repair a server that fails to reach a running system after a reboot.

# Check boot logs
journalctl -b -1
dmesg | grep -i error

# Rebuild initramfs
update-initramfs -u

# Reinstall GRUB
grub-install /dev/sda
grub-mkconfig -o /boot/grub/grub.cfg

# Check fstab
blkid
mount -a

# Emergency boot
# Add to GRUB: init=/bin/bash

Interview talking point: “journalctl -b -1 reads the previous boot, which is the fastest way to see what broke before the current attempt, and init=/bin/bash at the boot menu gives a root shell for offline repair.”

Q717: How do you troubleshoot boot issues in Linux?

What it is: Boot debugging reads journal logs from the current and previous boot, boots into emergency or rescue targets from the GRUB menu, and checks or repairs the root filesystem.

Why it is used: To recover a server that hangs or drops to an emergency prompt during startup.

# Check boot logs
journalctl -b -1  # Previous boot
journalctl -b --priority=err

# Emergency mode
# At GRUB menu, add 'systemd.unit=emergency.target'

# Recovery mode
# At GRUB menu, add 'systemd.unit=rescue.target'

# Check filesystem
fsck /dev/sda1
mount -o remount,rw /

Interview talking point: “journalctl -b -1 shows the previous boot, which is where the answer lives when a machine reboots after a crash. Emergency and rescue targets differ in how much of the system gets mounted and started.”

Q718: How do you troubleshoot boot issues?

What it is: Boot troubleshooting reads the current boot with journalctl -b and kernel messages with dmesg, checks failed units with systemctl —failed, and can drop to an emergency target from GRUB.

Why it is used: Finding the service or driver that stopped a machine from coming up normally.

# Check boot logs
journalctl -b
dmesg
dmesg | grep -i error

# Check failed services
systemctl --failed

# Boot options
# Edit GRUB and add: systemd.unit=emergency.target

Interview talking point: “journalctl -b covers this boot while dmesg covers kernel messages, so dmesg piped through grep -i error narrows driver faults, and systemctl —failed names the units that never came up.”

Q719: How do you troubleshoot boot issues?

What it is: Boot troubleshooting reads journalctl for the current and previous boot, edits a GRUB entry to reach a shell, checks fstab and block IDs, and can rebuild initramfs or reinstall GRUB.

Why it is used: To recover a server that will not boot after a kernel, initramfs, or bootloader change.

# Check logs
journalctl -b
journalctl -b -1
dmesg

# GRUB
# Press Shift during boot
# Edit boot entry, add: init=/bin/bash

# Mount root
mount -o remount,rw /

# Check fstab
cat /etc/fstab
blkid

# Rebuild initramfs
update-initramfs -u

# Reinstall GRUB
grub-install /dev/sda
grub-mkconfig -o /boot/grub/grub.cfg

Interview talking point: “journalctl -b -1 shows logs from the previous boot, which is how you compare against a change that broke startup. Remounting root read-write from the emergency shell lets you edit fstab or regenerate configs before booting normally again.”

Q720: How do you troubleshoot kernel panics?

What it is: After a panic, journalctl -b -1 reads the previous boot’s logs and dmesg the kernel ring buffer, while kdump captures a vmcore for the crash utility.

Why it is used: To learn why the kernel died and to keep a memory dump when logs alone are not enough.

# Check logs
journalctl -b -1
dmesg

# Enable crash dump
# /etc/default/kdump-tools
USE_KDUMP=1

# After crash
# Analyze with crash utility
crash /var/crash/vmcore /usr/lib/debug/boot/vmlinux-$(uname -r)

Interview talking point: “journalctl -b -1 means the boot before the current one, which is exactly where panic messages live after an automatic reboot.”

Data and System Recovery

Q721: How do you restore from backup?

What it is: Restoring reverses the backup: tar -xzf extracts an archive to a target with -C, rsync copies the backup tree back, and dd writes an image onto a device.

Why it is used: Recovering data after deletion, corruption, or hardware loss.

# Extract tar
tar -xzf backup.tar.gz -C /restore/path

# Rsync restore
rsync -avz /backup/ /restore/

# dd restore
dd if=/backup/sda.img of=/dev/sda

Interview talking point: “dd restore overwrites an entire device from an image, while tar and rsync recover selected files, which is why images suit total failures and file copies suit single-file mistakes.”

Q722: How do you design for data recovery?

What it is: Data recovery designed with multiple backups, point-in-time capability, regular testing, documented procedures, and team training.

Why it is used: Restores lost data quickly even when the original responders are unavailable.

# Data recovery
# 1. Backups
# Multiple

# 2. Point in time
# Capability

# 3. Testing
# Regular

# 4. Documentation
# Procedures

# 5. Team
# Training

Interview talking point: “Point-in-time capability matters for corruption noticed late, since a nightly backup can only rewind to the last snapshot. Multiple backups plus regular testing ensure at least one copy is both available and known to work.”

Q723: How do you design for data recovery?

What it is: Data recovery design involves maintaining multiple backups, point-in-time capabilities, regular testing, documented procedures, and a trained team.

Why it is used: It enables restoring systems quickly after failures or data loss.

# Backups
# Multiple

# Point in time
# Capability

# Testing
# Regular

# Documentation
# Procedures

# Team
# Training

Interview talking point: “Regular testing of recovery procedures is as critical as having backups, since untested backups can fail when needed most.”

Level 15: Advanced Linux and SRE

Namespaces and cgroups

Q724: What are Linux namespaces and how are they used?

What it is: Namespaces partition kernel resources so a process set sees its own PIDs, mounts, network, and users. unshare creates a new namespace and ip netns manages network namespaces.

Why it is used: To isolate containers and test networking without touching the host’s real interfaces.

# View namespaces
ls -la /proc/$$/ns/

# Create a new namespace (user namespace requires kernel 3.8+)
unshare --mount --pid --fork --user --map-root-user bash

# Network namespace
ip netns add mynet
ip netns exec mynet ip link list

Interview talking point: “unshare —user —map-root-user gives a root identity inside a user namespace without granting real host root. ip netns exec then runs commands inside an isolated network namespace, giving a second interface stack to experiment with.”

Q725: How do you configure namespace isolation?

What it is: Namespaces partition kernel views. ip netns creates a network namespace, while unshare starts a new user, PID, or mount namespace, with —fork and —mount-proc used for PID namespaces.

Why it is used: To isolate processes so they see their own network, PIDs, mounts, or user IDs, as containers do.

# Create network namespace
ip netns add myns

# Create user namespace
unshare --user

# Create PID namespace
unshare --pid --fork --mount-proc

# Mount namespace
unshare --mount

Interview talking point: “unshare —pid alone leaves the old process view, so —fork —mount-proc gives a fresh PID 1 and a new procfs. Each namespace type is created separately, which shows containers are simply several namespaces composed together.”

Q726: How do you configure network namespace?

What it is: A network namespace is an isolated stack of interfaces, routes, and addresses. ip netns creates and lists them, and ip netns exec runs a command inside one.

Why it is used: To isolate container or test traffic and to experiment without touching the host network.

# Create namespace
ip netns add red
ip netns add blue

# List namespaces
ip netns list

# Execute in namespace
ip netns exec red ip link
ip netns exec red ping 8.8.8.8

# Connect namespaces with veth
ip link add veth0 type veth peer name veth1
ip link set veth1 netns red
ip netns exec red ip link set veth1 name eth0

# Delete
ip netns delete red

Interview talking point: “A veth pair acts as a cable between namespaces because moving veth1 into red leaves veth0 on the host, and renaming it to eth0 inside the namespace changes only the label.”

Q727: How do you configure systemd network namespace?

What it is: ip netns manages network namespaces, isolated network stacks with their own interfaces, routes, and addresses. Commands create, enter, list, and delete them.

Why it is used: To isolate networking for testing, service sandboxing, or containers on a single host.

ip netns add myns
ip netns exec myns ip link
ip netns exec myns ping 8.8.8.8

# List
ip netns list

# Delete
ip netns delete myns

Interview talking point: “A namespace is created with ip netns add, then commands run inside it via ip netns exec myns, and ip netns list shows what exists. The exec form is what makes the namespace usable rather than just present.”

Q728: How do you use cgroups v2?

What it is: cgroups v2 is a unified control tree under /sys/fs/cgroup. stat -fc %T reports the filesystem type, and creating a directory makes a group whose files set limits and hold processes.

Why it is used: To cap CPU usage and confine groups of processes so one noisy workload cannot starve the others.

# Check cgroup version
stat -fc %T /sys/fs/cgroup/

# Create group
mkdir -p /sys/fs/cgroup/mygroup
echo 100000000 > /sys/fs/cgroup/mygroup/cpu.max

# Add process
echo PID > /sys/fs/cgroup/mygroup/cgroup.procs

Interview talking point: “Writing to cpu.max sets the quota, and echoing a PID into cgroup.procs moves a task into the group. cgroup v2 uses a single hierarchy, in contrast to v1 where each controller had its own mount.”

Q729: How do you use cgroups v2?

What it is: cgroups v2 hierarchically limit resources; you mkdir a group under /sys/fs/cgroup, write cpu.max and memory.max, then move a PID into cgroup.procs.

Why it is used: To cap CPU and memory per service so one workload cannot starve the host.

# Create cgroup
mkdir -p /sys/fs/cgroup/mygroup

# Set limits
echo 100000000 > /sys/fs/cgroup/mygroup/cpu.max
echo 1G > /sys/fs/cgroup/mygroup/memory.max

# Add process
echo PID > /sys/fs/cgroup/mygroup/cgroup.procs

# Make persistent
# /etc/cgconfig.conf
group mygroup {
    cpu {
        cpu.max = "100000000";
    }
    memory {
        memory.max = "1073741824";
    }
}

Interview talking point: “Limits are plain files like cpu.max and memory.max, and the cgconfig.conf group block shown makes the same settings survive a reboot instead of being recreated by hand.”

Q730: How do you use Docker user namespace remapping?

What it is: Docker user namespace remapping maps container root (UID 0) to an unprivileged host UID via daemon.json userns-remap, with subuid and subgid ranges assigned to a subordinate user like dockremap.

Why it is used: So root inside a container is not root on the host, limiting damage from a container breakout.

# Enable in daemon
{
    "userns-remap": "default"
}

# Or custom
{
    "userns-remap": "myuser"
}

# Create user
useradd -r -u 100000 -g dockremap dockremap

# Configure subuid/subgid
usermod -v 100000-165536 -w 100000-165536 dockremap

Interview talking point: “userns-remap set to default creates a dockremap user for you, but a custom entry needs useradd plus usermod subuid and subgid ranges, and image files owned by high UIDs can break after remapping.”

Q731: How do you use namespaces?

What it is: Namespaces give a process its own view of a resource; ip netns creates network namespaces and unshare creates user, PID, or mount namespaces on the fly.

Why it is used: To isolate containers or test networking and privileges without a second machine.

# Network namespace
ip netns add red
ip netns exec red ip link

# User namespace
unshare --user

# PID namespace
unshare --pid --fork --mount-proc

# Mount namespace
unshare --mount

# Combined
unshare --mount --user --pid --fork --mount-proc --map-root-user

Interview talking point: “unshare —pid needs —fork and —mount-proc for the new PID 1 to work, and combining flags with —map-root-user yields a container-like sandbox from a single command.”

Containers

Q732: How do you configure container orchestration?

What it is: Kubernetes clusters can be bootstrapped with kubeadm, applying a CNI plugin for pod networking.

Why it is used: Orchestrate containerized applications with automated scaling and self-healing.

# Kubernetes with kubeadm
kubeadm init --pod-network-cidr=10.244.0.0/16

# Install CNI (Flannel)
kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml

# Allow pods on master
kubectl taint nodes --all node-role.kubernetes.io/master-

# Deploy application
# deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: myapp
spec:
  replicas: 3
  selector:
    matchLabels:
      app: myapp
  template:
    metadata:
      labels:
        app: myapp
    spec:
      containers:
      - name: myapp
        image: myapp:latest
        ports:
        - containerPort: 8080

# Apply
kubectl apply -f deployment.yaml

# Scale
kubectl scale deployment myapp --replicas=5

Interview talking point: “A CNI plugin is required for pod-to-pod communication; tainting the control plane allows scheduling pods on master nodes if needed.”

Q733: How do you configure container registries?

What it is: Harbor runs as a container publishing ports 80 and 443 with data persisted on disk; images are pushed to the registry and pulled after docker login.

Why it is used: To host private images close to your cluster with credentials controlling push and pull access.

# Harbor
# Install
docker run -d -p 80:80 -p 443:443 -v /data/harbor:/etc/harbor --name harborVMWare/harbor

# Push
docker push myregistry.com/myimage:tag

# Pull with credentials
docker login myregistry.com

Interview talking point: “docker login stores credentials that later pulls reuse, and the tag chosen at push time is what a deployment references, so tagging matters more than the registry brand.”

Q734: How do you configure containerd?

What it is: containerd is a container runtime daemon. Its /etc/containerd/config.toml configures the CRI plugin, including the pause sandbox image and which runc runtime type, engine, and root to use.

Why it is used: To supply the runtime layer beneath Kubernetes or other orchestrators that talk to it over CRI.

# Install containerd
apt install containerd

# Configure /etc/containerd/config.toml
[plugins]
  [plugins."io.containerd.grpc.v1.cri"]
    sandbox_image = "registry.k8s.io/pause:3.9"
    [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc]
      runtime_type = "io.containerd.runtime.v1.linux"
      runtime_engine = ""
      runtime_root = ""

Interview talking point: “sandbox_image names the pause container every pod receives, so a mismatch with the cluster version is a classic breakage. containerd sits above runc: it manages images and containers while runc performs the actual process creation.”

Q735: How do you configure containerd?

What it is: containerd is a container runtime configured through /etc/containerd/config.toml. That file sets the CRI plugin, the pause sandbox image, and which runtime, here runc, handles containers.

Why it is used: To configure the runtime Kubernetes nodes use, including the sandbox image and runtime handler.

# Install
apt install containerd

# Configure /etc/containerd/config.toml
version = 2
[plugins]
  [plugins."io.containerd.grpc.v1.cri"]
    sandbox_image = "registry.k8s.io/pause:3.9"
    [plugins."io.containerd.grpc.v1.cri".containerd.runtimes]
      [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc]
        runtime_type = "io.containerd.runtime.v1.linux"

Interview talking point: “sandbox_image defines the pause container every pod gets, so changing it matters during Kubernetes upgrades, and runtime_type selects the runc handler used for each container.”

Q736: How do you configure Docker Compose?

What it is: Docker Compose declares services, volumes, and networks in one YAML file so a multi-container application starts together with build, ports, environment, and dependencies.

Why it is used: To reproduce an entire application stack with a single command across environments.

version: '3.8'
services:
  web:
    build: .
    ports:
      - "8080:80"
    environment:
      - NODE_ENV=production
    volumes:
      - ./data:/data
    depends_on:
      - db
    networks:
      - frontend
      - backend

  db:
    image: postgres:14
    volumes:
      - db-data:/var/lib/postgresql/data
    networks:
      - backend

volumes:
  db-data:

networks:
  frontend:
  backend:

Interview talking point: “depends_on controls start order but does not wait for readiness, so a database may still be initializing when the application connects. Named volumes declared under volumes outlive the project’s containers, while bind mounts use a host path.”

Q737: How do you configure Docker Compose?

What it is: Docker Compose declares services, ports, environment, volumes, dependencies, named volumes and networks in one docker-compose.yml so a stack starts together.

Why it is used: To reproduce a multi-container application locally or on a server with a single command.

version: '3.8'
services:
  web:
    build: .
    ports:
      - "8080:80"
    environment:
      - NODE_ENV=production
    volumes:
      - ./data:/data
    depends_on:
      - db
    networks:
      - frontend
      - backend
    restart: always

  db:
    image: postgres:14
    environment:
      POSTGRES_PASSWORD: secret
    volumes:
      - db-data:/var/lib/postgresql/data
    networks:
      - backend

  redis:
    image: redis:alpine
    networks:
      - backend

volumes:
  db-data:

networks:
  frontend:
  backend:

Interview talking point: “depends_on only controls startup order, not readiness, so the web service can launch before Postgres accepts connections, while restart: always keeps crashed containers coming back on their own.”

Q738: How do you configure Docker healthchecks?

What it is: A HEALTHCHECK instruction tells Docker how to probe a running container. Interval, timeout, start-period, and retries set how often and how forgiving the check is, here curl against localhost.

Why it is used: So Docker can flag a container that is running but not actually serving.

HEALTHCHECK --interval=30s --timeout=3s --start-period=5s --retries=3 \
  CMD curl -f http://localhost/ || exit 1

Interview talking point: “start-period of 5s gives the process time to boot before failures count, and once retries are exhausted the container is marked unhealthy rather than restarted automatically.”

Q739: How do you configure Docker networking?

What it is: Docker networks are created with drivers: bridge for isolated container traffic, host using the host network stack directly, overlay across Swarm nodes, plus per-run DNS.

Why it is used: To control how containers reach each other and the outside network.

# Create network
docker network create mynetwork

# Run with network
docker run -d --network mynetwork --name web nginx

# Bridge network
docker network create --driver bridge mybridge

# Host network
docker run --network host nginx

# Overlay (Swarm)
docker network create --driver overlay myoverlay

# DNS resolution
docker run -d --network mynetwork --dns 8.8.8.8 nginx

Interview talking point: “Containers on a user-defined bridge resolve each other by container name, while host mode removes that isolation and lets the container bind host ports directly.”

Q740: How do you configure Docker networking?

What it is: Docker networks come in bridge drivers for a single host and overlay drivers for clusters, with containers reachable by name and ports mapped to the host.

Why it is used: To give containers isolated connectivity, service discovery by name, and controlled host port access.

# List networks
docker network ls
docker network inspect bridge

# Create network
docker network create --driver bridge mynetwork
docker network create --driver overlay myoverlay

# Connect container
docker run -d --network mynetwork --name web nginx

# Port mapping
docker run -d -p 8080:80 --name web2 nginx

# DNS resolution
docker run -d --network-alias db --network mynetwork mysql

Interview talking point: “Port mapping with -p 8080:80 exposes a container to the host, while containers on the same user-defined network reach each other by name and alias without any published ports.”

Q741: How do you configure Docker networking?

What it is: Docker networking modes: custom bridge networks with chosen subnets, overlay for swarm, host networking, macvlan attached to a physical parent, plus DNS and port mapping.

Why it is used: To give containers the right isolation and reachability for their role.

# Custom bridge network
docker network create --driver bridge mynetwork
docker network create --subnet=192.168.100.0/24 mynetwork

# Overlay network
docker network create --driver overlay myoverlay

# Host network
docker run --network host nginx

# Macvlan
docker network create -d macvlan \
  --subnet=192.168.1.0/24 \
  --gateway=192.168.1.1 \
  -o parent=eth0 mymacvlan

# DNS configuration
docker run --dns 8.8.8.8 --network-alias myapp myimage

# Port mapping
docker run -p 8080:80 nginx

# Connect to network
docker network connect mynetwork container

Interview talking point: “Containers on a user-defined bridge resolve each other by name automatically, while host network mode removes port mapping entirely and shares the host network stack.”

Q742: How do you configure Docker networking?

What it is: Docker networking creates bridge, overlay or macvlan networks, maps ports with -p, connects containers, and supports host networking, custom DNS and network aliases.

Why it is used: To control how containers reach each other and the outside world.

# Create custom network
docker network create --driver bridge mynetwork
docker network create --driver overlay myoverlay

# Network inspection
docker network inspect bridge

# Connect container
docker network connect mynetwork container

# Port mapping
docker run -d -p 8080:80 nginx

# Host network
docker run --network host nginx

# DNS configuration
docker run --dns 8.8.8.8 nginx
docker run --network-alias db mysql

# Macvlan
docker network create -d macvlan \
  --subnet=192.168.1.0/24 \
  --gateway=192.168.1.1 \
  -o parent=eth0 mymacvlan

Interview talking point: “-p 8080:80 publishes a container port on the host, whereas —network host drops the container onto the host network namespace so no port mapping or NAT is involved at all.”

Q743: How do you configure Docker on cloud?

What it is: Installing Docker with the get.docker.com script, then editing daemon.json to set the overlay2 storage driver, capped json-file logs, live-restore, and address pools, with systemd enabling the service.

Why it is used: To prepare a cloud VM for containers with sane log and storage defaults.

# Install Docker
curl -fsSL https://get.docker.com | sh

# Configure Docker daemon
# /etc/docker/daemon.json
{
  "storage-driver": "overlay2",
  "log-driver": "json-file",
  "log-opts": {
    "max-size": "10m",
    "max-file": "3"
  },
  "live-restore": true,
  "default-address-pools": [
    {"base": "172.17.0.0/16", "size": 24}
  ]
}

# Enable Docker service
systemctl enable docker
systemctl start docker

# Docker swarm (for multi-host)
docker swarm init
docker node ls

Interview talking point: “The log-opts max-size and max-file settings cap container log growth, which otherwise fills the root filesystem, and live-restore keeps containers running across a daemon restart.”

Q744: How do you configure Docker storage driver?

What it is: The storage-driver key in daemon.json chooses how image and container layers are stored on disk; this config selects overlay2 and sets an option to override the kernel compatibility check.

Why it is used: To pick the layer implementation and keep it working when the kernel check disagrees.

{
    "storage-driver": "overlay2",
    "storage-opts": [
        "overlay2.override_kernel_check=true"
    ]
}

Interview talking point: “The override_kernel_check option forces overlay2 even if the kernel check fails, which is the knob to remember when a host refuses the driver you asked for.”

Q745: How do you configure Docker storage?

What it is: Docker storage choices: named volumes, host bind mounts, tmpfs memory-only mounts, and NFS-backed volumes created with a local driver and mount options.

Why it is used: To persist data beyond container lifetime or share existing host paths.

# Create volume
docker volume create mydata

# Mount volume
docker run -v mydata:/data mysql

# Bind mount
docker run -v /host/path:/container/path nginx

# tmpfs mount
docker run --tmpfs /tmp nginx

# NFS volume
docker volume create --driver local \
  --opt type=nfs \
  --opt o=addr=192.168.1.100,rw \
  --opt device=:/path/to/share \
  nfsvolume

# Backup volume
docker run --rm -v mydata:/data -v $(pwd):/backup alpine \
  tar cvf /backup/backup.tar /data

Interview talking point: “A named volume is managed by Docker and portable, whereas a bind mount ties the container to a fixed host path, and tmpfs leaves nothing on disk after exit.”

Q746: How do you configure Docker Swarm services?

What it is: Docker Swarm services run replicated tasks with declared update and rollback policies, scaling, overlay networks, and secrets mounted from the swarm secret store.

Why it is used: To deploy and roll out containerized apps across a cluster with automatic recovery.

# Initialize swarm
docker swarm init --advertise-addr 192.168.1.10

# Create service
docker service create \
  --name myapp \
  --replicas 3 \
  --publish 8080:80 \
  --update-delay 10s \
  --update-parallelism 1 \
  --update-failure-action rollback \
  --rollback-monitor 5s \
  --rollback-max-failure-ratio 0.1 \
  myimage:latest

# Update service
docker service update --image myimage:v2 myapp

# Scale
docker service scale myapp=5

# Networks
docker network create -d overlay myoverlay
docker service update --network-add myoverlay myapp

# Secrets
echo "mypassword" | docker secret create mysecret -
docker secret create mysecret mysecret.txt
docker service update --secret-add mysecret myapp

Interview talking point: “update-parallelism 1 with a 10s delay changes one task at a time, and update-failure-action rollback with a 0.1 failure ratio auto-reverts if too many updates fail.”

Q747: How do you configure Docker Swarm?

What it is: Docker Swarm creates a cluster where manager nodes schedule services across worker nodes.

Why it is used: Scale containerized services with load balancing and rolling updates.

# Initialize swarm
docker swarm init --advertise-addr 192.168.1.10

# Join nodes
docker swarm join --token SWMTKN-1-xxx 192.168.1.10:2377

# Create service
docker service create \
  --name myapp \
  --replicas 3 \
  --publish 8080:80 \
  myapp:latest

# Scale service
docker service scale myapp=5

# Update service
docker service update \
  --image myapp:v2 \
  myapp

# Stack deploy
docker stack deploy -c docker-compose.yml myapp

# Visualize
docker node ls
docker service ls
docker service ps myapp

Interview talking point: “Services define desired state and replicas; swarm join tokens securely add new manager or worker nodes to the cluster.”

Q748: How do you configure LXC containers?

What it is: LXC creates system containers sharing the host kernel. lxc-create makes one from a template, lxc-start and lxc-attach run and enter it, and the config file sets rootfs, name, and a veth network link.

Why it is used: To run lightweight isolated environments with faster startup than full virtual machines.

# Install
apt install lxc

# Create container
lxc-create -n mycontainer -t ubuntu

# Start container
lxc-start -n mycontainer
lxc-attach -n mycontainer

# Configuration
# /var/lib/lxc/mycontainer/config
lxc.include = /usr/share/lxc/config/ubuntu.common.conf
lxc.arch = amd64
lxc.rootfs.path = dir:/var/lib/lxc/mycontainer/rootfs
lxc.uts.name = mycontainer
lxc.network.type = veth
lxc.network.link = lxcbr0
lxc.network.hwaddr = 00:16:3e:xx:xx:xx

# Clone
lxc-copy -n mycontainer -N mycontainer2

# Snapshots
lxc-snapshot -n mycontainer
lxc-snapshot -n mycontainer -L

Interview talking point: “Because containers share the host kernel, the config sets architecture and network type rather than emulated hardware, and lxc-copy plus lxc-snapshot give cloning and rollback without rebuilding from a template.”

Q749: How do you configure LXC?

What it is: LXC containers are created with lxc-create from a template, started with lxc-start, entered with lxc-attach, and configured in a plain config file under /var/lib/lxc.

Why it is used: To run lightweight system containers that share the host kernel with near-native speed.

# Install LXC
apt install lxc

# Create container
lxc-create -n mycontainer -t ubuntu

# Start container
lxc-start -n mycontainer
lxc-attach -n mycontainer

# Configuration
# /var/lib/lxc/mycontainer/config
lxc.include = /usr/share/lxc/config/ubuntu.common.conf
lxc.uts.name = mycontainer
lxc.network.type = veth
lxc.network.link = lxcbr0

# Clone container
lxc-copy -n mycontainer -N mycontainer2

# Snapshot
lxc-snapshot -n mycontainer

# Manage
lxc-ls -f
lxc-info -n mycontainer
lxc-stop -n mycontainer
lxc-destroy -n mycontainer

Interview talking point: “lxc-attach gives you a shell inside a running container, and unlike Docker you set networking by editing lxc.network.type veth and lxc.network.link directly in the config file.”

Q750: How do you create custom Docker images?

What it is: A Dockerfile builds an image from a base, sets labels and environment, installs packages, creates a non-root user, copies files, exposes ports, health-checks, and sets entrypoint.

Why it is used: To package an application with its dependencies into a reproducible image.

# Dockerfile
FROM ubuntu:20.04

# Labels
LABEL maintainer="admin@example.com"
LABEL version="1.0"

# Environment
ENV APP_HOME=/opt/app
ENV NODE_ENV=production

# Install dependencies
RUN apt-get update && \
    apt-get install -y curl nginx && \
    apt-get clean && \
    rm -rf /var/lib/apt/lists/*

# Create user
RUN useradd -m -s /bin/bash appuser

# Copy files
COPY --chown=appuser:appuser . /opt/app

# Switch user
USER appuser

# Expose
EXPOSE 80 443

# Health check
HEALTHCHECK --interval=30s --timeout=3s \
    CMD curl -f http://localhost/ || exit 1

# Entrypoint
ENTRYPOINT ["/opt/app/start.sh"]
CMD ["nginx", "-g", "daemon off;"]

Interview talking point: “ENTRYPOINT fixes the executable while CMD supplies default arguments a docker run can override, and HEALTHCHECK lets an orchestrator detect and replace a broken container.”