Linux_Practical_Interview_1001 1250

Linux Practical Interview Questions (1001-1250)

Level 15: Advanced Linux and SRE

Kubernetes

Q1001: How do you secure the cluster?

What it is: Securing a Kubernetes cluster involves hardening control plane, nodes, and network layers.

Why it is used: It protects all workloads from cluster-wide compromise and external threats.

# Use RBAC
# Network policies

# Encrypt etcd
# Enable audit logging

# Update regularly
# Use CIS benchmarks

# Monitor
# Use Falco

# Use secrets management

Interview talking point: “Key controls include RBAC, NetworkPolicies, etcd encryption, audit logging, CIS benchmark adherence, runtime monitoring with Falco, and secrets management.”

Q1002: How do you secure the container registry?

What it is: Securing a container registry involves controlling access and verifying image integrity.

Why it is used: It ensures only trusted, scanned images are pulled into production environments.

# Use authentication
# Use TLS
# Scan images
# Use signed images

# Access control
# Role-based

# Regular cleanup
# Remove old images

# Use private registry
# Harbor, Nexus

Interview talking point: “Using authentication, TLS, RBAC, image signing, and regular cleanup helps prevent untrusted images from entering the supply chain.”

Q1003: How do you secure the control plane?

What it is: Securing the control plane involves isolating it on a separate network, enabling RBAC and TLS, encrypting etcd, limiting access, using admission controllers, and enabling audit logging.

Why it is used: The control plane manages the entire cluster, so hardening it is essential to prevent unauthorized cluster-wide control.

# Use separate networks
# Enable RBAC
# Use TLS everywhere
# Encrypt etcd
# Limit etcd access
# Use admission controllers
# Enable audit logging

# Network policies
# Isolate control plane from workloads

Interview talking point: “Isolating the control plane from workload networks with network policies creates a strong boundary that limits the impact of compromised workloads.”

Q1004: How do you secure the control plane?

What it is: Securing the control plane involves using RBAC, limiting access, enabling audit logging, using TLS, isolating the control plane network, avoiding public exposure, and applying patches regularly.

Why it is used: Since the control plane governs the entire cluster, its compromise would affect all workloads and data.

# Use RBAC
# Limit access
# Enable audit logging
# Use TLS

# Network isolation
# Use separate network
# Don't expose publicly

# Updates
# Patch quickly
# Regular updates

Interview talking point: “Keeping the control plane on a separate, non-routable network from workloads is a key boundary that limits exposure to potential attacks.”

Q1005: How do you secure the control plane?

What it is: Securing the Kubernetes control plane means protecting its components and access paths.

Why it is used: It prevents attackers from taking control of the entire cluster.

# Use RBAC
# Limit access
# Enable audit logging
# Use TLS

# Network isolation
# Separate network

# Updates
# Patch quickly

Interview talking point: “RBAC, TLS encryption, audit logging, network isolation of control plane components, and prompt patching are key to hardening it.”

Q1006: How do you secure the data plane?

What it is: Securing the data plane involves encrypting traffic with mTLS, using NetworkPolicies, implementing service mesh, container isolation with gVisor or Kata Containers, user namespaces, and runtime security tools like Falco or Sysdig.

Why it is used: Data plane hardening protects pod-to-pod communication and prevents compromised workloads from affecting others.

# Use encryption
# mTLS
# Network policies
# Service mesh

# Container isolation
# gVisor, Kata
# User namespaces

# Runtime security
# Falco
# Sysdig

Interview talking point: “Combining service mesh mTLS with sandboxed runtimes provides strong encryption and isolation, though this adds complexity to operations.”

Q1007: How do you secure the data plane?

What it is: Securing the data plane means protecting runtime traffic and container isolation.

Why it is used: It prevents lateral movement and limits the blast radius of compromised workloads.

# Use encryption
# mTLS
# Network policies
# Service mesh

# Container isolation
# gVisor, Kata
# User namespaces

# Runtime security
# Falco

Interview talking point: “mTLS, NetworkPolicies, service mesh, strong container isolation (like gVisor or Kata), user namespaces, and runtime security tools harden the data plane.”

Q1008: How do you secure the registry?

What it is: Securing a container registry involves authentication, encryption, scanning, and access control.

Why it is used: It ensures only verified, trusted images are distributed and deployed.

# Use authentication
# Use TLS
# Scan images
# Use signed images

# Access control
# Role-based

# Regular cleanup

# Use private registry
# Harbor, Nexus

Interview talking point: “TLS, RBAC, vulnerability scanning, image signing, regular cleanup, and using private registries help maintain registry integrity.”

Q1009: How do you secure worker nodes?

What it is: Securing worker nodes involves using minimal OS images, applying security patches, following CIS benchmarks, enabling firewalls, restricting access, and hardening with SELinux or AppArmor.

Why it is used: Compromised worker nodes can access all pods and data on them, making node hardening critical for cluster security.

# Use minimal OS
# Apply security patches
# Use CIS benchmarks
# Enable firewall
# Restrict access

# Node hardening
# Disable unnecessary services
# Use SELinux/AppArmor
# Regular updates

Interview talking point: “Disabling unnecessary services and applying regular updates reduces the attack surface while preventing exploitation of known vulnerabilities.”

Q1010: How do you secure worker nodes?

What it is: Securing worker nodes involves hardening the underlying OS and Kubernetes components.

Why it is used: It prevents nodes from becoming attack vectors for the entire cluster.

# Use minimal OS
# Apply patches

# CIS benchmarks
# Enable firewall

# Use SELinux
# Restrict access

# Regular updates
# Automated patching

Interview talking point: “Minimal OS, regular patching, CIS benchmarks, firewalls, and SELinux/AppArmor help reduce node-level vulnerabilities.”

Q1011: How do you secure workloads?

What it is: Securing Kubernetes workloads involves applying Pod Security Standards and runtime constraints.

Why it is used: It prevents privilege escalation and limits the blast radius of compromised pods.

# Pod Security Standards
# Restricted profile

# Use security contexts
securityContext:
  runAsNonRoot: true
  readOnlyRootFilesystem: true

# Use NetworkPolicies
# Default deny

# Use Secrets
# External secrets

Interview talking point: “Using a restricted security context with read-only rootfs, non-root users, and NetworkPolicies is key to hardening workloads.”

Q1012: How do you secure workloads?

What it is: Securing Kubernetes workloads involves enforcing Pod Security Standards and runtime constraints.

Why it is used: It prevents privilege escalation and reduces the impact of compromised pods.

# Pod Security Standards
# Restricted profile

# Use security contexts
securityContext:
  runAsNonRoot: true
  readOnlyRootFilesystem: true

# Use NetworkPolicies

# Use Secrets
# External secrets

Interview talking point: “Using restricted profiles, security contexts (runAsNonRoot, readOnlyRootFilesystem), NetworkPolicies, and external secrets helps harden workloads.”

Q1013: How do you use Aqua Security?

What it is: Aqua Security is deployed by applying its manifest; the CLI reports the CVE library version, and labeling a namespace with aqua.security enforce turns on enforcement there.

Why it is used: To scan images and enforce admission rules so only approved workloads run in production.

# Install
kubectl apply -f https://download.aquasec.com/aquasec/aquasec.yaml

# Scan
aqua cvelib --version

# Enforce
# Add labels to namespace
kubectl label namespace production aqua.security=enforce

Interview talking point: “Enforcement is scoped by namespace label: kubectl label namespace production aqua.security=enforce opts only that namespace into blocking, so policies can be rolled out gradually across environments.”

Q1014: How do you use Kubernetes pod security admission?

What it is: Pod security settings live in securityContext: run as non-root user 1000, no privilege escalation, read-only root filesystem, and every capability dropped for the container.

Why it is used: To apply restricted pod security standards directly in workload manifests.

# Pod
securityContext:
  runAsNonRoot: true
  runAsUser: 1000
  fsGroup: 2000
  seccompProfile:
    type: RuntimeDefault

# Container
securityContext:
  allowPrivilegeEscalation: false
  readOnlyRootFilesystem: true
  capabilities:
    drop:
    - ALL

Interview talking point: “These settings express the restricted profile directly in the manifest instead of a separate policy object, and readOnlyRootFilesystem true forces writes onto mounted volumes or memory.”

Q1015: How do you use Kubernetes pod security policy?

What it is: A PodSecurityPolicy under policy/v1beta1 restricts pods: no privileged containers, run as non-root required, and only secret and configMap volumes permitted.

Why it is used: To block workloads from requesting privileged settings or mounting risky volumes.

apiVersion: policy/v1beta1
kind: PodSecurityPolicy
metadata:
  name: restricted
spec:
  privileged: false
  seLinux:
    rule: RunAsAny
  runAsUser:
    rule: MustRunAsNonRoot
  fsGroup:
    rule: RunAsAny
  volumes:
  - 'secret'
  - 'configMap'

Interview talking point: “Combining privileged false with MustRunAsNonRoot and a short volume allow-list closed off both privilege escalation and host path mounts in one policy.”

Q1016: How do you use Kubernetes secrets encryption?

What it is: Secrets encryption configures the API server with an encryption provider file so Secret objects are encrypted at rest in etcd, here using AES-GCM.

Why it is used: To protect secret data in the datastore from anyone who reads etcd directly.

apiVersion: v1
kind: Secret
metadata:
  name: aes-key
data: <base64-aes-key>

---
# api-server
--encryption-provider-config=encryption-config.yaml

# At rest
# Secrets will be encrypted with AES-GCM

Interview talking point: “The encryption-provider-config flag on the API server activates this, and it protects data at rest only, since authorized clients still receive decrypted secrets.”

Resource Limits and Kernel Tuning

Q1017: Explain Linux virtual memory management.

What it is: Linux gives each process a virtual address space backed by demand paging, so pages load from swap or disk only when touched. /proc/meminfo, free, and vmstat show system memory state.

Why it is used: To diagnose memory pressure, swap churn, and which process holds how much memory.

# View memory info
cat /proc/meminfo
free -h
vmstat 1

# View process memory maps
pmap -x 1234
cat /proc/1234/maps

# Memory zones
cat /proc/buddyinfo

Interview talking point: “free -h summarizes whole-system memory while pmap -x and /proc/PID/maps show one process’s mappings, so a host that looks fine overall can still have one bloated process consuming everything.”

Q1018: How do you modify kernel parameters?

What it is: sysctl reads and changes kernel parameters at runtime, such as net.ipv4.tcp_fin_timeout. A one-off change uses sysctl -w, while appending to /etc/sysctl.conf and running sysctl -p makes it persistent.

Why it is used: Tuning kernel and network behavior without rebuilding or rebooting the kernel.

# Temporarily
sysctl -w net.ipv4.tcp_fin_timeout=30

# Persistently
echo "net.ipv4.tcp_fin_timeout=30" >> /etc/sysctl.conf
sysctl -p

# View current
sysctl -a
sysctl net.ipv4.tcp_fin_timeout

Interview talking point: “sysctl -w lasts only until reboot, so writing the same line into /etc/sysctl.conf and running sysctl -p survives restarts; sysctl -a lists every parameter while a single name shows one value.”

Q1019: How do you configure CPU governor?

What it is: The CPU frequency governor controls how the kernel scales clock speed. scaling_available_governors lists the choices, writing performance to each scaling_governor sets it, and cpufrequtils persists GOVERNOR=“performance” in its config.

Why it is used: To favor maximum CPU performance on servers, or to save power on laptops and idle hosts.

# Check available
cat /sys/devices/system/cpu/cpu0/cpufreq/scaling_available_governors

# Set for all
for cpu in /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor; do
    echo performance > $cpu
done

# Install cpufrequtils
apt install cpufrequtils

# /etc/default/cpufrequtils
GOVERNOR="performance"

# With systemd
systemctl enable cpufreqd

Interview talking point: “The loop writes to every cpu*/cpufreq/scaling_governor because each core has its own setting, so changing cpu0 alone would leave the rest untouched. The cpufrequtils config or cpufreqd service is what makes the choice survive reboots.”

Q1020: How do you configure huge pages for database?

What it is: Reserved huge pages are set with vm.nr_hugepages and counted in /proc/meminfo, while PostgreSQL opts in with shared_huge_pages.

Why it is used: To give a database large contiguous pages and cut TLB misses.

# Check
cat /proc/meminfo | grep Huge

# Set
sysctl -w vm.nr_hugepages=128

# Persistent
# /etc/sysctl.conf
vm.nr_hugepages = 128

# For PostgreSQL
# postgresql.conf
shared_huge_pages = on

Interview talking point: “vm.nr_hugepages reserves the pages and persists in /etc/sysctl.conf, and the database setting then points its shared buffer at those reserved pages.”

Q1021: How do you configure kernel parameters at boot?

What it is: Kernel boot-time parameters are options passed to the kernel via the bootloader and stored in the kernel command line.

Why it is used: Set boot behavior like device naming or crashkernel without rebuilding the kernel.

# Edit GRUB config
vim /etc/default/grub

# Add to GRUB_CMDLINE_LINUX
GRUB_CMDLINE_LINUX="net.ifnames=0 biosdevname=0 crashkernel=auto"

# Regenerate GRUB
update-grub  # Debian/Ubuntu
grub2-mkconfig -o /boot/grub2/grub.cfg  # RHEL/CentOS

# View current parameters
cat /proc/cmdline

# Temporary changes (current session only)
sysctl -w parameter=value

# Permanent changes
# /etc/sysctl.conf or /etc/sysctl.d/99-custom.conf

Interview talking point: “GRUB edits the cmdline via GRUB_CMDLINE_LINUX; changes require regenerating GRUB, while sysctl affects only the running kernel.”

Q1022: How do you configure system limits?

What it is: /etc/security/limits.conf sets per-user resource ceilings such as open files and processes, with soft and hard variants plus a root entry. ulimit -n raises nofile for the current shell.

Why it is used: To raise file descriptor or process ceilings so busy services do not hit default limits.

* soft nofile 65535
* hard nofile 65535
* soft nproc 4096
* hard nproc 8192
root soft nofile unlimited

# Apply without logout
ulimit -n 65535

Interview talking point: “A soft limit a user may raise up to the hard limit, which only a privileged change can move. ulimit -n 65535 affects only the current session, so limits.conf is what governs future logins.”

Q1023: How do you configure transparent huge pages?

What it is: Transparent huge pages let the kernel back memory with large pages, and the enabled and defrag files accept always, madvise, or never.

Why it is used: To stop automatic huge page allocation and defragmentation from causing latency spikes.

# Check current
cat /sys/kernel/mm/transparent_hugepage/enabled

# Disable
echo never > /sys/kernel/mm/transparent_hugepage/enabled
echo never > /sys/kernel/mm/transparent_hugepage/defrag

# Make persistent
# /etc/rc.d/rc.local
if test -f /sys/kernel/mm/transparent_hugepage/enabled; then
    echo never > /sys/kernel/mm/transparent_hugepage/enabled
fi

Interview talking point: “Both enabled and defrag must be set to never, and since no sysctl covers this, the rc.local snippet shown reapplies it after every reboot.”

Q1024: How do you use BBR congestion control?

What it is: BBR is a TCP congestion control algorithm; tcp_bbr is its module, loaded and selected via sysctl with fq as the queueing discipline.

Why it is used: To get better throughput on lossy or high-latency links than loss-based algorithms deliver.

# Check
sysctl net.ipv4.tcp_available_congestion_control

# Enable
modprobe tcp_bbr

# Configure
sysctl -w net.core.default_qdisc=fq
sysctl -w net.ipv4.tcp_congestion_control=bbr

# Make persistent
echo "tcp_bbr" >> /etc/modules-load.d/modules.conf

Interview talking point: “Check net.ipv4.tcp_available_congestion_control first, then load tcp_bbr and set both net.core.default_qdisc=fq and net.ipv4.tcp_congestion_control=bbr, because BBR expects fq, and the modules-load.d entry keeps the module loaded after reboot.”

Q1025: How do you use chcpu?

What it is: chcpu enables or disables logical CPUs; -e and -d toggle a CPU and -l lists their current state.

Why it is used: To take a CPU offline or bring it back without rebooting.

# Enable CPU
chcpu -e 1

# Disable CPU
chcpu -d 1

# List
chcpu -l

Interview talking point: “chcpu changes CPU state directly, so after chcpu -d 1 the processor is out of scheduling until chcpu -e 1 brings it back.”

Q1026: How do you use early page reclaim?

What it is: vm.pagecache controls early page cache reclaim; sysctl -w vm.pagecache=1 turns it on and /etc/sysctl.conf makes it persistent.

Why it is used: To reclaim cached pages sooner and keep more free memory available.

# Check
cat /proc/sys/vm/pagecache

# Configure
sysctl -w vm.pagecache=1

# Make persistent
# /etc/sysctl.conf
vm.pagecache = 1

Interview talking point: “The setting is a plain sysctl, so reading /proc/sys/vm/pagecache shows the current value and the sysctl.conf entry applies it again at every boot.”

Q1027: How do you use huge pages?

What it is: Huge pages are 2 MB mappings checked in /proc/meminfo and reserved by writing nr_hugepages, then used via MAP_HUGETLB or Java large page flags.

Why it is used: To cut TLB misses and speed up memory heavy applications.

# Check
cat /proc/meminfo | grep Huge

# Configure
echo 256 > /sys/kernel/mm/hugepages/hugepages-2048kB/nr_hugepages

# Use in application
# mmap with MAP_HUGETLB flag

# For Java
# -XX:+UseLargePages

Interview talking point: “Reserving 256 pages fixes how many are available, and applications must still opt in, since unused reservations simply sit idle.”

Q1028: How do you use ktune?

What it is: ktune runs a startup script, here writing performance into every CPU scaling_governor file, and is enabled as a systemd unit.

Why it is used: To apply custom boot-time performance settings without hand-editing sysctl files.

# Install
apt install ktune

# Configure
# /etc/ktune/ktune.sh
#!/bin/bash
for cpu in /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor; do
    echo performance > $cpu
done

# Enable
systemctl enable ktune
systemctl start ktune

Interview talking point: “The ktune.sh loop sets each cpu*/cpufreq/scaling_governor to performance, achieving the same core pinning as tuned but driven by your own shell logic.”

Q1029: How do you use KVM huge pages?

What it is: Huge pages pre-allocate 2MB pages through nr_hugepages in sysfs, and a VM’s XML requests them with a hugepages block alongside its memory size.

Why it is used: Cuts page table overhead and TLB misses for memory-hungry guests.

# Enable huge pages
echo 1024 > /sys/kernel/mm/hugepages/hugepages-2048kB/nr_hugepages

# VM config
# Memory
<memory unit='KiB'>4194304</memory>
<currentMemory unit='KiB'>4194304</currentMemory>
<hugepages>
    <page size='2048' unit='KiB'/>
</hugepages>

Interview talking point: “Reserving nr_hugepages guarantees contiguous pages before the guest starts, and asking for hugepages without that reservation is the classic reason a VM silently falls back to ordinary 4KB pages with none of the benefit.”

Q1030: How do you use zram?

What it is: zram creates a compressed RAM block device; you set comp_algorithm and disksize, run mkswap, then swapon with a priority.

Why it is used: To add fast swap backed by RAM on machines with no spare disk.

# Load module
modprobe zram num_devices=2

# Configure
echo lz4 > /sys/block/zram0/comp_algorithm
echo 1G > /sys/block/zram0/disksize

# Format and mount
mkswap /dev/zram0
swapon /dev/zram0 -p 100

# Multiple devices
echo 512M > /dev/zram1/disksize
mkswap /dev/zram1
swapon /dev/zram1

Interview talking point: “swapon /dev/zram0 -p 100 gives it higher priority than disk swap so it is used first, and num_devices=2 creates two zram devices at load.”

Q1031: How do you use zswap?

What it is: zswap is a compressed write-back cache in front of swap; the module exposes an enabled parameter plus compressor and zpool options.

Why it is used: To compress pages before they reach disk and slow down swap traffic.

# Enable
modprobe zswap

# Check
cat /sys/module/zswap/parameters/enabled

# Compressor
# /etc/modprobe.d/zswap.conf
options zswap compressor=lz4

# Pool
# /etc/modprobe.d/zswap.conf
options zswap zpool=z3fold

Interview talking point: “The compressor such as lz4 and the zpool such as z3fold are set in /etc/modprobe.d/zswap.conf options lines, and reading the enabled parameter confirms state after modprobe.”

Q1032: How do you configure kernel tuning for low latency?

What it is: sysctl settings in /etc/sysctl.conf raise network buffer limits and shrink scheduler latency and granularity values, applied with sysctl -p.

Why it is used: To reduce scheduling delay and buffer pressure on latency sensitive hosts.

# Network
net.core.rmem_max = 134217728
net.core.wmem_max = 134217728

# Kernel
kernel.sched_latency_ns = 10000000
kernel.sched_min_granularity_ns = 1000000
kernel.sched_wakeup_granularity_ns = 1000000

# Apply
sysctl -p

Interview talking point: “sysctl -p reloads the file, so tuning persists across reboots instead of living only at runtime, and the sched values lower the target time slices.”

Q1033: How do you optimize system limits?

What it is: /etc/security/limits.conf sets per-login soft and hard limits for open files and processes, while fs.file-max is the kernel-wide file descriptor ceiling. ulimit applies a limit to the current shell.

Why it is used: To prevent a service from exhausting file descriptors or spawning too many processes.

* soft nofile 65535
* hard nofile 65535
* soft nproc 4096
* hard nproc 8192
root soft nofile unlimited

# Kernel limits
echo 65535 > /proc/sys/fs/file-max

# Apply without logout
ulimit -n 65535
sysctl -w fs.file-max=65535

Interview talking point: “ulimit -n changes limits only for your current shell, so a service still needs limits.conf entries, and fs.file-max is a system-wide ceiling separate from any single user.”

Q1034: How do you optimize TCP stack?

What it is: TCP stack tuning through /etc/sysctl.conf: raising receive and send buffer defaults and maximums, setting tcp_rmem and tcp_wmem, choosing cubic congestion control, enabling tcp_fastopen, enlarging the SYN backlog, shortening fin timeout and keepalive timers, applied with sysctl -p.

Why it is used: To improve throughput and latency on busy or high-bandwidth links.

# TCP buffer sizes
net.core.rmem_default=262144
net.core.rmem_max=16777216
net.core.wmem_default=262144
net.core.wmem_max=16777216

# TCP settings
net.ipv4.tcp_rmem=4096 87380 16777216
net.ipv4.tcp_wmem=4096 65536 16777216
net.ipv4.tcp_congestion_control=cubic
net.ipv4.tcp_fastopen=3
net.ipv4.tcp_max_syn_backlog=8192

# TCP performance
net.core.netdev_max_backlog=65535
net.ipv4.tcp_fin_timeout=15

# TCP keepalive
net.ipv4.tcp_keepalive_time=600
net.ipv4.tcp_keepalive_intvl=60
net.ipv4.tcp_keepalive_probes=5

# Apply
sysctl -p

Interview talking point: “The three-value tcp_rmem line is min, default, and max, and sysctl -p reloads the file so changes take effect without a reboot.”

Q1035: How do you optimize TCP/IP stack?

What it is: TCP stack tuning in sysctl adjusts connection tracking limits and timeouts, FIN and keepalive timers, and maximum socket buffer sizes, then reloads everything with sysctl -p.

Why it is used: To handle many short-lived connections without running out of tracked entries or leaving sockets lingering.

# Connection tracking
net.netfilter.nf_conntrack_max = 1048576
net.netfilter.nf_conntrack_tcp_timeout_established = 7200

# TCP timeouts
net.ipv4.tcp_fin_timeout = 15
net.ipv4.tcp_keepalive_time = 300
net.ipv4.tcp_keepalive_probes = 5
net.ipv4.tcp_keepalive_intvl = 15

# TCP buffer
net.core.rmem_max = 134217728
net.core.wmem_max = 134217728

# Apply
sysctl -p

Interview talking point: “Shortening tcp_fin_timeout and tcp_keepalive_time reclaims idle connections faster, while nf_conntrack_max must be raised on busy firewalls or the table fills and new packets get dropped.”

Q1036: How do you tune kernel parameters at runtime?

What it is: sysctl reads and writes kernel parameters under /proc/sys live, with -a to list, -w to set, and /etc/sysctl.conf plus sysctl -p for persistence.

Why it is used: To tune kernel networking and performance settings without recompiling anything.

# View all parameters
sysctl -a

# View specific
sysctl net.ipv4.tcp_rmem

# Set temporarily
sysctl -w net.ipv4.tcp_rmem="4096 87380 6291456"

# Set persistently
echo "net.ipv4.tcp_rmem = 4096 87380 6291456" >> /etc/sysctl.conf
sysctl -p

# View parameter documentation
ls /proc/sys/
# or
man sysctl.conf

Interview talking point: “sysctl -w changes only the running kernel and is lost on reboot, so persistence comes from appending to /etc/sysctl.conf and reloading it with sysctl -p.”

Q1037: How do you tune kernel parameters for performance?

What it is: /etc/sysctl.conf holds tunable kernel parameters such as socket buffer sizes, TCP memory, the file descriptor ceiling, and shared memory limits, applied with sysctl -p.

Why it is used: To raise kernel network and file limits for databases and high-connection servers.

# Network
net.core.rmem_max=16777216
net.core.wmem_max=16777216
net.ipv4.tcp_rmem=4096 87380 16777216
net.ipv4.tcp_wmem=4096 65536 16777216

# File descriptors
fs.file-max=65535

# Kernel
kernel.shmmax=68719476736
kernel.shmall=4294967296

# Apply
sysctl -p

Interview talking point: “sysctl -p reloads the file so changes take effect, and net.core.rmem_max caps how large a socket receive buffer may grow, which matters for high-bandwidth TCP links.”

Q1038: How do you tune kernel parameters?

What it is: sysctl reads and writes kernel parameters exposed under proc sys. A -w change lasts until reboot, while entries in sysctl.conf are applied permanently with sysctl -p.

Why it is used: To tune networking, memory, and swap behavior for the workload the machine runs.

# View parameters
sysctl -a
cat /proc/sys/.../...

# Set temporarily
sysctl -w net.ipv4.ip_forward=1

# Set permanently
# /etc/sysctl.conf
net.ipv4.ip_forward=1
vm.swappiness=10
net.core.somaxconn=1024

# Apply
sysctl -p

# For specific interface
sysctl -w net.ipv4.tcp_congestion_control=bbr

# Network performance
net.core.rmem_max=16777216
net.core.wmem_max=16777216
net.ipv4.tcp_rmem="4096 87380 16777216"
net.ipv4.tcp_wmem="4096 65536 16777216"

Interview talking point: “sysctl -w is temporary and lost on reboot, so permanent tuning belongs in sysctl.conf followed by sysctl -p, which is the classic distinction interviewers ask about.”

Q1039: How do you tune network kernel parameters?

What it is: sysctl network parameters control TCP/IP stack behavior including buffers, backlog, and keepalives.

Why it is used: Optimize throughput and latency for high-load network servers.

# Network core
net.core.somaxconn=65535
net.core.netdev_max_backlog=65535
net.core.rmem_max=16777216
net.core.wmem_max=16777216
net.core.optmem_max=25165824

# TCP tuning
net.ipv4.tcp_rmem=4096 87380 16777216
net.ipv4.tcp_wmem=4096 65536 16777216
net.ipv4.tcp_congestion_control=cubic
net.ipv4.tcp_fastopen=3
net.ipv4.tcp_max_syn_backlog=8192
net.ipv4.tcp_fin_timeout=15
net.ipv4.tcp_keepalive_time=600
net.ipv4.tcp_keepalive_intvl=60
net.ipv4.tcp_keepalive_probes=5
net.ipv4.tcp_tw_reuse=1

# Apply
sysctl -p

Interview talking point: “tcp_rmem and tcp_wmem define min/default/max socket buffers; tuning must balance memory use against expected traffic load.”

Q1040: How do you tune process limits?

What it is: Process limits live in /etc/security/limits.conf: soft and hard nofile raise open files, nproc caps processes, core controls core dumps, and memlock allows locked memory; ulimit sets a value for the current shell and /proc/PID/limits shows actual limits.

Why it is used: To stop services from failing with too many open files or fork errors.

# Max open files
* soft nofile 65535
* hard nofile 65535
root soft nofile 65535
root hard nofile 65535

# Max processes
* soft nproc 4096
* hard nproc 8192

# Core dumps
* soft core 0
* hard core unlimited

# Locked memory
* soft memlock unlimited
* hard memlock unlimited

# Apply without logout
ulimit -n 65535

# View limits
ulimit -a
cat /proc/<pid>/limits

Interview talking point: “Soft versus hard matters: soft can be raised by the user up to the hard value, and limits.conf applies to new login sessions, which is why ulimit covers the current shell.”

Q1041: How do you tune TCP stack for high performance?

What it is: sysctl settings in /etc/sysctl.conf tune the TCP stack: backlog and local port range, timeouts, keepalive timing, and socket buffer sizes including rmem and wmem maxima.

Why it is used: To carry many concurrent connections with low latency on high-throughput servers.

net.core.somaxconn = 65535
net.ipv4.tcp_max_syn_backlog = 65535
net.ipv4.ip_local_port_range = 1024 65535
net.ipv4.tcp_fin_timeout = 15
net.ipv4.tcp_max_tw_buckets = 2000000
net.ipv4.tcp_tw_reuse = 1
net.ipv4.tcp_keepalive_time = 300
net.ipv4.tcp_keepalive_intvl = 15
net.ipv4.tcp_keepalive_probes = 5
net.core.rmem_max = 16777216
net.core.wmem_max = 16777216
net.ipv4.tcp_rmem = 4096 87380 16777216
net.ipv4.tcp_wmem = 4096 65536 16777216

Interview talking point: “tcp_fin_timeout 15 and tcp_tw_reuse 1 recycle connections faster, while tcp_rmem and tcp_wmem set per-socket buffer min, default, and max. Raise net.core.rmem_max first, otherwise the tcp_rmem maximum can never actually be reached.”

Q1042: How do you tune virtual memory?

What it is: Virtual memory tuning via sysctl: vm.swappiness and vfs_cache_pressure steer reclaim, dirty ratios bound writeback, overcommit settings govern allocation, and hugepages are reserved through nr_hugepages or the sysfs file, with transparent hugepages set to never.

Why it is used: To reduce latency spikes and improve memory behavior for databases and heavy workloads.

# Swappiness
vm.swappiness=10
vm.vfs_cache_pressure=50

# Memory management
vm.dirty_ratio=15
vm.dirty_background_ratio=5
vm.dirty_expire_centisecs=3000
vm.dirty_writeback_centisecs=500

# Overcommit
vm.overcommit_memory=1
vm.overcommit_ratio=50

# Huge pages
vm.nr_hugepages=512

# Apply
sysctl -p

# Configure huge pages
echo 1024 > /sys/kernel/mm/hugepages/hugepages-2048kB/nr_hugepages

# Transparent huge pages
echo never > /sys/kernel/mm/transparent_hugepage/enabled
echo never > /sys/kernel/mm/transparent_hugepage/defrag

Interview talking point: “Setting transparent hugepages to never avoids the defragmentation pauses databases suffer, while explicit nr_hugepages reserves deterministic huge page blocks for applications that want them.”

Q1043: How do you use tuned?

What it is: tuned is a profile-based tuning daemon; tuned-adm list shows profiles, tuned-adm profile applies one, and tuned-adm verify checks it.

Why it is used: To apply a tested bundle of settings with a single command.

# Install
apt install tuned

# List profiles
tuned-adm list

# Active profile
tuned-adm active

# Set profile
tuned-adm profile throughput-performance
tuned-adm profile virtual-guest

# Verify
tuned-adm verify

Interview talking point: “tuned-adm active reports the current profile and verify confirms settings still match, so a drifted host can be spotted and re-applied quickly.”

CPU Scheduling, Affinity, and NUMA

Q1044: How does Linux handle process scheduling?

What it is: Linux schedules CPU time with the Completely Fair Scheduler, giving each runnable process a fair share. Priority appears as nice values, and chrt assigns real-time scheduling classes.

Why it is used: To explain how CPU time is shared and how to raise or lower a workload’s priority.

# View process priority and nice value
ps -eo pid,ni,pri,pcpu,comm
renice 10 -p 1234

# Set real-time priority
chrt -f 50 -p 1234
chrt -r -p 50 1234

# View scheduler
cat /proc/1234/sched

Interview talking point: “renice adjusts the normal CFS nice value from -20 to 19, while chrt -f and chrt -r assign real-time policies that can starve ordinary processes. Check a process with cat /proc/PID/sched.”

Q1045: How do you configure CPU isolation?

What it is: isolcpus=1,2,3 on the kernel command line keeps those CPUs out of the scheduler, applied by editing GRUB and running update-grub.

Why it is used: To reserve cores for a latency-critical process with no scheduler noise.

# Add kernel parameter
GRUB_CMDLINE_LINUX_DEFAULT="isolcpus=1,2,3"

# Update GRUB
update-grub

# Or using cgroups
# cset shield -c 1,2,3

Interview talking point: “isolcpus removes the cores from general scheduling at boot, while the cset shield alternative works at runtime, so the choice depends on whether you can reboot.”

Q1046: How do you configure KVM CPU pinning?

What it is: CPU pinning assigns guest vCPUs and the emulator thread to host cores: virsh vcpupin pins a vCPU, emulatorpin pins the emulation thread, and numatune prefers NUMA nodes.

Why it is used: Reduces scheduler jitter and keeps a guest’s memory close to its CPUs.

# Pin to cores
virsh vcpupin domain 0 0,1,2,3

# Pin with emulator
virsh emulatorpin domain 0 0-3

# NUMA
virsh numatune domain --mode preferred --nodeset 0-1

Interview talking point: “Pinning vCPUs alone is not enough because the emulator thread can still land on another socket, so emulatorpin plus numatune keeps compute and memory local, and that NUMA locality is where the performance win comes from.”

Q1047: How do you use cpuset?

What it is: cpuset manages named groups of CPUs; cpuset -c creates a group, -g sets its CPUs, and -a attaches a PID to it.

Why it is used: To pin a workload to specific cores so it stays off shared ones.

# Install
apt install cpuset

# Create cpuset
cpuset -c -g system:myapp

# Set CPUs
cpuset -g cpus:0-3 system:myapp

# Add process
cpuset -a PID system:myapp

# Check
cpuset -g system:myapp

Interview talking point: “cpuset -g system:myapp both creates the group and reports its current cpus value, so membership and assignment live in one command family.”

Q1048: How do you use I/O schedulers?

What it is: The I/O scheduler queues requests per block device; you read and write /sys/block/sda/queue/scheduler to pick deadline or none.

Why it is used: To match scheduling behavior to the device, deadline for HDDs and none for SSDs.

# Check
cat /sys/block/sda/queue/scheduler

# Set deadline (for HDD)
echo deadline > /sys/block/sda/queue/scheduler

# Set none (for SSD)
echo none > /sys/block/sda/queue/scheduler

# Persistent
# /etc/udev/rules.d/60-ioschedulers.rules
ACTION=="add|change", KERNEL=="sd[a-z]", ATTR{queue/scheduler}="deadline"

Interview talking point: “The udev rule under /etc/udev/rules.d reapplies the scheduler whenever a disk is added or changed, so the choice survives reboots without a separate init script.”

Q1049: How do you use numactl?

What it is: numactl shows NUMA topology with —hardware and binds a process to a node’s CPUs or memory, including —interleave=all.

Why it is used: To avoid remote memory access penalties on multi-socket machines.

# Show NUMA topology
numactl --hardware
numactl --show

# Run on specific node
numactl --membind=0 command
numactl --cpunodebind=0 command

# Interleave
numactl --interleave=all command

Interview talking point: “—cpunodebind pins execution while —membind pins allocations, and forgetting one means the process can still allocate across nodes and quietly lose locality.”

Q1050: How do you use parallel processing?

What it is: GNU parallel, parallel-ssh, and xargs -P run many jobs concurrently, with -j or -P controlling how many execute at once.

Why it is used: To cut wall clock time on fleet-wide commands or bulk file processing.

# GNU Parallel
apt install parallel

# Parallel execution
cat servers.txt | parallel -j 10 "ssh {} 'uptime'"

# With SSH
parallel-ssh -h hosts.txt -i "uptime"

# Process files in parallel
ls *.jpg | parallel -j 4 convert -resize 800x600 {} {}_small.jpg

# With xargs
find . -name "*.log" | xargs -P 4 -I {} gzip {}

# GNU parallel with variables
seq 1 100 | parallel -j 10 'echo "Number {}"'

Interview talking point: “parallel -j 10 caps concurrency at ten jobs and xargs -P 4 does the same in a pipeline, and that cap matters when hitting dozens of remote hosts at once.”

Q1051: How do you use slurm?

What it is: slurm is a console network monitor; slurm -i eth0 watches that interface and -s splits transmit and receive into separate views.

Why it is used: Gives a simple real-time load picture inside an ongoing terminal session.

# Install
apt install slurm

# Usage
slurm -i eth0
slurm -i eth0 -s  # split TX/RX

Interview talking point: “The -s option that separates TX from RX is the notable one, because a combined total can hide a one-sided problem such as a saturated upload that the split view exposes immediately.”

Q1052: How do you use taskset?

What it is: taskset sets CPU affinity; -c takes a core list like 0-3 for a command, while -p sets or shows the mask of a running PID.

Why it is used: To bind a process to particular cores and reduce cross-core migration.

# Set CPU affinity
taskset -c 0,1 command
taskset -p 0x03 PID

# Show affinity
taskset -p PID

# Run on cores
taskset -c 0-3 myapp

Interview talking point: “taskset -p prints existing affinity as a hex mask such as 0x03, while taskset -c accepts readable lists like 0-3, so both forms describe the same thing.”

Q1053: How do you optimize disk I/O scheduler?

What it is: The block layer scheduler decides how I/O requests are queued. You can write noop, mq-deadline, or bfq into /sys/block/sda/queue/scheduler, and a udev rule makes the choice persist.

Why it is used: To reduce latency or raise throughput by matching the scheduler to SSD or spinning-disk hardware.

# For SSD (no-op or deadline)
echo noop > /sys/block/sda/queue/scheduler

# For HDD (mq-deadline or bfq)
echo mq-deadline > /sys/block/sda/queue/scheduler

# Make persistent
# /etc/udev/rules.d/60-ioschedulers.rules
ACTION=="add|change", KERNEL=="sd[a-z]", ATTR{queue/scheduler}="mq-deadline"

Interview talking point: “The comments suggest noop for SSD and mq-deadline or bfq for HDD, and a direct write takes effect immediately but is volatile. The udev rule with ACTION add|change reapplies the scheduler automatically whenever the disk is discovered.”

Q1054: How do you optimize for real-time?

What it is: Real time tuning installs an RT kernel, configures rtirq for interrupt priority, and isolates CPUs by adding isolcpus to the GRUB line.

Why it is used: To give time critical processes dedicated cores with minimal interrupt interference.

# Install RT kernel
apt install linux-image-rt-*

# Configure
# /etc/default/rtirq
RTIRQ_RESET_ALL=on
RTIRQ_POLLING="rtc"

# CPU isolation
# Add to GRUB: isolcpus=1,2,3

Interview talking point: “isolcpus removes CPUs from the scheduler entirely, so those cores only run processes you explicitly bind to them.”

Q1055: How do you optimize Linux for HPC?

What it is: HPC tuning sets the CPU governor to performance, loads RDMA modules such as mlx5_core, and mounts parallel filesystems with noatime.

Why it is used: To squeeze consistent maximum throughput from compute and interconnect.

# CPU governor
echo performance > /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor

# Network
# Use RDMA
modprobe mlx5_core
# or
modprobe rdma_ucm

# Filesystem
# Use Lustre, GPFS, or parallel fs
# Mount with noatime

Interview talking point: “scaling_governor set to performance pins CPUs at full clock, and noatime on the parallel filesystem removes a metadata update on every read.”

Q1056: How do you optimize Linux for networking?

What it is: Linux network tuning covering buffer sizes, offloading, TCP parameters, queue tuning, and network metrics monitoring.

Why it is used: Raises throughput and lowers latency on busy Linux hosts.

# Linux network optimization
# 1. Buffer sizes
# Tuning

# 2. Offloading
# Enable

# 3. TCP
# Parameters

# 4. Queue
# Tuning

# 5. Monitoring
# Metrics

Interview talking point: “Buffer sizes and TCP parameters are the knobs most often adjusted under load, while offloading moves work to the NIC to save CPU. Watching metrics after each change keeps tuning grounded in evidence.”

Q1057: How do you optimize Linux for networking?

What it is: Linux networking optimization tunes buffer sizes, enables offloading, adjusts TCP parameters, tunes queues, and monitors network metrics.

Why it is used: To improve throughput, reduce latency, and handle high packet rates efficiently.

# Buffer sizes
# Tuning

# Offloading
# Enable

# TCP
# Parameters

# Queue
# Tuning

# Monitoring
# Metrics

Interview talking point: “TCP parameters and buffer sizes must match workload characteristics; network metrics validate improvements under real traffic conditions.”

Q1058: How do you use NUMA optimization?

What it is: numactl —hardware reports NUMA nodes, and numactl with cpunodebind and membind runs a process with CPU and memory bound to one node.

Why it is used: To keep a process on one node so its memory stays local to its CPUs.

# Check NUMA
numactl --hardware

# Pin process
numactl --cpunodebind=0 --membind=0 process

# Use libnuma
# In code
numa_run_on_node(0);
numa_tonodememory(0, size);

Interview talking point: “Binding both CPU and memory matters because remote node memory costs extra interconnect latency even when the thread runs on the correct core.”

Q1059: How do you design for real-time processing?

What it is: Real-time processing uses stream processing with Kafka or Spark, optimizes for low latency, scales horizontally, monitors metrics, and handles backpressure.

Why it is used: To react to events as they happen rather than in batches.

# Real-time processing
# 1. Stream processing
# Kafka/Spark

# 2. Low latency
# Optimization

# 3. Scalability
# Horizontal

# 4. Monitoring
# Metrics

# 5. Backpressure
# Handling

Interview talking point: “Backpressure handling is what keeps a fast producer from overwhelming consumers, so the pipeline degrades gracefully instead of silently dropping events under load.”

Q1060: How do you design for real-time processing?

What it is: Real-time processing design uses stream processing (Kafka/Spark), low-latency optimization, horizontal scalability, metrics monitoring, and backpressure handling.

Why it is used: To process and act on data as it arrives for timely decisions.

# Stream processing
# Kafka/Spark

# Low latency
# Optimization

# Scalability
# Horizontal

# Monitoring
# Metrics

# Backpressure
# Handling

Interview talking point: “Backpressure handling and horizontal scaling keep systems responsive under load, while metrics monitoring helps maintain low latency.”

eBPF and Kernel Tracing

Q1061: How do you configure BPF programs?

What it is: This is an XDP BPF program in C whose xdp_drop section returns XDP_DROP for every packet, with the license string set to GPL.

Why it is used: To drop unwanted network packets inside the kernel before they reach the network stack.

#include <linux/bpf.h>
#include <bpf/bpf_helpers.h>

SEC("xdp_drop")
int xdp_drop_prog(struct xdp_md *ctx) {
    return XDP_DROP;
}

char _license[] SEC("license") = "GPL";

Interview talking point: “Returning XDP_DROP at the XDP hook discards the packet at the driver, which is much cheaper than filtering the same traffic with iptables rules later.”

Q1062: How do you configure dynamic tracing?

What it is: bpftrace runs short tracing scripts against kernel probe points; mytrace.bt prints a start message then logs PID and nice value on each kprobe:do_nice hit.

Why it is used: To instrument the kernel on demand and see what a system is doing at runtime.

# Use bpftrace
apt install bpftrace

# Write script
# mytrace.bt
#!/usr/bin/bpftrace
BEGIN
{
    printf("Tracing... Hit Ctrl-C to end.\n")
}
kprobe:do_nice
{
    printf("PID %d nice: %d\n", pid, arg1)
}

# Run
bpftrace mytrace.bt

Interview talking point: “The script attaches a kprobe to do_nice and reads arg1, showing how bpftrace turns a probe into ad hoc instrumentation you can start and stop with Ctrl-C.”

Q1063: How do you configure eBPF?

What it is: eBPF runs sandboxed programs inside the kernel, and bpftrace plus bpfcc-tools are tracing front ends. btrace attaches to a PID or runs a script file.

Why it is used: To trace syscalls, process starts, and connections in production without rebuilding or restarting the target program.

# Install bpftrace
apt install bpftrace

# Simple trace
btrace -p PID
btrace /path/to/program.bt

# Using bpfcc
apt install bpfcc-tools
execsnoop-bpfcc
opensnoop-bpfcc
tcpconnect-bpfcc

Interview talking point: “The bpfcc package ships purpose-built tools: execsnoop logs new processes, opensnoop logs file opens, and tcpconnect logs outbound connections. btrace is the quick path, either attaching to a running PID or feeding it a script.”

Q1064: How do you use Calico eBPF?

What it is: Calico can run its dataplane in eBPF mode instead of the default. The calicoctl patch turns on bpfEnabled with bpfLogLevel set to info.

Why it is used: To move packet handling from iptables to eBPF for better throughput and lower latency.

# Install Calico
kubectl apply -f https://docs.projectcalico.org/manifests/calico.yaml

# Enable eBPF mode
calicoctl patch felixdefault -p '{"spec":{"bpfLogLevel":"info","bpfEnabled":true}}'

# Check
calicoctl node status

Interview talking point: “The patch targets Felix configuration, which holds Calico’s per-node settings, and calicoctl node status is the check that confirms the node is healthy after the switch.”

Q1065: How do you use eBPF for tracing?

What it is: bpftrace attaches to tracepoints and kprobes: one script counts syscalls per command, another quantizes do_nice latency, a third counts tcp_connect calls.

Why it is used: To measure syscall volume and function latency directly from the kernel in real time.

# bpftrace examples
# Count system calls
bpftrace -e 'tracepoint:syscalls:sys_enter_* { @[comm] = count(); }'

# Measure latency
bpftrace -e 'kprobe:do_nice { @start[comm] = nsecs(); } kretprobe:do_nice /@start[comm]/ { @ns[comm] = quantize(nsecs() - @start[comm]); }'

# Network connections
bpftrace -e 'kprobe:tcp_connect { @[comm] = count(); }'

Interview talking point: “Storing nsecs at kprobe and quantizing the delta at kretprobe is the standard latency pattern, turning entry and return pairs into a distribution.”

Q1066: How do you use eBPF?

What it is: eBPF runs small sandboxed programs inside the kernel; bpftrace scripts and bpfcc tools trace syscalls, function calls, and latency live.

Why it is used: To diagnose performance and see what the kernel or a process is doing without restarting anything.

# Install bpftrace
apt install bpftrace

# Trace
btrace -p PID
btrace /path/to/script.bt

# Using bpfcc
apt install bpfcc-tools
execsnoop-bpfcc
opensnoop-bpfcc
funclatency-bpfcc

# Write BPF program
cat > test.bt <<EOF
BEGIN
{
    printf("Tracing... Hit Ctrl-C to end.\n")
}
kprobe:do_nice
{
    printf("%s %d\n", comm, arg1)
}
EOF
btrace test.bt

Interview talking point: “btrace runs a bpftrace script such as a kprobe on do_nice, while bpfcc ships ready tools like execsnoop-bpfcc and funclatency-bcc that answer common questions without writing a program.”

Q1067: How do you use eBPF?

What it is: eBPF attaches small programs to kernel tracepoints; bpftrace scripts print file opens for a filtered process, and bcc ships ready tools like execsnoop and opensnoop.

Why it is used: To observe syscalls and kernel behavior live with almost no overhead or custom code.

# Install bpftrace
apt install bpftrace

# Simple trace
bpftrace -e 'BEGIN { printf("Tracing... Hit Ctrl-C to end.\n"); }'

# Trace syscalls
bpftrace -e 'tracepoint:syscalls:sys_enter_openat { printf("%s\n", str(args->filename)); }'

# Custom bpf program
# hello.bt
#!/usr/bin/bpftrace
tracepoint:syscalls:sys_enter_read
/pid == 1234/
{
    @[comm] = count();
}

# Run
chmod +x hello.bt
./hello.bt

# Using bcc
/usr/share/bcc/tools/execsnoop
/usr/share/bcc/tools/opensnoop
/usr/share/bcc/tools/tcplife

Interview talking point: “The slash predicate around pid equals 1234 is a bpftrace filter that runs only for matching events, and bcc tools give the same visibility without writing any program yourself.”

Q1068: How do you use ftrace?

What it is: ftrace is the kernel’s built-in tracer exposed under debugfs, where you choose a tracer, filter function names, and read results from the trace file.

Why it is used: To record which kernel functions run, using a tracer already compiled into the kernel.

# Enable
mount -t debugfs nodev /sys/kernel/debug

# List available tracers
cat /sys/kernel/debug/tracing/available_tracers

# Set tracer
echo function > /sys/kernel/debug/tracing/current_tracer

# Enable function
echo '*tcp*' > /sys/kernel/debug/tracing/set_ftrace_filter

# Read
cat /sys/kernel/debug/tracing/trace

Interview talking point: “set_ftrace_filter narrows tracing to matching names such as a tcp wildcard, which keeps overhead low, and ftrace is driven by writing files rather than passing command arguments.”

Q1069: How do you use perfetto?

What it is: Perfetto is a system tracer configured by a text config file that selects ftrace events such as sched_switch and cpu_frequency, writing a trace.perfetto capture.

Why it is used: To capture scheduler and power events into one timeline for performance analysis.

# Install
apt install perfetto

# Create config
# config.pbtxt
buffers: {
    size_kb: 63488
    fill_policy: DISCARD
}
data_sources: {
    config {
        name: "linux.ftrace"
        ftrace_config {
            ftrace_events: "sched/sched_switch"
            ftrace_events: "power/cpu_frequency"
        }
    }
}

# Trace
perfetto -c config.pbtxt -o trace.perfetto

Interview talking point: “The buffer sets fill_policy DISCARD, meaning older events are dropped once full, so raising size_kb matters more than trying to recover a truncated capture.”

Q1070: How do you use Sysdig?

What it is: Sysdig captures every system call into a .scap file and replays it later, with chisels such as topprocs_cpu for analysis.

Why it is used: To record what happened and analyze it offline instead of guessing while live.

# Install
apt install sysdig

# Capture
sysdig -w dump.scap
sysdig -w dump.scap -C 100  # 100MB files

# Analyze
sysdig -r dump.scap
sysdig -r dump.scap -c "topprocs_cpu"
sysdig -r dump.scap -c "topfiles_bytes"

# Live
sysdig -c topprocs_cpu
sysdig -c fdcount_by proc.name "evt.type=read"

Interview talking point: “sysdig -w records, sysdig -r replays, and -C 100 rotates the capture at 100MB so long recordings stay manageable.”

Q1071: How do you use BPF for performance?

What it is: bpfcc-tools ships one purpose built BPF tool per question: funccount counts kernel functions, biolatency shows block I/O latency, tcplife tracks TCP sessions.

Why it is used: To answer performance questions live with low overhead instead of guessing from averages.

# Use bpfcc-tools
apt install bpfcc-tools

# CPU analysis
funccount-bpfcc 't:kernel_function'

# I/O analysis
biolatency-bpfcc

# Network analysis
tcplife-bpfcc

Interview talking point: “biolatency prints a latency histogram rather than a single average, which exposes tail latency that a mean would hide.”

Q1072: How do you use BPF for performance?

What it is: eBPF allows safe kernel tracing and performance analysis without modifying application or kernel source code.

Why it is used: Diagnose latency, I/O, and syscall behavior in production with low overhead.

# Install bpftrace
apt install bpftrace

# List probes
tplist -l

# Trace file opens
bpftrace -e 'tracepoint:syscalls:sys_enter_openat { printf("%s %s\n", comm, str(args->filename)); }'

# Trace TCP connect
bpftrace -e 'tracepoint:syscalls:sys_enter_connect { printf("Connect: %s\n", comm); }'

# Custom program
# /root/bpf/hello.bt
#!/usr/bin/bpftrace
BEGIN
{
    printf("Tracing... Hit Ctrl-C to end.\n");
}

tracepoint:syscalls:sys_enter_read
{
    @read_bytes = comm;
}

# Using perf
perf record -g ./program
perf report

# Using BCC tools
# /usr/share/bcc/tools/
/usr/share/bcc/tools/ext4slower 1
/usr/share/bcc/tools/tcpconnect

Interview talking point: “bpftrace provides a high-level language for tracepoints and syscalls; BCC tools offer prebuilt performance observability.”

Q1073: How do you use Sysdig Secure?

What it is: Sysdig Secure installs its agent into a namespace by applying a downloaded manifest with the cluster name substituted, and the sysdig CLI shows container views.

Why it is used: To monitor and secure container workloads with per-container visibility from a single agent.

# Install agent
kubectl create namespace sysdig-agent
curl -s https://download.sysdig.com/stable/agent-k8s.yaml | sed 's/CLUSTER_NAME/cluster-name/g' | kubectl apply -f -

# Use
sysdig -pc -c "topcontainers_cpu"

Interview talking point: “The install pipes the downloaded manifest through sed to substitute CLUSTER_NAME before applying it, and sysdig -pc -c topcontainers_cpu then ranks containers by CPU — deployment and diagnosis use the same CLI.”

Live Patching

Q1074: How do you patch kernel live?

What it is: Live kernel patching: kpatch on RHEL builds and loads a patch module with kpatch list to check it, while canonical-livepatch on Ubuntu installs as a snap, enables with a token, and reports with a status command.

Why it is used: To apply critical kernel fixes without rebooting production machines.

# Using kpatch (RHEL/CentOS)
yum install kpatch
kpatch install

# Build patch
kpatch build patch.diff

# Apply
kpatch load kpatch-mypatch.ko

# Check
kpatch list

# Using livepatch (Ubuntu)
snap install canonical-livepatch
canonical-livepatch enable <token>

# Check status
canonical-livepatch status

Interview talking point: “Live patching removes the reboot window for urgent fixes, but it does not replace normal patching, since a reboot is still needed to reach a fully updated kernel.”

Q1075: How do you configure kernel live patching?

What it is: Ubuntu Livepatch is Canonical’s kernel live patching service, installed as a snap and enabled with canonical-livepatch, applying fixes to the running kernel without a reboot.

Why it is used: To close kernel vulnerabilities immediately on servers that have no reboot window.

# Ubuntu Livepatch
# Install
snap install canonical-livepatch

# Enable
sudo canonical-livepatch enable

# Check status
canonical-livepatch status

Interview talking point: “canonical-livepatch status shows the patched kernel and service state, and Livepatch only covers selected critical CVEs, so ordinary reboots are still needed for full updates.”

Q1076: How do you implement kernel live patching?

What it is: Kernel live patching applies security fixes to a running kernel without requiring a reboot.

Why it is used: Minimize downtime on critical systems while applying urgent kernel fixes.

# Using kpatch (RHEL/CentOS)
yum install kpatch
kpatch install

# Create patch
# 1. Create patch file
# patch.diff
diff -Naur orig/file.c new/file.c

# 2. Build patch
kpatch build patch.diff

# 3. Apply
kpatch load kpatch-mypatch.ko

# Using livepatch (Ubuntu)
snap install canonical-livepatch
canonical-livepatch enable <token>

# Check status
canonical-livepatch status
kpatch list

Interview talking point: “Tools like kpatch (RHEL) or canonical-livepatch apply validated binary patches to the running kernel without interrupting services.”

Q1077: How do you use kpatch?

What it is: kpatch compiles a patch.diff into a loadable module with kpatch-build, applies it with kpatch load, and reports active patches with kpatch list.

Why it is used: To apply compiled kernel fixes to a live kernel without restarting the machine.

# Install
apt install kpatch-dkms

# Create patch
kpatch-build patch.diff

# Apply
kpatch load patch.ko

# List
kpatch list

Interview talking point: “kpatch-build compiles the fix while kpatch load activates the resulting patch.ko, so building and applying are separate steps and kpatch list confirms what is running.”

Kernel Development and Custom Builds

Q1078: How do you compile a custom Linux kernel?

What it is: Building a custom kernel means installing build dependencies, cloning the kernel source, choosing a config with menuconfig or by copying the running config and running olddefconfig, then building and installing modules and the kernel.

Why it is used: To enable or remove drivers and features the distribution kernel does not match.

# Install build dependencies
apt build-dep linux
apt install git bc bison flex libssl-dev

# Download source
git clone --depth 1 https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git
cd linux

# Configure
make menuconfig
# or
make xconfig
# or copy current config
cp /boot/config-$(uname -r) .config
make olddefconfig

# Build
make -j$(nproc)
make modules_install
make install

# Update boot loader
update-grub

# Reboot to new kernel
reboot

Interview talking point: “Copying the running kernel config and then running olddefconfig answers new options with defaults, giving a known starting point, and make with -j plus nproc parallelizes compilation across every core.”

Q1079: How do you compile Linux kernel?

What it is: Building the kernel from source: install the toolchain, fetch the tarball, configure with menuconfig, build in parallel, install modules, and refresh GRUB.

Why it is used: To add patches, enable drivers, or customize kernel behavior for a specific workload.

# Install dependencies
apt install build-essential libncurses-dev bison flex libssl-dev libelf-dev

# Download kernel
wget https://cdn.kernel.org/pub/linux/kernel/v5.x/linux-5.15.tar.xz
tar -xf linux-5.15.tar.xz
cd linux-5.15

# Configure
make menuconfig

# Build
make -j$(nproc)
make modules_install
make install

# Update GRUB
update-grub
reboot

Interview talking point: “make -j$(nproc) spreads compilation across all cores, and make modules_install plus make install are what actually place the kernel before update-grub registers the new boot entry.”

Q1080: How do you handle kernel upgrades?

What it is: Kernel upgrades follow a staged workflow: test in staging, check compatibility, back up, schedule a window, apply, monitor, and keep a rollback plan ready.

Why it is used: To change the kernel on production hosts without risking boot failures or outages.

# Kernel upgrade
# 1. Test in staging
# 2. Check compatibility
# 3. Backup
# 4. Schedule window
# 5. Apply
# 6. Monitor
# 7. Rollback plan

Interview talking point: “A rollback plan sits alongside apply, so keep the previous kernel available in the bootloader menu so the host can boot back if the new kernel fails.”

Q1081: How do you handle kernel upgrades?

What it is: Kernel upgrades update the core operating system to patch vulnerabilities and improve performance.

Why it is used: They keep systems secure and compatible with newer hardware and software.

# Test in staging
# Check compatibility
# Backup
# Schedule window
# Apply
# Monitor
# Rollback plan

Interview talking point: “Testing in staging, taking backups, scheduling during maintenance windows, and having a rollback plan reduce upgrade risk.”

Q1082: How do you perform kernel upgrades?

What it is: apt upgrade or dist-upgrade and yum update install packaged kernels before a reboot; a manual build runs menuconfig, modules_install, make install and update-grub from a tarball.

Why it is used: Take security fixes and new features while keeping a bootable fallback kernel.

# Debian/Ubuntu
apt update
apt list --upgradable
apt upgrade
apt-get dist-upgrade
reboot

# RHEL/CentOS
yum update
reboot

# Manual kernel compile
cd /usr/src
wget https://cdn.kernel.org/pub/linux/kernel/v5.x/linux-5.15.tar.xz
tar -xJf linux-5.15.tar.xz
cd linux-5.15
cp /boot/config-$(uname -r) .config
make menuconfig
make -j$(nproc)
make modules_install
make install
update-grub
reboot

Interview talking point: “apt upgrade avoids changes that would remove packages while dist-upgrade permits them, and after a manual compile update-grub is what puts the new kernel into the boot menu.”

Q1083: How do you add system calls to kernel?

What it is: Adding a syscall means writing the handler, registering a number in the architecture syscall table, defining the NR macro, then calling it with syscall() from user space.

Why it is used: To expose new kernel functionality to user-space programs through a stable entry point.

// mysyscall.c (in kernel source)
asmlinkage long sys_mysyscall(int arg) {
    printk(KERN_INFO "My syscall called with %d\n", arg);
    return 0;
}

// Add to syscall table (arch/x86/entry/syscalls/syscall_64.tbl)
555 64  mysyscall  sys_mysyscall

// In kernel headers
#define __NR_mysyscall 555

// User space
#include <sys/syscall.h>
syscall(555, arg);

Interview talking point: “The number 555 must match in both the syscall table and the NR define, because user space passes that raw number straight to the syscall() wrapper.”

Kernel Debugging

Q1084: How do you debug kernel issues?

What it is: Kernel debugging uses debug config options, kgdb over a serial console, kexec crash capture with the crash tool, and logs from dmesg and journalctl -k.

Why it is used: To find why the kernel panicked or misbehaved when ordinary tooling is unavailable.

# Kernel debugging
# Enable debug
CONFIG_DEBUG_INFO=y
CONFIG_DEBUG_KERNEL=y
CONFIG_KALLSYMS=y

# Use kgdb
CONFIG_KGDB=y
# Add to boot: kgdboc=ttyS0,115200

# Use crash dump
# Install crash
apt install crash
# Capture
kexec -p /boot/vmlinuz --initrd=/boot/initrd.img --append="-- crash"

# Kernel logs
dmesg
journalctl -k
/var/log/dmesg

Interview talking point: “journalctl -k and dmesg both expose the kernel ring buffer, while kgdboc on ttyS0 attaches a debugger through the serial port before a crash loses the state.”

Q1085: How do you debug kernel issues?

What it is: Kernel debugging reads dmesg for messages, installs kdump to capture crash dumps, inspects kernel config, and traces activity with ftrace and perf.

Why it is used: To diagnose panics, hangs, or unexpected driver behavior after an incident.

# Kernel messages
dmesg
dmesg | tail -100

# Kernel panic
# Enable kdump
apt install kdump-tools
kdump-config load

# Test kdump
echo c > /proc/sysrq-trigger

# Analyze crash
crash /var/crump/ vmcore

# Kernel config
zcat /proc/config.gz
# or
cat /boot/config-$(uname -r)

# System calls
strace -c program
strace -f program

# ftrace
echo function > /sys/kernel/debug/tracing/current_tracer
cat /sys/kernel/debug/tracing/trace

# perf
perf record -g program
perf report

Interview talking point: “echo c to /proc/sysrq-trigger deliberately panics the kernel to test kdump, so only run it on a machine you intend to crash.”

Q1086: How do you debug kernel panics?

What it is: kdump reserves a crashkernel region from the GRUB command line, captures vmcore under /var/crash with makedumpfile, and the crash utility inspects it against debug vmlinux.

Why it is used: Preserve system memory at panic time so the root cause can be analyzed offline.

# Configure kdump
GRUB_CMDLINE_LINUX="crashkernel=auto"

# Install kdump
apt install kdump-tools

# Configure
# /etc/kdump.conf
path /var/crash
core_collector makedumpfile -c --page-analysis

# Enable
systemctl enable kdump
systemctl start kdump

# Test
echo c > /proc/sysrq-trigger

# Analyze crash dump
crash /var/crash/2024-01-01-01:01/vmcore /usr/lib/debug/boot/vmlinux-$(uname -r)

# Kernel panic symptoms
# "Kernel panic - not syncing: VFS: Unable to mount root fs"
# "Kernel panic - not syncing: Out of memory and no killable processes"

# Debug
# /etc/sysctl.conf
kernel.panic=10
kernel.panic_on_oops=1

Interview talking point: “Writing c to /proc/sysrq-trigger deliberately panics the kernel to prove kdump is armed, and sysctl kernel.panic_on_oops=1 converts an oops into a reboot rather than a wedged host.”

Advanced Networking

Q1087: How do you configure 802.1Q VLAN tagging?

What it is: 802.1Q is the tag format for VLANs on Ethernet. The record loads the 8021q module, adds it to /etc/modules, creates VLAN 100 with vconfig, assigns an address, and verifies with ip -d link show.

Why it is used: To carry tagged VLAN traffic on a host, and to have the tagging support load automatically at boot.

# Load kernel module
modprobe 8021q

# Make persistent
echo "8021q" >> /etc/modules

# Create VLAN
vconfig add eth0 100

# Configure IP
ip addr add 192.168.100.10/24 dev eth0.100
ip link set eth0.100 up

# Verify
ip -d link show eth0.100

Interview talking point: “VLAN tagging needs the 8021q kernel module first, modprobe loads it now while /etc/modules makes it persistent, and ip -d link show eth0.100 confirms the interface with its VLAN id.”

Q1088: How do you configure advanced routing?

What it is: Policy routing adds named routing tables plus rules that pick a table by source or destination, alongside NAT and weighted multi-path default routes.

Why it is used: To send traffic from certain hosts out a second WAN, or to publish internal services.

# Policy routing
# Add table
echo "200 wan2" >> /etc/iproute2/rt_tables

# Add route
ip route add default via 192.168.2.1 dev eth1 table wan2

# Add rule
ip rule add from 192.168.2.10 table wan2
ip rule add to 192.168.2.0/24 table wan2

# NAT with iptables
# SNAT
iptables -t nat -A POSTROUTING -s 192.168.1.0/24 -o eth0 -j SNAT --to-source 203.0.113.10

# DNAT
iptables -t nat -A PREROUTING -i eth0 -p tcp --dport 8080 -j DNAT --to-destination 192.168.1.10:80

# Masquerade
iptables -t nat -A POSTROUTING -o eth0 -j MASQUERADE

# Multi-path routing
ip route add default scope global nexthop via 192.168.1.1 dev eth0 weight 1 \
    nexthop via 192.168.2.1 dev eth1 weight 1

Interview talking point: “ip rule decides which table is consulted before the main table, so traffic from 192.168.2.10 can exit via eth1 even though the default route points elsewhere.”

Q1089: How do you configure BGP?

What it is: This Bird BGP block sets local as 65000, points at neighbor 192.168.1.20 as 65001, accepts and advertises all routes, and enables add paths for both transmit and receive.

Why it is used: Peers a Linux router with a partner network to exchange routes.

# Bird config
protocol bgp Peering {
    local as 65000;
    neighbor 192.168.1.20 as 65001;

    import all;
    export all;

    add paths tx;
    add paths rx;
}

Interview talking point: “ASNs in the 65000 range are private and never meant for the public internet, and add paths tx and rx let the session carry multiple paths for one prefix instead of a single best path.”

Q1090: How do you configure bonding with LACP?

What it is: LACP bonding uses mode 4 so the switch and host negotiate link aggregation. BONDING_OPTS sets mode=4, miimon=100 and lacp_rate=1, the slave NIC gets MASTER and SLAVE plus speed settings, and /proc/net/bonding/bond0 shows status.

Why it is used: To aggregate switch ports for extra bandwidth with automatic negotiation and failover.

DEVICE=bond0
ONBOOT=yes
BOOTPROTO=none
BONDING_OPTS="mode=4 miimon=100 lacp_rate=1"

# /etc/sysconfig/network-scripts/ifcfg-eth0
DEVICE=eth0
ONBOOT=yes
MASTER=bond0
SLAVE=yes
ETHTOOL_OPTS="speed 1000 duplex full"

# Verify
cat /proc/net/bonding/bond0

Interview talking point: “Mode 4 means LACP, so the switch side must also be configured to negotiate, and cat /proc/net/bonding/bond0 is the quickest way to confirm the bond state and which links are up.”

Q1091: How do you configure bridging?

What it is: A Linux bridge connects multiple network interfaces at Layer 2, functioning like a software switch.

Why it is used: Enable virtual machines and containers to share the same network segment as physical hosts.

# Create bridge
brctl addbr br0
ip addr add 192.168.1.1/24 dev br0
ip link set br0 up

# Add interfaces
brctl addif br0 eth0
brctl addif br0 eth1

# Persistent configuration
# /etc/network/interfaces
auto br0
iface br0 inet static
    address 192.168.1.1
    netmask 255.255.255.0
    bridge_ports eth0 eth1
    bridge_stp off
    bridge_fd 0
    bridge_maxwait 0

# Using iproute2
ip link add name br0 type bridge
ip link set dev eth0 master br0
ip link set dev eth1 master br0

# View
brctl show
ip link show type bridge

Interview talking point: “Bridges forward frames based on MAC tables; STP can prevent loops when multiple bridges connect to the same network.”

Q1092: How do you configure KVM SR-IOV?

What it is: SR-IOV splits a physical NIC into virtual functions: writing to sriov_numvfs creates them, lspci confirms them, and virsh attach-interface assigns a VF to a VM as a hostdev.

Why it is used: Gives VMs near-native network performance with dedicated hardware queues.

# Enable SR-IOV
echo 2 > /sys/bus/pci/devices/0000:01:00.0/sriov_numvfs

# Create VF
# Check
lspci | grep -i virtual

# Assign to VM
virsh attach-interface --domain vm --type hostdev --source 0000:01:00.2 --managed --config

Interview talking point: “A VF passed to a guest bypasses the host network stack, so guest traffic no longer crosses the bridge, which is both the performance gain and the reason host firewall rules stop applying to that traffic.”

Q1093: How do you configure network bonding (advanced)?

What it is: Network bonding joins NICs into one interface. The bonding module is set to mode 4 with miimon and lacp_rate, and nmcli can build a team using the lacp runner instead.

Why it is used: To aggregate link bandwidth and survive a NIC or cable failure.

# Mode 4 (LACP)
cat > /etc/modprobe.d/bonding.conf <<EOF
options bond0 mode=4 miimon=100 lacp_rate=1
EOF

# Team driver (CentOS)
nmcli con add type team ifname team0 con-name team0
nmcli con add type team-slave ifname eth0 master team0
nmcli con add type team-slave ifname eth1 master team0
nmcli con modify team0 team.runner lacp

Interview talking point: “Mode 4 is LACP, which negotiates with the switch, so the partner must support it, whereas the team driver configured through nmcli reaches the same lacp runner with NetworkManager in control.”

Q1094: How do you configure network bonding for high availability?

What it is: Network bonding joins several NICs into one interface for redundancy or throughput, configured with BONDING_OPTS giving a mode, a link monitoring interval, and slave interfaces.

Why it is used: To survive a NIC, cable, or switch failure without losing connectivity.

# Load bonding module
modprobe bonding mode=1 miimon=100

DEVICE=bond0
BONDING_OPTS="mode=1 miimon=100 primary=eth0"
IPADDR=192.168.1.10
NETMASK=255.255.255.0
GATEWAY=192.168.1.1
ONBOOT=yes

# /etc/sysconfig/network-scripts/ifcfg-eth0
DEVICE=eth0
MASTER=bond0
SLAVE=yes
ONBOOT=yes

# /etc/sysconfig/network-scripts/ifcfg-eth1
DEVICE=eth1
MASTER=bond0
SLAVE=yes
ONBOOT=yes

# Mode 4 (LACP)
# /etc/sysconfig/network-scripts/ifcfg-bond0
BONDING_OPTS="mode=4 miimon=100 lacp_rate=1"

# Monitor
cat /proc/net/bonding/bond0

# ethtool
ethtool -S bond0

Interview talking point: “mode=1 is active-backup with a primary interface while mode=4 negotiates LACP with the switch, and /proc/net/bonding/bond0 shows which slave is currently active.”

Q1095: How do you configure network bonding in Linux?

What it is: Network bonding joins several NICs into one interface, here bond0 with an IP, while eth0 is marked as a slave through MASTER and SLAVE settings. BONDING_OPTS sets mode 1, link monitoring every 100 ms, and fail_over_mac.

Why it is used: To keep the host reachable if one NIC or cable fails, and sometimes to add link capacity.

# Create bond interface
cat > /etc/sysconfig/network-scripts/ifcfg-bond0 <<EOF
DEVICE=bond0
ONBOOT=yes
BOOTPROTO=none
IPADDR=192.168.1.10
NETMASK=255.255.255.0
GATEWAY=192.168.1.1
BONDING_OPTS="mode=1 miimon=100 fail_over_mac=2"
EOF

# Configure slave interfaces
cat > /etc/sysconfig/network-scripts/ifcfg-eth0 <<EOF
DEVICE=eth0
MASTER=bond0
SLAVE=yes
ONBOOT=yes
EOF

# Reload network
systemctl restart network

Interview talking point: “The bond carries the address while the physical NICs carry MASTER and SLAVE only, and after editing these files systemctl restart network reloads the whole set at once.”

Q1096: How do you configure network bonding modes?

What it is: Network bonding joins NICs into one interface; writing to /sys/class/net/bond0/bonding/mode selects mode 0 round-robin, mode 1 active-backup, or mode 4 LACP, and miimon sets the link check interval.

Why it is used: Adds redundancy and bandwidth between a server and its switch.

# Mode 0 (Round-robin)
echo 0 > /sys/class/net/bond0/bonding/mode

# Mode 1 (Active-backup)
echo 1 > /sys/class/net/bond0/bonding/mode

# Mode 4 (LACP)
echo 4 > /sys/class/net/bond0/bonding/mode

# Monitoring
echo 100 > /sys/class/net/bond0/bonding/miimon

Interview talking point: “Mode 1 needs no switch support and simply fails over, while mode 4 LACP requires a matching port-channel on the switch; miimon 100 means a dead link is noticed within roughly 100 milliseconds.”

Q1097: How do you configure network bonding modes?

What it is: Network bonding combines multiple NICs into one logical interface using different modes for load balancing or failover.

Why it is used: Increase bandwidth or provide high availability for network connectivity.

# Mode 0: Round-robin
alias bond0 bonding
options bond0 mode=0 miimon=100

# Mode 1: Active-backup
options bond0 mode=1 miimon=100 primary=eth0

# Mode 4: 802.3ad (LACP)
options bond0 mode=4 miimon=100 lacp_rate=1

# Mode 5: Balance-tlb
options bond0 mode=5 miimon=100

# Interface configuration
# /etc/sysconfig/network-scripts/ifcfg-bond0
DEVICE=bond0
BONDING_OPTS="mode=5 miimon=100"
IPADDR=192.168.1.10
NETMASK=255.255.255.0
ONBOOT=yes

# Slave interfaces
# /etc/sysconfig/network-scripts/ifcfg-eth0
DEVICE=eth0
MASTER=bond0
SLAVE=yes
ONBOOT=yes

Interview talking point: “Mode 4 (802.3ad/LACP) requires switch support, while mode 1 (active-backup) provides simple hardware redundancy.”

Q1098: How do you configure network bonding?

What it is: bonding joins several NICs into one interface; bond0 is configured with mode=1 and miimon=100 plus an address, and eth0 becomes a slave through MASTER and SLAVE lines.

Why it is used: To add link redundancy or throughput on a server network connection.

# Create bond interface
cat /etc/sysconfig/network-scripts/ifcfg-bond0
DEVICE=bond0
BONDING_OPTS="mode=1 miimon=100"
IPADDR=192.168.1.10
NETMASK=255.255.255.0

# Add slave
cat /etc/sysconfig/network-scripts/ifcfg-eth0
DEVICE=eth0
MASTER=bond0
SLAVE=yes

Interview talking point: “mode=1 is active-backup failover, and miimon=100 sets link monitoring every 100 milliseconds, so a dead cable is detected quickly and traffic switches to the surviving NIC.”

Q1099: How do you configure network bonding?

What it is: Network bonding joins multiple NICs into one interface for redundancy or throughput, with modes such as active-backup and link health checked by miimon.

Why it is used: To keep a host reachable if one NIC or cable fails.

# Load bonding module
modprobe bonding mode=active-backup

# Create bond interface
cat > /etc/sysconfig/network-scripts/ifcfg-bond0 << EOF
DEVICE=bond0
TYPE=Bond
IPADDR=192.168.1.10
NETMASK=255.255.255.0
ONBOOT=yes
BONDING_OPTS="mode=active-backup miimon=100"
EOF

# Add slave interfaces
echo "MASTER=bond0" >> /etc/sysconfig/network-scripts/ifcfg-eth0
echo "MASTER=bond0" >> /etc/sysconfig/network-scripts/ifcfg-eth1

# View bond status
cat /proc/net/bonding/bond0

Interview talking point: “mode=active-backup fails over to the second NIC while miimon=100 polls link health every 100 milliseconds. cat /proc/net/bonding/bond0 shows which slave is currently active and carrying traffic.”

Q1100: How do you configure network bridging?

What it is: A bridge, created with brctl addbr br0, groups interfaces eth0 and eth1 into one link layer device. The bridge takes the IP address and is brought up with ip link set br0 up.

Why it is used: To let several ports act as one switch segment, commonly for virtual machines or host networking.

# Create bridge
brctl addbr br0
brctl addif br0 eth0
brctl addif br0 eth1

# Configure IP
ip addr add 192.168.1.10/24 dev br0
ip link set br0 up

# Make persistent (CentOS/RHEL)
cat > /etc/sysconfig/network-scripts/ifcfg-br0 <<EOF
DEVICE=br0
TYPE=Bridge
IPADDR=192.168.1.10
NETMASK=255.255.255.0
ONBOOT=yes
EOF

Interview talking point: “The IP address and netmask are configured on br0 while eth0 and eth1 only get enrolled with brctl addif, and TYPE=Bridge in ifcfg makes the bridge persist across reboots.”

Q1101: How do you configure network bridging?

What it is: A network bridge joins several interfaces into one layer 2 device. Both brctl and ip link create one, enslaving eth0 and eth1 to br0, which then carries the IP address.

Why it is used: To let virtual machines or container hosts share physical NICs at layer 2.

# Using brctl
brctl addbr br0
brctl addif br0 eth0
brctl addif br0 eth1
ip addr add 192.168.1.10/24 dev br0

# Using ip
ip link add name br0 type bridge
ip link set eth0 master br0
ip link set eth1 master br0
ip link set br0 up

# Make persistent (systemd)
cat > /etc/systemd/network/25-bridge.network <<EOF
[Match]
Name=br0

[Network]
Address=192.168.1.10/24
Gateway=192.168.1.1
EOF

Interview talking point: “The ip approach uses type bridge plus link set master to enslave ports, and the systemd network file is what persists the address and gateway rather than relying on runtime commands alone.”

Q1102: How do you configure network bridging?

What it is: A bridge links several interfaces at layer two so they behave like one switch; brctl or ip creates br0 and adds eth0 and eth1 to it.

Why it is used: To share one network segment across interfaces, commonly for virtual machines or container hosts.

# Create bridge
brctl addbr br0
ip addr add 192.168.1.1/24 dev br0
ip link set br0 up

# Add interfaces
brctl addif br0 eth0
brctl addif br0 eth1

# View bridge
brctl show
ip link show type bridge

# Delete bridge
ip link set br0 down
brctl delbr br0

Interview talking point: “Bridging forwards layer two frames while a bond merges NICs into one logical device, so bridges connect segments and bonds add redundancy. ip link show type bridge lists bridges without needing the brctl tool.”

Q1103: How do you configure OpenVPN?

What it is: OpenVPN builds a TLS-based VPN. easy-rsa creates the PKI: init-pki and build-ca make the CA, then build-server-full and build-client-full issue certificates copied into /etc/openvpn.

Why it is used: To connect remote clients to a private network through an encrypted tunnel terminated on a central server.

# Install
apt install openvpn easy-rsa

# Setup CA
cd /usr/share/easy-rsa
./easyrsa init-pki
./easyrsa build-ca

# Build server
./easyrsa build-server-full server nopass

# Build client
./easyrsa build-client-full client1 nopass

# Server config
cp pki/ca.crt /etc/openvpn/
cp pki/issued/server.crt /etc/openvpn/
cp pki/private/server.key /etc/openvpn/

# Client config
client
dev tun
remote vpn.example.com 1194
proto udp
ca ca.crt
cert client1.crt
key client1.key

Interview talking point: “The server needs ca.crt, server.crt, and server.key in /etc/openvpn, while the client config names its own cert and key plus remote vpn.example.com 1194 with proto udp. Each peer’s certificate must match the identity the server expects.”

Q1104: How do you configure OpenVPN?

What it is: OpenVPN server and client configuration files. The server listens on UDP 1194 using a tun device, references CA, certificate, key, and DH files, serves a 10.8.0.0 client subnet, and pushes gateway and DNS settings to clients.

Why it is used: To give remote users an encrypted tunnel into an internal network from untrusted locations.

# Server config
port 1194
proto udp
dev tun
ca ca.crt
cert server.crt
key server.key
dh dh.pem
server 10.8.0.0 255.255.255.0
ifconfig-pool-persist ipp.txt
push "redirect-gateway def1 bypass-dhcp"
push "dhcp-option DNS 8.8.8.8"

# Client config
client
dev tun
proto udp
remote vpn.example.com 1194
ca ca.crt
cert client.crt
key client.key

Interview talking point: “OpenVPN identities are X.509 material, so each side carries separate ca, cert, and key files, and the server’s push lines decide the client’s routes and DNS after connecting.”

Q1105: How do you configure OSPF?

What it is: This Bird OSPF config names protocol ospf MyOSPF, sets tick 2, and places the interfaces matching eth in area 0 with type broadcast.

Why it is used: Lets a Linux box learn and flood link-state routes on a broadcast segment.

# Bird config
protocol ospf MyOSPF {
    tick 2;

    area 0 {
        interface "eth*" {
            type broadcast;
        };
    };
}

Interview talking point: “type broadcast tells OSPF to run neighbor discovery with a designated-router election, which matches Ethernet, and area 0 is the backbone that every other OSPF area must connect to.”

Q1106: How do you configure policy routing?

What it is: Policy routing picks routes by rule rather than by destination alone. A named table is added to rt_tables, ip rule matches source or destination, and ip route fills that table.

Why it is used: To send traffic from a particular address down a different path, then masquerade it out.

# Create routing table
echo "200 custom" >> /etc/iproute2/rt_tables

# Add rule
ip rule add from 192.168.1.10 table custom
ip rule add to 192.168.1.10 table custom

# Add route
ip route add 10.0.0.0/24 via 192.168.1.1 table custom

# NAT/Masquerade
iptables -t nat -A POSTROUTING -s 10.0.0.0/24 -o eth0 -j MASQUERADE

# Make persistent
cat > /etc/network/if-up.d/custom-routes <<EOF
#!/bin/sh
ip rule add from 192.168.1.10 table custom
EOF

Interview talking point: “ip rule entries are evaluated in order, so rule priority matters, and the custom table only resolves by name after its number and name are registered in /etc/iproute2/rt_tables.”

Q1107: How do you configure policy routing?

What it is: Policy routing adds named routing tables and ip rules selecting a table by source address, destination, or firewall mark instead of using only the main table.

Why it is used: To send different traffic classes down different paths on the same host.

# Additional routing tables
echo "100 fast" >> /etc/iproute2/rt_tables
echo "200 slow" >> /etc/iproute2/rt_tables

# Add routes to tables
ip route add 192.168.1.0/24 dev eth1 src 192.168.2.1 table fast
ip route add default via 192.168.1.1 table fast

# Policy routing rules
ip rule add from 192.168.2.10 table fast
ip rule add to 192.168.1.0/24 table fast
ip rule add fwmark 1 table slow

# Mark packets
iptables -A PREROUTING -s 192.168.2.10 -j MARK --set-mark 1
iptables -A OUTPUT -s 192.168.2.10 -j MARK --set-mark 1

# Persistent
# /etc/network/interfaces
post-up ip rule add from 192.168.2.10 table fast
post-down ip rule del from 192.168.2.10 table fast

Interview talking point: “ip rule evaluates rules in priority order before falling back to the main table, and the iptables MARK set in PREROUTING and OUTPUT is what the fwmark rule matches on.”

Q1108: How do you configure QoS?

What it is: Traffic shaping with tc: an HTB root qdisc is added on eth0, a class under it is given a rate and ceiling of 100mbit, and further classes can be nested to give different traffic classes their own bandwidth budgets.

Why it is used: To limit or guarantee bandwidth per class so one flow cannot starve others.

# Traffic shaping with tc
# Limit outgoing bandwidth
tc qdisc add dev eth0 root handle 1: htb default 10
tc class add dev eth0 parent 1: classid 1:10 htb rate 100mbit ceil 100mbit
tc class add dev eth0 parent 1: classid

Interview talking point: “HTB distinguishes rate, the guaranteed share, from ceil, the maximum borrowed bandwidth, and tc shaping applies to egress only, so outgoing traffic on eth0 is what gets shaped.”

Q1109: How do you configure Quagga?

What it is: Quagga splits routing into daemons: zebra manages interfaces and addresses in zebra.conf, while ospfd runs OSPF from ospfd.conf with router ospf and its network statements.

Why it is used: Provides OSPF and other routing protocols on Linux systems that predate FRR.

# Install
apt install quagga

# Configure
# /etc/quagga/zebra.conf
interface eth0
 ip address 192.168.1.10/24

# /etc/quagga/ospfd.conf
router ospf
 network 192.168.1.0/24 area 0

Interview talking point: “With Quagga the interface address lives in zebra.conf and the protocol lives in ospfd.conf, and FRR, which forked from Quagga, keeps that same split of a routing manager plus per-protocol daemons.”

Q1110: How do you configure VLAN tagging?

What it is: VLAN tagging adds an 802.1Q ID to frames so one physical link carries several logical networks. vconfig or ip link create a device such as eth0.100 bound to that ID.

Why it is used: To separate traffic classes such as guest, storage, and management on a single NIC.

# Using vconfig
vconfig add eth0 100

# Using ip
ip link add link eth0 name eth0.100 type vlan id 100
ip link set eth0.100 up
ip addr add 192.168.100.10/24 dev eth0.100

# Persistent (CentOS)
cat > /etc/sysconfig/network-scripts/ifcfg-eth0.100 <<EOF
DEVICE=eth0.100
VLAN=yes
ONBOOT=yes
IPADDR=192.168.100.10
NETMASK=255.255.255.0
EOF

Interview talking point: “The IP address lives on the subinterface eth0.100 while tagging happens on parent eth0, and on CentOS persistence comes from a separate ifcfg file carrying VLAN=yes instead of the main interface file.”

Q1111: How do you configure VLAN tagging?

What it is: VLAN tagging creates logical subinterfaces on a physical NIC to isolate network traffic at Layer 2.

Why it is used: Segment networks for security, performance, or multi-tenant environments.

# Create VLAN interface
ip link add link eth0 name eth0.100 type vlan id 100
ip link set eth0.100 up
ip addr add 192.168.100.1/24 dev eth0.100

# Using vconfig
vconfig add eth0 100
ifconfig eth0.100 192.168.100.1 netmask 255.255.255.0 up

# Persistent configuration
# /etc/network/interfaces (Debian)
auto eth0.100
iface eth0.100 inet static
    address 192.168.100.1
    netmask 255.255.255.0
    vlan-raw-device eth0

# RHEL
# /etc/sysconfig/network-scripts/ifcfg-eth0.100
VLAN=yes
DEVICE=eth0.100
PHYSDEV=eth0
VLAN_ID=100
IPADDR=192.168.100.1
NETMASK=255.255.255.0

Interview talking point: “Interfaces like eth0.100 use 802.1Q tagging with a specified VLAN ID on the parent physical interface.”

Q1112: How do you configure VLANs in Linux?

What it is: A VLAN tags Ethernet frames with an ID so one physical NIC carries several logical networks; Linux creates eth0.100 via the 8021q module using ip or vconfig.

Why it is used: To separate traffic for different networks over a single uplink port.

# Enable 8021q module
modprobe 8021q

# Create VLAN interface
ip link add link eth0 name eth0.100 type vlan id 100
ip addr add 192.168.100.1/24 dev eth0.100
ip link set eth0.100 up

# Or using vconfig
vconfig add eth0 100
ifconfig eth0.100 192.168.100.1 netmask 255.255.255.0 up

# Persistent VLAN config (RHEL)
# /etc/sysconfig/network-scripts/ifcfg-eth0.100

Interview talking point: “The VLAN interface name and tag usually match, as with eth0.100 for id 100, and the interface inherits the parent link state. Persistence comes from an ifcfg file rather than the ip command itself.”

Q1113: How do you configure VLANs?

What it is: A VLAN gives one physical port logical sub-interfaces tagged with a VLAN ID. The record shows vconfig adding ID 100 to eth0, an ifcfg file for eth0.100 with its own IP, and the equivalent ip link and ip addr commands.

Why it is used: To carry separate broadcast domains over one cable, such as separating management and guest traffic.

# Install VLAN package
apt install vlan

# Create VLAN interface
vconfig add eth0 100

# Configure
cat > /etc/sysconfig/network-scripts/ifcfg-eth0.100 <<EOF
DEVICE=eth0.100
VLAN=yes
ONBOOT=yes
IPADDR=192.168.100.10
NETMASK=255.255.255.0
EOF

# Or use ip command
ip link add link eth0 name eth0.100 type vlan id 100
ip addr add 192.168.100.10/24 dev eth0.100

Interview talking point: “The VLAN shows up as a normal interface named eth0.100 with its own address, and you can create it either persistently in ifcfg or on the fly with ip link add link eth0 name eth0.100 type vlan id 100.”

Q1114: How do you configure VPN?

What it is: An OpenVPN setup built with easy-rsa creating a CA and certificates, a server config assigning a 10.8.0.0 tunnel network, and a matching client config.

Why it is used: To give remote users encrypted access to internal networks as if they were on site.

# OpenVPN
apt install openvpn easy-rsa

# Generate keys
cd /etc/openvpn/easy-rsa
./easyrsa init-pki
./easyrsa build-ca
./easyrsa build-server-full server nopass
./easyrsa build-client-full client1 nopass

# Server config
# /etc/openvpn/server.conf
port 1194
proto udp
dev tun
ca ca.crt
cert server.crt
key server.key
dh dh.pem
server 10.8.0.0 255.255.255.0
push "redirect-gateway def1 bypass-dhcp"
push "dhcp-option DNS 8.8.8.8"
keepalive 10 120
cipher AES-256-GCM
auth SHA256
persist-key
persist-tun
status openvpn-status.log

# Client config
client
dev tun
proto udp
remote vpn.example.com 1194
resolv-retry infinite
nobind
persist-key
persist-tun
remote-cert-tls server
cipher AES-256-GCM
auth SHA256

Interview talking point: “Both sides must agree on cipher and auth, here AES-256-GCM with SHA256, and the server pushes redirect-gateway and DNS so a connected client routes all traffic through the tunnel.”

Q1115: How do you configure WireGuard?

What it is: WireGuard is a modern VPN configured in /etc/wireguard/wg0.conf. wg genkey derives a private key and pipes its public key out, and each side lists the peer key and AllowedIPs.

Why it is used: To build a fast, low-configuration encrypted tunnel between hosts or for remote access.

# Install
apt install wireguard

# Generate keys
wg genkey | tee private.key | wg pubkey > public.key

# Server config /etc/wireguard/wg0.conf
[Interface]
Address = 10.0.0.1/24
ListenPort = 51820
PrivateKey = <server-private-key>

[Peer]
PublicKey = <client-public-key>
AllowedIPs = 10.0.0.2/32

# Client config
[Interface]
Address = 10.0.0.2/24
PrivateKey = <client-private-key>

[Peer]
PublicKey = <server-public-key>
Endpoint = vpn.example.com:51820
AllowedIPs = 0.0.0.0/0
PersistentKeepalive = 25

Interview talking point: “The client sets AllowedIPs 0.0.0.0/0 to send all traffic through the tunnel and uses PersistentKeepalive 25 to stay reachable behind NAT. The server instead pins each peer with a narrow /32 AllowedIPs entry such as 10.0.0.2/32.”

Q1116: How do you configure WireGuard?

What it is: WireGuard VPN configuration with an Interface section holding keys, address, and listen port, and Peer sections holding the remote key, endpoint, and allowed IP ranges. The server uses 10.0.0.1 and the client 10.0.0.2.

Why it is used: To build a fast encrypted tunnel between a client and server with minimal configuration.

# Server
[Interface]
Address = 10.0.0.1/24
ListenPort = 51820
PrivateKey = <server-private-key>

[Peer]
PublicKey = <client-public-key>
AllowedIPs = 10.0.0.2/32

# Client
[Interface]
Address = 10.0.0.2/24
PrivateKey = <client-private-key>

[Peer]
PublicKey = <server-public-key>
Endpoint = vpn.example.com:51820
AllowedIPs = 0.0.0.0/0

Interview talking point: “AllowedIPs is the routing control: the server lists only the client address, while the client’s 0.0.0.0/0 sends all of its traffic through the tunnel.”

Q1117: How do you implement Quality of Service?

What it is: Traffic control with tc attaches a qdisc to an interface, classes define bandwidth rates, and filters match packets to direct them into a class.

Why it is used: To guarantee bandwidth for critical traffic like SSH while capping bulk transfers.

# Traffic control with tc
# Create qdisc
tc qdisc add dev eth0 root handle 1: htb default 10

# Create classes
tc class add dev eth0 parent 1: classid 1:10 htb rate 10Mbit ceil 10Mbit
tc class add dev eth0 parent 1: classid 1:20 htb rate 5Mbit ceil 5Mbit

# Filter traffic
tc filter add dev eth0 parent 1: protocol all prio 1 u32 match ip dst 192.168.1.10 flowid 1:20

# Example: Prioritize SSH
tc qdisc add dev eth0 root handle 1: prio
tc filter add dev eth0 parent 1: protocol ip prio 10 u32 match ip dport 22 0xffff flowid 1:2
tc filter add dev eth0 parent 1: protocol ip prio 20 u32 match ip sport 22 0xffff flowid 1:2

# View
tc qdisc show
tc class show
tc filter show

Interview talking point: “HTB classes carry a rate and a ceil, where rate is the guaranteed share and ceil is the burstable maximum, and a prio qdisc prioritizes SSH via filters.”

Q1118: How do you use BandwidthD?

What it is: BandwidthD is configured with a subnet directive that scopes charting to a specific network, then started as a service.

Why it is used: To chart bandwidth usage of a chosen subnet over days and weeks.

# Install
apt install bandwidthd

# Configure
# /etc/bandwidthd/bandwidthd.conf
subnet 192.168.1.0/24

# Start
systemctl start bandwidthd

Interview talking point: “Because charting is scoped by the subnet directive, bandwidthd reports only on the range you list, which suits watching one LAN segment rather than the whole host.”

Q1119: How do you use Bird routing?

What it is: Bird is a lightweight routing daemon whose config declares protocol static for fixed routes, protocol direct for connected interfaces, and protocol kernel to install routes into the kernel table.

Why it is used: Exchanges or injects routes on a Linux router without a full routing suite.

# Install
apt install bird

# Configure
# /etc/bird/bird.conf
protocol static {
    route 10.0.0.0/8 via 192.168.1.1;
}

protocol direct {
    interface "eth*";
}

protocol kernel {
    learn;
    export all;
}

Interview talking point: “The protocol kernel block with export all is what actually pushes Bird’s routes into the kernel, so a perfectly valid static route can sit unused if that export step is missing.”

Q1120: How do you use Bird2?

What it is: Bird2 is the second generation of the Bird daemon: bird.conf sets the router id, declares a static protocol, and declares protocol bgp with a local AS, neighbor AS, and import and export policy.

Why it is used: Speaks BGP and static routing from a Linux host, common at edges and in lab fabrics.

# Install
apt install bird2

# Configure
# /etc/bird/bird.conf
router id 192.168.1.10;

protocol static {
    route 10.0.0.0/8 via 192.168.1.1;
}

protocol bgp AS65000 {
    neighbor 192.168.1.20 as AS65001;
    import all;
    export all;
}

Interview talking point: “Bird2 writes BGP as protocol bgp NAME and takes import and export lines right in the block, so tightening route policy later means editing the same import and export machinery rather than adding a new tool.”

Q1121: How do you use bmon?

What it is: bmon is a terminal graphing tool for per-interface throughput; bmon -p eth0 selects the interface and -b -o interactive chooses the interactive output mode.

Why it is used: Watches link rates and interface counters together on one screen.

# Install
apt install bmon

# Usage
bmon
bmon -p eth0
bmon -b -o interactive

Interview talking point: “bmon reports interface counters and rates rather than per-flow detail, so it answers whether a NIC is saturated, while questions about individual flows still need iftop or nethogs to break the number down.”

Q1122: How do you use conntrack?

What it is: conntrack is the userspace front end to the kernel connection tracker: -L lists entries optionally filtered by protocol, -C counts them, and -D deletes matching flows.

Why it is used: Inspects live connection state, debugs NAT, and clears stuck flows.

# View connections
conntrack -L
conntrack -L -p tcp

# Count
conntrack -C

# Delete
conntrack -D -p tcp --dport 80

# NAT
conntrack -L -n | grep NAT

Interview talking point: “Because NAT happens inside the tracker, conntrack -L -n filtered for NAT shows translated addresses that firewall rules alone never reveal, and -C gives the current entry count you compare against the table limit.”

Q1123: How do you use Darkstat?

What it is: Darkstat’s interface and daemon flag live in init.cfg, and the service is started with systemctl to begin capturing on that interface.

Why it is used: To keep a lightweight continuous view of bandwidth over time.

# Install
apt install darkstat

# Configure
# /etc/darkstat/init.cfg
INTERFACE="-i eth0"
START_DAEMON=1

# Start
systemctl start darkstat

Interview talking point: “The interface choice is set in init.cfg rather than in the systemd unit, so changing interfaces means editing that file and restarting the service.”

Q1124: How do you use DPDK?

What it is: DPDK sets aside hugepages through the kernel sysfs file, binds the NIC to a UIO driver with dpdk-devbind.py, then runs testpmd with core and memory channel flags.

Why it is used: To benchmark packet processing at very high rates in userspace.

# Install
# From dpdk.org

# Setup huge pages
echo 1024 > /sys/kernel/mm/hugepages/hugepages-2048kB/nr_hugepages

# Bind NIC
dpdk-devbind.py --bind igb_uio 0000:01:00.0

# Run testpmd
./build/app/testpmd -l 0-3 -n 4 -- -i

Interview talking point: “DPDK bypasses the kernel network stack, so two prerequisites come first: allocate hugepages and bind the NIC to the DPDK driver, otherwise testpmd has nothing to run on.”

Questions 481-1000 continue in next file…

Q1125: How do you use DPDK?

What it is: DPDK is a user space packet processing framework built with make config and make, and it needs hugepages reserved through kernel sysfs.

Why it is used: To process network packets at very high rates by bypassing the kernel stack.

# Install DPDK
# Download from dpdk.org

# Setup
export DPDK_DIR=/path/to/dpdk
export RTE_SDK=$DPDK_DIR
export RTE_TARGET=x86_64-native-linuxapp-gcc

# Build
make config T=$RTE_TARGET
make -j$(nproc)

# Huge pages
echo 1024 > /sys/kernel/mm/hugepages/hugepages-2048kB/nr_hugepages

Interview talking point: “The hugepages step reserves 2048 kB pages because DPDK leans on huge pages to cut TLB misses in its memory pools.”

Q1126: How do you use ebtables?

What it is: ebtables filters and translates frames on Linux bridges at layer 2; ebtables -L lists rules, -A appends a frame match such as IPv4 TCP port 80, and the nat table can redirect.

Why it is used: Controls traffic crossing a bridge, which layer 3 filtering does not see in the same way.

# List
ebtables -L

# Bridge filter
ebtables -A FORWARD -p IPv4 --ip-proto tcp --ip-dport 80 -j DROP

# NAT
ebtables -t nat -A PREROUTING -i eth0 -j redirect --redirect-target ACCEPT

Interview talking point: “The distinction is layer: iptables governs routed packets while ebtables governs bridged frames before routing, so traffic crossing a bridge that never routes will not be touched by an iptables FORWARD rule.”

Q1127: How do you use FRRouting?

What it is: FRRouting is a suite of routing daemons: /etc/frr/daemons enables ospfd, and /etc/frr/ospfd.conf declares router ospf plus the network statements defining which interfaces join area 0.

Why it is used: Runs OSPF or other dynamic protocols on a Linux router or switch.

# Install
apt install frr

# Enable
# /etc/frr/daemons
ospfd=yes

# Configure
# /etc/frr/ospfd.conf
router ospf
 network 192.168.1.0/24 area 0

Interview talking point: “Daemons are toggled individually in /etc/frr/daemons, so installing frr does nothing until ospfd=yes is set, and a network statement must match an interface address or that interface stays out of OSPF entirely.”

Q1128: How do you use IPFIX?

What it is: IPFIX is the standardized flow record format; you install ipfixprobe, configure an exporter to send records, and let Elasticsearch or another collector store and query them.

Why it is used: To export and store standardized flow records for traffic analysis.

# Install
apt install ipfixprobe

# Capture
# Configure exporter

# Analyze
# Use elasticsearch or other collectors

Interview talking point: “IPFIX is the IETF standard behind flow data, so the pieces are an exporter sending records and a collector ingesting them, with Elasticsearch serving the queries.”

Q1129: How do you use iproute2 for advanced routing?

What it is: iproute2 can install multipath default routes with per-nexthop weights, several nexthops for one prefix, and blackhole routes that silently discard traffic. ip monitor route streams routing table changes.

Why it is used: To balance traffic over uplinks and to drop unwanted prefixes without a firewall rule.

# Load balancing
ip route add default scope global \
  nexthop via 192.168.1.1 dev eth0 weight 1 \
  nexthop via 192.168.2.1 dev eth1 weight 1

# Multipath
ip route add 10.0.0.0/8 \
  nexthop via 192.168.1.1 dev eth0 \
  nexthop via 192.168.2.1 dev eth1

# Blackhole route
ip route add blackhole 10.0.0.0/8

# Monitor
ip monitor route

Interview talking point: “A weighted default route spreads traffic across eth0 and eth1, while a blackhole route discards packets silently, which shows up as a timeout rather than a rejection and is easy to misread.”

Q1130: How do you use iptraf-ng?

What it is: iptraf-ng is a curses-based IP traffic monitor; iptraf-ng -i all or -i eth0 opens a per-interface view of current connections and rates.

Why it is used: Gets a quick breakdown of traffic by address and protocol straight from a terminal.

# Install
apt install iptraf-ng

# Usage
iptraf-ng
iptraf-ng -i all
iptraf-ng -i eth0

Interview talking point: “iptraf-ng sits between the flow view of iftop and the counters of bmon with menu-driven protocol and detail screens, which is handy when you want breakdowns without assembling several smaller tools.”

Q1131: How do you use Linux traffic control?

What it is: Traffic control (tc) attaches qdiscs to an interface to shape traffic. netem adds delay, while tbf caps rate at 1mbit with a burst and latency budget.

Why it is used: To simulate latency or cap bandwidth when testing how applications behave on constrained links.

# Add qdisc
tc qdisc add dev eth0 root netem delay 100ms

# Rate limiting
tc qdisc add dev eth0 root tbf rate 1mbit burst 1540 latency 50ms

# View
tc qdisc show
tc -s qdisc show

Interview talking point: “tc qdisc show lists the qdiscs on each device, while tc -s qdisc show adds per-qdisc counters so drops and usage are visible. netem shapes by delay and tbf shapes by rate, and both are installed as the root qdisc.”

Q1132: How do you use Moloch?

What it is: Moloch runs in Docker with an Elasticsearch URL and ports 8000 and 9200, and its capture settings live in a config.ini under etc.

Why it is used: To keep full packet captures searchable for forensic investigation.

# Install
# Using Docker
docker run -d --name moloch \
  -e ELASTICSEARCH=http://localhost:9200 \
  -p 8000:8000 \
  -p 9200:9200 \
  aircan/moloch

# Configure capture
# /data/moloch/etc/config.ini

Interview talking point: “Because Moloch indexes capture metadata in Elasticsearch, the Elasticsearch environment variable at startup is what links the stored packet archive to the searchable web viewer.”

Q1133: How do you use Netflow analyzer?

What it is: nfcapd captures NetFlow records on an interface into dated files, and nfdump reads those files to aggregate by IP or filter on a source subnet.

Why it is used: To capture and rank flow records by IP to find heavy talkers.

# Install nfdump
apt install nfdump nfcapd

# Capture
nfcapd -w -D -i eth0 -z -n "ip,/var/cache/netflow,YYYYMMDD,HHMM"

# Analyze
nfdump -r nfcapd.202301011300 -a -s ip/bytes

# Filter
nfdump -r nfcapd.202301011300 'src ip 192.168.1.0/24'

Interview talking point: “nfcapd writes time-stamped capture files and nfdump reads them afterwards, so aggregation with the sort key and filtering by source network both work offline on saved data.”

Q1134: How do you use netsniff-ng?

What it is: netsniff-ng captures from an interface into a pcap, can replay from a pcap with -r, and takes a BPF filter with -f such as tcp port 80.

Why it is used: To capture or re-send filtered packets quickly during performance testing.

# Install
apt install netsniff-ng

# Capture
netsniff-ng -i eth0 -o capture.pcap

# Replay
netsniff-ng -i eth0 -r capture.pcap

# BPF filter
netsniff-ng -i eth0 -f "tcp port 80" -o http.pcap

Interview talking point: “The -f option accepts standard BPF syntax, so the same tcp port 80 expression used with tcpdump narrows what lands in the output capture file.”

Q1135: How do you use NFQueue?

What it is: NFQUEUE diverts packets from iptables to a userspace queue; the sample Python program binds queue 0, prints each packet’s data, and returns a verdict of NF_ACCEPT.

Why it is used: Hands packets to custom programs for inspection or decisions beyond what static rules can express.

# Mark packets
iptables -A INPUT -j NFQUEUE --queue-num 0

# Process with Python
#!/usr/bin/env python3
import NFQueue
def callback(payload):
    print(payload.get_data())
    payload.set_verdict(NFQueue.NF_ACCEPT)

q = NFQueue.Queue()
q.set_callback(callback)
q.open()
q.bind()
q.try_run()

Interview talking point: “The verdict is mandatory, since a queued packet stays frozen until userspace answers, and queue-num 0 must match the queue the program actually binds or no packets ever arrive for it to process.”

Q1136: How do you use ntop?

What it is: ntop runs with -i naming the interface to analyze and then serves its results over HTTP on port 3000.

Why it is used: To see live per-protocol and per-host traffic on an interface.

# Install
apt install ntop

# Run
ntop -i eth0

# Access
# http://localhost:3000

Interview talking point: “ntop does both collection and presentation in one process, so the -i flag decides what is analyzed and the same tool later serves the browsable report.”

Q1137: How do you use ntopng?

What it is: ntopng is configured with a pid file, the interface to watch, and the HTTP port, started as a service and then opened at localhost:3000.

Why it is used: To watch who is using the network through a web dashboard.

# Install
apt install ntopng

# Configure
# /etc/ntopng/ntopng.conf
-G=/var/run/ntopng.pid
-i=eth0
-w=3000

# Start
systemctl start ntopng

# Access
# http://localhost:3000

Interview talking point: “ntopng is launched and supervised with systemctl like any daemon, and the -w value sets the web port you then open in a browser for the live view.”

Q1138: How do you use PF_RING?

What it is: PF_RING is a kernel module loaded with modprobe, verified by reading the info file under /proc/net/pf_ring, and then used by capture applications such as tcpdump.

Why it is used: To capture high-rate traffic without dropping packets in the kernel path.

# Install
# Download from ntop.org

# Load module
modprobe pf_ring

# Check
cat /proc/net/pf_ring/info

# Use with applications
# tcpdump -i eth0

Interview talking point: “modprobe is what brings PF_RING in, and the /proc/net/pf_ring/info file is the quick check that the module is really loaded before any application relies on it.”

Q1139: How do you use pktstat?

What it is: pktstat displays live packet statistics by protocol or host; -i eth0 selects the interface, -T shows timing, and -W widens the display.

Why it is used: Quickly identifies which traffic types are hitting a link at a given moment.

# Install
apt install pktstat

# Usage
pktstat -i eth0
pktstat -i eth0 -T  # show times
pktstat -i eth0 -w  # wide

Interview talking point: “pktstat breaks traffic down by protocol and message type rather than by process or connection pair, so it complements nethogs when you want to know what the packets are rather than who produced them.”

Q1140: How do you use RDMA?

What it is: rdma-core provides RDMA tooling; rdma link lists devices and ib_write_bw measures bandwidth against a specific device such as mlx5_0.

Why it is used: To benchmark network cards that bypass the kernel with low latency, high throughput transfers.

# Install
apt install rdma-core

# Check devices
rdma link

# Configure
# Use Mellanox or Intel RDMA cards

# Test
ib_write_bw -d mlx5_0 -a

Interview talking point: “ib_write_bw takes a device flag, so the test targets one NIC, and running it with -a reports all statistics rather than a short summary.”

Q1141: How do you use sFlow?

What it is: sFlow is sampled flow data from switches; sflowtool listens on an interface or reads a saved dat file and can emit JSON for other tools.

Why it is used: To collect sampled flow data from switches and inspect it locally.

# Install
apt install sflowtool

# Configure switch
# sflow collector <ip> <port>

# Analyze
sflowtool -l eth0
sflowtool -r sflow.dat -json

Interview talking point: “sflowtool -l receives live samples from the network while -r replays a saved capture file, and the JSON output is what lets a downstream collector ingest the data.”

Q1142: How do you use SR-IOV?

What it is: SR-IOV splits a physical NIC into virtual functions: enable intel_iommu in GRUB, set sriov_numvfs, then attach a function to a VM with virsh.

Why it is used: To give virtual machines direct, near native access to network hardware.

# Enable in BIOS

# Enable in kernel
# Add to GRUB: intel_iommu=on

# Configure
echo 2 > /sys/bus/pci/devices/0000\:01\:00.0/sriov_numvfs

# Verify
lspci | grep -i virtual

# Assign to VM
virsh attach-interface --domain vm --type hostdev --source 0000:01:00.2 --managed

Interview talking point: “Writing 2 to sriov_numvfs creates two virtual functions, and each function is then passed to a guest as a hostdev device for dedicated throughput.”

Q1143: How do you use tc qdisc?

What it is: tc manages queueing disciplines: an HTB root qdisc with default class 10, an HTB class shaped to 100mbit, and a u32 filter steering a source subnet into that class.

Why it is used: Applies bandwidth limits and per-flow traffic classes on an interface.

# Add htb qdisc
tc qdisc add dev eth0 root handle 1: htb default 10

# Add class
tc class add dev eth0 parent 1: classid 1:10 htb rate 100mbit ceil 100mbit

# Add filter
tc filter add dev eth0 parent 1: protocol all prio 1 u32 match ip src 192.168.1.0/24 flowid 1:10

Interview talking point: “The three objects layer together, with the qdisc as the root scheduler, classes holding the rates, and the filter deciding which packets land where, so a class rate never takes effect until a filter matches traffic.”

Q1144: How do you use tc shaping?

What it is: tc shaping here uses tbf to cap a device at 10mbit with burst and latency, and netem to add delay, packet loss, or reordering on the root qdisc.

Why it is used: Rate-limits links and deliberately degrades them to test how applications behave.

# Rate limit
tc qdisc add dev eth0 root tbf rate 10mbit burst 15k latency 50ms

# Delay
tc qdisc add dev eth0 root netem delay 100ms

# Loss
tc qdisc add dev eth0 root netem loss 10%

# Reorder
tc qdisc add dev eth0 root netem delay 100ms reorder 25%

Interview talking point: “tbf is a rate limiter while netem is an impairment tool, and options like delay with reorder combine in one netem qdisc, which makes tc the easiest way to simulate a poor WAN inside a lab.”

Q1145: How do you use tcpdump?

What it is: tcpdump captures packets on an interface with host, port, and protocol filters, writes pcaps with -w, reads them with -r, and supports header-level expressions.

Why it is used: To grab live traffic at the command line when a box has no GUI.

# Capture packets
tcpdump -i eth0

# Filter by host
tcpdump host 192.168.1.1

# Filter by port
tcpdump port 80
tcpdump src port 443

# Filter by protocol
tcpdump icmp
tcpdump tcp
tcpdump udp

# Write to file
tcpdump -w capture.pcap

# Read from file
tcpdump -r capture.pcap
tcpdump -r capture.pcap | grep "pattern"

# Advanced filters
tcpdump 'tcp[tcpflags] & (tcp-syn|tcp-fin) != 0'
tcpdump -i eth0 'port 80 and host 192.168.1.1'

Interview talking point: “The tcpflags expression selects SYN or FIN packets by reading header bytes directly, which shows tcpdump filtering goes well beyond simple host and port matches.”

Q1146: How do you use tcpreplay?

What it is: tcpreplay sends a saved pcap back out an interface; -l repeats the capture a set number of times and -M scales the replay speed.

Why it is used: To re-send captured traffic for testing network devices and IDS rules.

# Install
apt install tcpreplay

# Replay pcap
tcpreplay -i eth0 capture.pcap

# Loop
tcpreplay -i eth0 -l 10 capture.pcap

# Speed
tcpreplay -i eth0 -M 10 capture.pcap

Interview talking point: “The options do different jobs: -l replays the file a fixed number of times, while -M multiplies how fast packets are sent, which matters for stressing a link.”

Q1147: How do you use teamd?

What it is: teamd is the userspace teaming daemon: it runs a runner such as LACP, adds ports with teamdctl, assigns addresses with ip, and reports health through teamdctl state.

Why it is used: Bonds NICs with a modern alternative to kernel bonding that is controlled entirely from userspace.

# Create team
teamd -o -n -g team0 -d '{"runner": {"name": "lacp"}}'

# Add ports
teamdctl team0 port add eth1
teamdctl team0 port add eth2

# Configure IP
ip addr add 192.168.1.10/24 dev team0

# Monitor
teamdctl team0 state

Interview talking point: “With teamd the runner policy and state live in a userspace process, so teamdctl state shows live port health directly, while the team interface itself still accepts an ordinary ip addr add like any other NIC.”

Q1148: How do you use trafshow?

What it is: trafshow is a curses traffic monitor; trafshow -i eth0 watches an interface and -c tcp narrows the display to TCP connections.

Why it is used: Observes live connection activity filtered by protocol while working in a shell.

# Install
apt install trafshow

# Usage
trafshow -i eth0
trafshow -i eth0 -c tcp

Interview talking point: “Filtering by protocol with -c shrinks a noisy screen down to a single traffic class, which is useful when the investigation is scoped to TCP sessions only.”

Q1149: How do you use Wireshark?

What it is: tshark is the terminal side of Wireshark, capturing to pcap with filters, reading files back, extracting protocol statistics, and feeding a remote tcpdump stream.

Why it is used: To capture and inspect packet traffic when application logs do not explain a problem.

# Install
apt install wireshark tshark

# Capture with tshark
tshark -i eth0 -w capture.pcap
tshark -i eth0 -f "tcp port 80" -w http.pcap

# Read pcap
tshark -r capture.pcap
tshark -r capture.pcap -Y "http.request"  # Filter

# Extract HTTP
tshark -r capture.pcap -z "http,tree"

# Remote capture
ssh user@host "tcpdump -i eth0 -w -" | wireshark -k -i -

Interview talking point: “The -f filter applies while capturing, whereas -Y filters on read, so a display filter for http.request narrows an already saved pcap without recapturing the traffic.”

Q1150: How do you use Zeek (Bro)?

What it is: Zeek runs on an interface and writes protocol logs to its current log directory, and zeek-cut extracts a field that can be piped through sort and uniq.

Why it is used: To build protocol-aware network logs for security and troubleshooting.

# Install
apt install zeek

# Run
zeek -i eth0

# Logs
ls /var/log/zeek/current/

# Analysis
zeek-cut < conn.log | awk '{print $5}' | sort | uniq -c | sort -rn

Interview talking point: “Zeek produces structured protocol logs rather than packet captures, and zeek-cut pulls a named column so ranking values is an awk, sort, uniq pipeline instead of manual parsing.”

Q1151: How do you configure tunneling?

What it is: A tunnel wraps one network protocol inside another. ip tunnel builds GRE or IPIP links between a local and remote address, while WireGuard generates keys and OpenVPN configures a tun device.

Why it is used: To carry private traffic across an untrusted network or link distant sites together.

# GRE tunnel
ip tunnel add gre0 mode gre remote 192.168.1.10 local 192.168.1.20
ip link set gre0 up
ip addr add 10.0.0.1/30 dev gre0

# IPIP tunnel
ip tunnel add ipip0 mode ipip remote 192.168.1.10 local 192.168.1.20
ip link set ipip0 up

# WireGuard
wg genkey | tee private.key | wg pubkey > public.key

# OpenVPN tunnel
openvpn --dev tun0 --ifconfig 10.0.0.1 10.0.0.2

Interview talking point: “GRE carries any IP protocol and needs only local and remote endpoints, while the WireGuard step shown only produces a private and public key pair before any tunnel interface exists.”

Q1152: How do you configure VPN security?

What it is: WireGuard is a VPN configured through an ini-style file with an Interface section and Peer sections holding keys and addresses. wg-quick brings the tunnel up.

Why it is used: To encrypt traffic between hosts, or to route all client traffic through a server securely.

# WireGuard setup
# Generate keys
wg genkey | tee private.key | wg pubkey > public.key

# Server configuration
# /etc/wireguard/wg0.conf
[Interface]
PrivateKey = <server-private-key>
Address = 10.0.0.1/24
ListenPort = 51820
PostUp = iptables -A FORWARD -i wg0 -j ACCEPT
PostUp = iptables -A FORWARD -o wg0 -j ACCEPT
PostUp = iptables -t nat -A POSTROUTING -o eth0 -j MASQUERADE
PostDown = iptables -D FORWARD -i wg0 -j ACCEPT
PostDown = iptables -D FORWARD -o wg0 -j ACCEPT
PostDown = iptables -t nat -D POSTROUTING -o eth0 -j MASQUERADE

[Peer]
PublicKey = <client-public-key>
AllowedIPs = 10.0.0.2/32

# Client configuration
[Interface]
PrivateKey = <client-private-key>
Address = 10.0.0.2/24

[Peer]
PublicKey = <server-public-key>
Endpoint = server.example.com:51820
AllowedIPs = 0.0.0.0/0
PersistentKeepalive = 25

wg-quick up wg0

Interview talking point: “AllowedIPs controls what routes cross the tunnel, so 0.0.0.0/0 on the client sends everything via the server while a /32 routes only that one peer address.”

Advanced Storage Systems

Q1153: How do you configure Ceph storage?

What it is: Ceph is distributed storage with monitors, OSDs, and metadata servers deployed with ceph-deploy. Pools back a Ceph filesystem that clients mount either through the kernel or with ceph-fuse.

Why it is used: To provide scalable block, object, or file storage that survives the loss of disks or nodes.

# Install
apt install ceph-mon ceph-osd ceph-mds

# Create cluster
ceph-deploy new mon1 osd1 osd2

# Deploy monitors
ceph-deploy mon create mon1

# Deploy OSDs
ceph-deploy osd create --data /dev/sdb1 mon1
ceph-deploy osd create --data /dev/sdb1 osd1

# Create filesystem
ceph osd pool create cephfs_data 128
ceph osd pool create cephfs_metadata 128
ceph fs new cephfs cephfs_metadata cephfs_data

# Mount
# Kernel
mount -t ceph mon1:6789:/ /mnt/ceph

# FUSE
ceph-fuse -n client.admin /mnt/ceph

Interview talking point: “A Ceph filesystem needs both a metadata and a data pool, and clients can mount with the kernel driver for speed or ceph-fuse when kernel support is unavailable.”

Q1154: How do you configure Ceph?

What it is: Ceph combines monitors, OSDs, managers, and metadata servers; disks are prepared as OSDs, a pool is created, and the cluster is mounted with the ceph type.

Why it is used: To provide scalable block, file, and object storage across a cluster.

# Install
apt install ceph-mon ceph-osd ceph-mgr ceph-mds

# Create monitor
ceph-mon --mkfs -i node1 --keyring /tmp/ceph.mon.keyring
systemctl start ceph-mon@node1

# Create OSD
ceph-disk prepare --data /dev/sdb
ceph-disk activate /dev/sdb1

# Create pool
ceph osd pool create mypool 100

# Mount
mount -t ceph node1:/ /mnt/ceph

Interview talking point: “The pool creation command takes a placement group count, and preparing the raw disk as an OSD is the step that actually adds usable capacity to the cluster.”

Q1155: How do you configure distributed storage?

What it is: GlusterFS creates a distributed filesystem by aggregating storage bricks across servers in a trusted pool.

Why it is used: Provide scalable, shared storage with replication for redundancy.

# GlusterFS installation
apt install glusterfs-server

# Add trusted pool
gluster peer probe server2
gluster peer probe server3
gluster peer status

# Create volume
gluster volume create gv0 \
  replica 3 \
  server1:/brick1/server \
  server2:/brick1/server \
  server3:/brick1/server

# Start volume
gluster volume start gv0

# Mount
mount -t glusterfs server1:/gv0 /mnt/glusterfs

# Volume options
gluster volume set gv0 performance.cache-size 256MB
gluster volume set gv0 network.ping_timeout 10

# Rebalance
gluster volume rebalance gv0 start

Interview talking point: “Replicated volumes protect against node failure; rebalancing redistributes data when bricks are added or changed.”

Q1156: How do you configure DRBD?

What it is: DRBD replicates a block device between two nodes over TCP using protocol C and HMAC authentication, defined per node, initialized with drbdadm, then formatted.

Why it is used: To keep an identical block-level copy on a second node for fast failover.

# Install
apt install drbd-utils

global {
    usage-count yes;
}

common {
    protocol C;
    handlers {
        pri-on-incon-degr "/usr/lib/drbd/notify-pri-on-incon-degr.sh; /usr/lib/drbd/notify-emergency-reboot.sh; echo b > /proc/sysrq-trigger ; reboot -f";
    }
    net {
        cram-hmac-alg sha1;
        shared-secret "mysecret";
    }
}

# /etc/drbd.d/r0.res
resource r0 {
    on node1 {
        device /dev/drbd0;
        disk /dev/sdb1;
        address 192.168.1.10:7789;
        meta-disk internal;
    }
    on node2 {
        device /dev/drbd0;
        disk /dev/sdb1;
        address 192.168.1.11:7789;
        meta-disk internal;
    }
}

# Initialize
drbdadm create-md r0
drbdadm up r0
drbdadm primary --force r0

# Filesystem
mkfs.ext4 /dev/drbd0

Interview talking point: “Protocol C completes writes only after both nodes acknowledge, trading latency for no data loss, and drbdadm primary —force promotes the node that starts the initial sync.”

Q1157: How do you configure DRBD?

What it is: DRBD replicates a block device between two nodes over the network. The resource file pairs the same local disk and address on each node; drbdadm creates metadata, brings the resource up, and forces primary role.

Why it is used: To keep a block-level copy of a disk on a second node for fast failover.

resource r0 {
    on node1 {
        device /dev/drbd0;
        disk /dev/sdb1;
        address 192.168.1.10:7789;
        meta-disk internal;
    }
    on node2 {
        device /dev/drbd0;
        disk /dev/sdb1;
        address 192.168.1.11:7789;
        meta-disk internal;
    }
}

# Initialize
drbdadm create-md r0
drbdadm up r0
drbdadm primary --force r0

Interview talking point: “DRBD is block replication rather than file-level sync: both nodes expose the same device and exactly one holds the primary role while the other mirrors it.”

Q1158: How do you configure DRBD?

What it is: DRBD replicates a block device between two nodes over the network, presenting drbd0 on each side. After creating metadata and bringing the resource up, one node is forced primary, gets a filesystem, and is mounted.

Why it is used: To give cluster nodes a shared-looking block device without expensive SAN hardware.

# Install
apt install drbd-utils

resource web {
    protocol C;

    on node1 {
        device /dev/drbd0;
        disk /dev/sdb1;
        address 192.168.1.10:7788;
        meta-disk internal;
    }

    on node2 {
        device /dev/drbd0;
        disk /dev/sdb1;
        address 192.168.1.11:7788;
        meta-disk internal;
    }
}

# Initialize
drbdadm create-md web
drbdadm up web

# Primary
drbdadm primary --force web

# Filesystem
mkfs.xfs /dev/drbd0
mount /dev/drbd0 /var/www

# Status
cat /proc/drbd
drbdadm status

Interview talking point: “Protocol C means writes complete only after both nodes acknowledge, so replication is synchronous, and only the primary side may carry a mounted filesystem while the secondary stays read-only.”

Q1159: How do you configure DRBD?

What it is: DRBD replicates a block device across two nodes over TCP using protocol C, with /dev/drbd0 backed by /dev/sdb1 on each host.

Why it is used: To get shared-storage semantics for a clustered filesystem without a SAN.

# Install
apt install drbd-utils

# Configure
# /etc/drbd.d/web.res
resource web {
    protocol C;

    on node1 {
        device /dev/drbd0;
        disk /dev/sdb1;
        address 192.168.1.10:7788;
        meta-disk internal;
    }

    on node2 {
        device /dev/drbd0;
        disk /dev/sdb1;
        address 192.168.1.11:7788;
        meta-disk internal;
    }
}

# Initialize
drbdadm create-md web
drbdadm up web

# Primary
drbdadm primary --force web

# Filesystem
mkfs.xfs /dev/drbd0
mount /dev/drbd0 /var/www

# Status
cat /proc/drbd

Interview talking point: “Only after drbdadm primary —force do you format and mount /dev/drbd0, cat /proc/drbd reports replication state, and protocol C means writes complete only once the peer acknowledges.”

Q1160: How do you configure GlusterFS?

What it is: GlusterFS is a distributed filesystem where peers are probed and volumes are built from bricks, here a replica 2 volume mounted with the glusterfs type.

Why it is used: To share a scalable filesystem across nodes without external SAN hardware.

# Install
apt install glusterfs-server

# Add peers
gluster peer probe node2

# Create volume
gluster volume create gv0 \
  replica 2 \
  node1:/brick1/gv0 \
  node2:/brick1/gv0

# Start volume
gluster volume start gv0

# Mount
mount -t glusterfs node1:/gv0 /mnt

# Volume info
gluster volume info
gluster volume status

Interview talking point: “replica 2 mirrors each brick pair so one node can fail, and the volume must be started before a glusterfs mount will attach it.”

Q1161: How do you configure object storage?

What it is: MinIO is an S3-compatible object storage server that stores objects in buckets on the filesystem.

Why it is used: Provide private, scalable object storage compatible with S3 SDKs and CLI tools.

# Install MinIO
wget https://dl.min.io/server/minio/release/linux-amd64/minio
chmod +x minio

# Start MinIO
export MINIO_ROOT_USER=minioadmin
export MINIO_ROOT_PASSWORD=minioadmin
./minio server /data --console-address ":9001"

# Using mc (MinIO Client)
mc alias set myminio http://localhost:9000 minioadmin minioadmin

# Create bucket
mc mb myminio/mybucket

# Set policy
mc anonymous set download myminio/mybucket

# Replication
mc mirror backup myminio/mybucket-archive

# Use with S3 SDK
# AWS CLI
aws configure
aws s3 ls s3://mybucket/

Interview talking point: “The mc client manages buckets and policies; mirroring supports replication to other S3-compatible endpoints.”

Q1162: How do you configure storage pools?

What it is: libvirt storage pools abstract storage backends like directories, LVM volumes, or NFS shares for VM disk management.

Why it is used: Centrally manage virtual machine storage volumes across different storage types.

# Create directory pool
virsh pool-define-as default dir --target /var/lib/libvirt/images
virsh pool-build default
virsh pool-start default
virsh pool-autostart default

# Create LVM pool
virsh pool-define-as vgpool logical \
  --source-name vg_libvirt \
  --target /dev/vg_libvirt

# Create NFS pool
virsh pool-define-as nfspool netfs \
  --source-format nfs \
  --source-host 192.168.1.10 \
  --source-path /share \
  --target /mnt/nfs

# View pools
virsh pool-list
virsh pool-info default

# Create volume
virsh vol-create default --name vm1.qcow2 --capacity 10G --allocation 10G
virsh vol-create-as default vm2.qcow2 20G

Interview talking point: “Pools define storage sources and targets; volumes created in pools can be automatically allocated and attached to VMs.”

Q1163: How do you implement distributed storage?

What it is: Ceph is deployed with ceph-deploy: monitors are created, OSDs back devices, a pool is created, and rbd creates a block image inside it.

Why it is used: To provide scalable block storage that is replicated across cluster nodes.

# Use Ceph
# Install
ceph-deploy install node1 node2 node3
ceph-deploy mon create node1
ceph-deploy osd create node1:/dev/sdb

# Create pool
ceph osd pool create mypool 100

# Use
rbd create mypool/image --size 10G

Interview talking point: “ceph osd pool create names the pool with a placement group count, and rbd create carves a sized image from it, separating pool from volume.”

Q1164: How do you manage storage tiers?

What it is: Storage tiers put fast media in front of capacity media. LVM cache adds a cache volume to a logical volume, btrfs spans devices, mdadm can add a disk to an array, and ZFS creates a pool with a cache device.

Why it is used: To accelerate slow bulk storage with SSD cache without paying for all-flash capacity.

# Using lvmcache
lvcreate --type cache --cachevol cachevol --pool vg_data/lv_data vg_data/cachepool

# Using btrfs
mkfs.btrfs -d single -m single /dev/sda1 /dev/sdb1

# Using mdadm with SSD cache
mdadm --create /dev/md0 --level=5 --raid-devices=3 /dev/sd[a-c]1
mdadm --manage /dev/md0 --add /dev/sdd1

# ZFS
zpool create data mirror /dev/sda1 /dev/sdb1 cache /dev/sdc1

#查看状态
zpool status
zpool list
zfs get all

Interview talking point: “zpool status and zpool list show tier health and layout after creation, and the ZFS pool uses a mirror of data disks plus a dedicated cache device rather than RAID across tiers.”

Q1165: How do you use Ceph?

What it is: Ceph is a storage cluster built from monitor, manager, OSD, and metadata daemons. ceph-deploy creates the cluster, builds an OSD on a raw disk, a pool is created for data, and the ceph filesystem type mounts it.

Why it is used: To provide block, file, and object storage that scales out across commodity machines.

# Install
apt install ceph-mon ceph-osd ceph-mgr ceph-mds

# Deploy
ceph-deploy new node1
ceph-deploy install node1
ceph-deploy mon create-initial

# Create OSD
ceph-deploy osd create --data /dev/sdb node1

# Create pool
ceph osd pool create mypool 100

# Mount
mount -t ceph node1:/ /mnt/ceph

Interview talking point: “ceph-deploy provisions the daemons and an OSD on a whole unused disk first; only after that does creating a pool give you a namespace to store anything in.”

Q1166: How do you use GlusterFS?

What it is: GlusterFS is a distributed filesystem assembled from bricks. Servers are peered, a replica 2 volume is built from bricks on two nodes, the volume is started, and clients mount it with the glusterfs filesystem type.

Why it is used: To share one replicated, scalable filesystem across a cluster of Linux servers.

# Install
apt install glusterfs-server

# Add peers
gluster peer probe node2

# Create volume
gluster volume create gv0 replica 2 \
  node1:/brick1/gv0 \
  node2:/brick1/gv0

# Start
gluster volume start gv0

# Mount
mount -t glusterfs node1:/gv0 /mnt

Interview talking point: “Replica 2 means every brick’s data exists on both nodes for redundancy, and mounting by volume name lets the client reach the volume through whichever peer responds.”

Q1167: How do you use minIO?

What it is: MinIO is an object storage server started against a data directory, and the mc client sets an alias then creates buckets through the S3 API.

Why it is used: To host S3 compatible object storage on your own hardware or in a lab.

# Install
wget https://dl.min.io/server/minio/release/linux-amd64/minio
chmod +x minio

# Start
./minio server /data

# Use S3 API
mc alias set myminio http://localhost:9000 accesskey secretkey
mc mb myminio/mybucket

Interview talking point: “mc alias set stores the endpoint and keys, so later commands like mb target the named alias instead of repeating credentials every time.”

Q1168: How do you use persistent memory?

What it is: Persistent memory is managed with ndctl, which creates namespaces on /dev/pmem devices that can be formatted and mounted with the dax option.

Why it is used: To get byte addressable storage that survives power loss at near memory speed.

# Check
ndctl list

# Create namespace
ndctl create-namespace

# Mount
# As DAX (Direct Access)
mkfs.ext4 -F /dev/pmem0
mount -o dax /dev/pmem0 /mnt/pmem

# Use in code
# Use libpmem

Interview talking point: “Mounting with dax maps files directly into the page cache, so reads skip the block layer and the data copy entirely.”

Q1169: How do you configure NVMe optimization?

What it is: NVMe tuning lists devices and smart logs, sets the I/O scheduler to none, and raises nr_requests to deepen the queue on the block device.

Why it is used: To keep fast NVMe devices from being slowed by scheduler overhead.

# Check
nvme list
nvme smart-log /dev/nvme0

# Optimize
# Use noop or deadline scheduler
echo none > /sys/block/nvme0n1/queue/scheduler

# Increase queue depth
echo 1024 > /sys/block/nvme0n1/queue/nr_requests

Interview talking point: “Setting the scheduler to none suits NVMe because its parallel queues make elevator reordering pointless, and nr_requests 1024 deepens the queue for throughput.”

Virtualization

Q1170: How do you configure GPU virtualization?

What it is: GPU virtualization shares a physical GPU across VMs using vGPU (with NVIDIA GRID) or dedicates it via passthrough.

Why it is used: Provide GPU acceleration for virtualized workloads like ML, VDI, or rendering.

# NVIDIA vGPU
# Install nvidia-vgpud and nvidia-vgpu-mgr
apt install nvidia-vgpu-mgr

# Configure vGPU
# /etc/nvidia-vgpu-mgr/vgpuConfig.txt
vgpuConfig=1
guestVmTimeout=60
NVMgrid

# Create vGPU
virsh nodedev-create vgpu0

# Or using NVIDIA GRID
# Install NVIDIA driver in VM
./NVIDIA-Linux-x86_64-grid.run

# Check
nvidia-smi
# Should show vGPU

# For AMD GPU passthrough
# See PCI passthrough

Interview talking point: “vGPU allows multi-tenant sharing with resource limits, while passthrough gives exclusive access to a single VM.”

Q1171: How do you configure KVM live migration?

What it is: Live migration moves a running VM to another host: libvirtd.conf enables listening on TCP with SASL auth, and virsh migrate —live transfers the domain, optionally with —compressed.

Why it is used: Evacuates hosts for maintenance or load balancing without downtime.

# Shared storage
# NFS or similar

# Allow migration
# /etc/libvirt/libvirtd.conf
listen_tls = 0
listen_tcp = 1
auth_tcp = "sasl"

# Migrate
virsh migrate --live --persistent vm qemu+tcp://dest/system

# With compression
virsh migrate --live --compressed vm qemu+tcp://dest/system

Interview talking point: “Shared storage such as NFS avoids copying disks, so migration streams only memory, and —compressed trades guest CPU for time by shrinking the pages sent across the link while the VM keeps running.”

Q1172: How do you configure KVM live migration?

What it is: KVM live migration moves a running virtual machine between hosts with minimal downtime using shared storage or pre-copied disks.

Why it is used: Enable hardware maintenance, load balancing, and high availability across hypervisor hosts.

# Enable migration on source
virsh migrate --live --persistent --undefinesource \
  vmname qemu+ssh://dest-host/system

# Using migrate command
virsh migrate --live vmname dest-host

# With compression
virsh migrate --live --compressed --desturi qemu+tcp://dest-host/system vmname

# With TLS
virsh migrate --live --tls dest-host vmname

# Configure for migration
# /etc/libvirt/libvirtd.conf
listen_tls = 0
listen_tcp = 1
auth_tcp = "none"

# Enable in /etc/default/libvirt-bin
# LIBVIRTD_ARGS="--listen"

# Verify migration
virsh dominfo vmname
virsh migrate-getspeed vmname

Interview talking point: “Migration traffic can be compressed or secured with TLS; persistent migration can undefine the source VM after transfer.”

Q1173: How do you configure KVM memory ballooning?

What it is: Memory ballooning lets the host reclaim guest memory: a virtio memballoon device is defined in XML, then virsh setmem changes current memory and setmaxmem changes the ceiling.

Why it is used: Overcommits memory by inflating one guest’s balloon to free RAM for another.

# Enable in VM
<memballoon model='virtio'>
</memballoon>

# Dynamic adjustment
virsh setmem vm 2G --config
virsh setmaxmem vm 4G --config

Interview talking point: “setmem adjusts the amount the guest currently sees while setmaxmem raises the limit it may grow to, so both must move together when you plan to hand memory back to a guest.”

Q1174: How do you configure KVM networking?

What it is: KVM guest networking defined through libvirt network definitions. It bridges a host interface into br0, defines a NAT network on virbr0 with STP enabled, and defines a host-only network bridging to br0.

Why it is used: To give guests internet access through NAT, or direct visibility on the physical LAN through a bridge.

# Bridge
virsh iface-bridge eth0 br0

# NAT network
virsh net-define /dev/stdin <<EOF
<network>
  <name>default</name>
  <forward mode='nat'/>
  <bridge name='virbr0' stp='on' delay='0'/>
</network>
EOF

# Host-only
virsh net-define /dev/stdin <<EOF
<network>
  <name>hostonly</name>
  <forward mode='bridge'>
    <bridge name='br0'/>
  </forward>
</network>
EOF

Interview talking point: “The forward mode attribute decides guest reachability: mode nat hides guests behind the host address, while mode bridge lets guest traffic appear directly on br0 as if it came from a real device.”

Q1175: How do you configure KVM?

What it is: KVM lets a Linux host run virtual machines using hardware virtualization. virt-install creates a VM with disk, vCPUs, network bridge, and an install source, and virsh then manages its lifecycle.

Why it is used: To provision and control virtual machines directly on a Linux host without a hypervisor GUI.

# Install
apt install qemu-kvm libvirt-daemon-system libvirt-clients bridge-utils

# Check support
kvm-ok

# Create VM
virt-install \
  --name webserver \
  --ram 2048 \
  --disk path=/var/lib/libvirt/images/webserver.qcow2,size=20 \
  --vcpus 2 \
  --os-type linux \
  --os-variant ubuntu22.04 \
  --network bridge=virbr0 \
  --graphics vnc \
  --console pty,target_type=serial \
  --location 'http://archive.ubuntu.com/ubuntu/dists/jammy/main/installer-amd64/' \
  --extra-args 'console=ttyS0'

# Manage VMs
virsh list --all
virsh start webserver
virsh shutdown webserver
virsh undefine webserver

Interview talking point: “kvm-ok and the vmx flag confirm hardware support before anything else, and virsh commands like start, shutdown, and undefine form the basic VM lifecycle you should know cold.”

Q1176: How do you configure KVM?

What it is: KVM virtualization installs qemu-kvm and libvirt, verifies with kvm-ok, creates a VM with virt-install, and manages it through virsh commands.

Why it is used: To run multiple isolated guest operating systems on one physical host.

# Install KVM
apt install qemu-kvm libvirt-daemon-system libvirt-clients bridge-utils

# Verify
kvm-ok

# Create VM
virt-install \
  --name webserver \
  --ram 2048 \
  --disk path=/var/lib/libvirt/images/webserver.qcow2,size=20 \
  --vcpus 2 \
  --os-type linux \
  --os-variant ubuntu22.04 \
  --network bridge=virbr0 \
  --graphics vnc \
  --location 'http://archive.ubuntu.com/ubuntu/dists/jammy/main/installer-amd64/' \
  --extra-args 'console=ttyS0'

# Manage VMs
virsh list --all
virsh start webserver
virsh shutdown webserver
virsh reboot webserver
virsh undefine webserver

# Manage storage pools
virsh pool-list
virsh pool-info default

Interview talking point: “virt-install creates and boots the guest from an install source in one shot, while virsh handles the lifecycle afterward, for example list —all, start, shutdown and undefine.”

Q1177: How do you configure libguestfs?

What it is: libguestfs-tools edits disk images without booting them. guestmount mounts an image partition at /mnt, guestfish lists files inside it, virt-edit edits a file, and virt-copy-in and virt-copy-out move files in and out.

Why it is used: To recover or repair files on a guest disk that will not boot.

# Install
apt install libguestfs-tools

# Mount
guestmount -a disk.img -m /dev/sda1 /mnt

# List files
guestfish -a disk.img -m /dev/sda1 ls /

# Edit
virt-edit -a disk.img /etc/passwd

# Copy in/out
virt-copy-in file.txt /root/
virt-copy-out /root/file.txt ./

Interview talking point: “guestmount exposes the image as an ordinary mount for normal shell work, while guestfish runs its own listing session; both reach the same files without ever starting a virtual machine.”

Q1178: How do you configure libvirt?

What it is: libvirt manages storage pools and networks for VMs; pool-define-as registers a directory as a pool, pool-start activates it, pool-autostart persists it, and net-define plus net-start bring up networking.

Why it is used: Giving virtual machines storage and network connectivity through one management layer.

# Pool management
virsh pool-define-as default dir - - - - /var/lib/libvirt/images
virsh pool-start default
virsh pool-autostart default

# Network
virsh net-define /etc/libvirt/qemu/networks/default.xml
virsh net-start default

Interview talking point: “A defined pool is not usable until pool-start, and pool-autostart keeps it available after a host reboot, the same way net-start activates the default network that guests attach to.”

Q1179: How do you configure libvirt?

What it is: libvirt management via virsh defines virtual networks and storage pools, creates snapshots, performs live migration and clones machines with virt-clone.

Why it is used: To automate guest lifecycle, snapshots and moves across hypervisor hosts.

# Connect to libvirt
virsh --connect qemu:///system

# Create network
virsh net-define /tmp/network.xml
virsh net-start mynetwork
virsh net-autostart mynetwork

# Create storage pool
virsh pool-define-as default dir --target /var/lib/libvirt/images
virsh pool-build default
virsh pool-start default

# Create snapshot
virsh snapshot-create-as webserver --name "before-update"
virsh snapshot-list webserver
virsh snapshot-revert webserver before-update

# Migrate VM
virsh migrate --live webserver qemu+ssh://dest-host/system

# Clone VM
virt-clone --original webserver --name webserver2 --auto-clone

Interview talking point: “virsh snapshot-revert rolls a guest back to a named point like before-update, and virsh migrate —live relocates a running VM to another host without shutting it down.”

Q1180: How do you configure nested virtualization?

What it is: Nested virtualization allows running hypervisors inside virtual machines by enabling nesting in the host CPU kernel module.

Why it is used: Test virtualization setups, run CI pipelines, or create lab environments without physical hardware.

# Enable nesting (Intel)
# Add to kernel parameters
kvm-intel.nested=1

# Check
cat /sys/module/kvm_intel/parameters/nested
# Y = enabled

# Create nested VM
virt-install \
  --name nested-vm \
  --ram 2048 \
  --vcpus 2 \
  --disk path=/var/lib/libvirt/images/nested.qcow2 \
  --os-variant ubuntu22.04 \
  --graphics vnc \
  --cpu host-passthrough

# Verify
# Inside nested VM
lscpu | grep Virtualization
# Should show VT-x or AMD-V

# Nested Docker
# On nested VM
apt install docker.io
# Should work
docker run hello-world

Interview talking point: “Host CPU passthrough exposes virtualization extensions to the nested VM, enabling it to run its own guests.”

Q1181: How do you configure PCI passthrough?

What it is: PCI passthrough assigns a physical PCI device directly to a VM, bypassing the host kernel driver.

Why it is used: Achieve near-native performance for specialized hardware like GPUs, NVMe, or network cards.

# Enable IOMMU
GRUB_CMDLINE_LINUX="intel_iommu=on"
# or
GRUB_CMDLINE_LINUX="amd_iommu=on"

# Update GRUB
update-grub
reboot

# Verify
dmesg | grep -e DMAR -e IOMMU

# Bind device to vfio
lspci -nnk -d 10de:  # NVIDIA
# 10de:1b80:01:00.0

# vfio-pci.ids
echo 10de 1b80 > /etc/modprobe.d/vfio.conf

# Unbind from driver
virsh nodedev-detach pci_0000_01_00_0

# Add to VM
# virsh edit vmname
<hostdev mode='subsystem' type='pci' managed='yes'>
  <source>
    <address domain='0x0000' bus='0x01' slot='0x00' function='0x0'/>
  </source>
</hostdev>

Interview talking point: “IOMMU must isolate device groups, and the vfio driver binds the device to prevent host access during VM use.”

Q1182: How do you configure QEMU?

What it is: QEMU emulates a machine and, with KVM enabled, runs guests at near-native speed. qemu-img manages virtual disks: create an image backed by a base file, inspect it, resize it, or convert between formats.

Why it is used: To boot a guest from an ISO, forward a host port to guest ssh, and manage image files.

# Run QEMU
qemu-system-x86_64 \
  -m 2048 \
  -hda disk.img \
  -cdrom ubuntu.iso \
  -boot d \
  -enable-kvm

# With network
qemu-system-x86_64 \
  -m 2048 \
  -hda disk.img \
  -netdev user,id=net0,hostfwd=tcp::2222-:22 \
  -device e1000,netdev=net0

# Snapshots
qemu-img create -f qcow2 -b base.img snapshot.img
qemu-img info snapshot.img

# Resize disk
qemu-img resize disk.img +10G

# Convert image
qemu-img convert -O vmdk img.qcow2 img.vmdk

Interview talking point: “The hostfwd rule forwards host port 2222 to guest port 22, and qcow2 images can be created backed by a base image so snapshots stay small instead of becoming full copies.”

Q1183: How do you manage KVM snapshots?

What it is: A KVM snapshot captures a virtual machine’s disk state at a moment in time. Commands create a named snapshot with a description, list snapshots, revert to one, and delete it.

Why it is used: To checkpoint a VM before a risky change and roll back quickly if it breaks.

# Create snapshot
virsh snapshot-create-as vm1 --name snap1 --description "Before update"

# List snapshots
virsh snapshot-list vm1

# Revert
virsh snapshot-revert vm1 snap1

# Delete
virsh snapshot-delete vm1 snap1

Interview talking point: “snapshot-revert rolls the guest back to the saved state, so anything done after snap1 is lost. If that work matters, create a new snapshot before reverting so it can be recovered later.”

Q1184: How do you manage libvirt?

What it is: libvirt is the management layer behind KVM, driven with virsh. It defines virtual networks from XML and starts them, and it manages snapshots plus storage pools for guest disks.

Why it is used: To manage VM networks, snapshots, and storage pools consistently across guests from one tool.

# Connect
virsh --connect qemu:///system

# Create network
# /tmp/network.xml
<network>
  <name>internal</name>
  <forward mode='nat'/>
  <ip address='192.168.100.1' netmask='255.255.255.0'>
    <dhcp>
      <range start='192.168.100.128' end='192.168.100.254'/>
    </dhcp>
  </ip>
</network>

virsh net-define /tmp/network.xml
virsh net-start internal

# Snapshot
virsh snapshot-create-as webserver --name "before-update"
virsh snapshot-list webserver
virsh snapshot-revert webserver before-update

# Pool management
virsh pool-list
virsh pool-info default

Interview talking point: “A snapshot taken before an update can be reverted with one command, far faster than rebuilding a guest, and net-define plus net-start is the pair needed to bring a new virtual network up.”

Q1185: How do you use cloud-init with KVM?

What it is: cloud-init bootstraps a guest from a cloud image: a prebuilt qcow2 is downloaded, virt-install attaches it with —cloud-init, or a cidata ISO made with genisoimage supplies user-data and meta-data.

Why it is used: Creates ready-to-use VMs with users, keys, and hostname without manual installs.

# Download cloud image
wget https://cloud-images.ubuntu.com/jammy/current/jammy-server-cloudimg-amd64.img

# Create VM with cloud-init
virt-install --name vm1 \
  --memory 2048 --vcpu 2 \
  --disk jammy-server-cloudimg-amd64.img,format=qcow2 \
  --cloud-init root-disk-generate=empty

# Or with manual
# genisoimage -o seed.iso -volid cidata -joliet -rock user-data meta-data

Interview talking point: “The seed ISO carries only user-data and meta-data while the image stays generic, so one golden image can be cloned many times with each instance personalized purely by its own seed.”

Q1186: How do you use KVM nested?

What it is: Nested virtualization lets a VM expose hardware virtualization to its own guests; modprobe kvm_intel or kvm_amd takes nested=1, and the sysfs module parameter reports the current setting.

Why it is used: Runs KVM inside a VM, useful for labs, CI pipelines, and nested test clusters.

# Enable nested
modprobe kvm_intel nested=1
# or
modprobe kvm_amd nested=1

# Check
cat /sys/module/kvm_intel/parameters/nested

# In VM
# CPU model
<cpu mode='host-passthrough'>

Interview talking point: “The guest must see a CPU model such as host-passthrough or it will not receive the virtualization extensions, and reading the sysfs nested parameter is the quick confirmation that the feature is actually on.”

Q1187: How do you use KVM PCI passthrough?

What it is: PCI passthrough gives a VM direct control of a device: the IOMMU is enabled with intel_iommu=on in GRUB, followed by update-grub, a reboot, and a hostdev XML attached with virsh attach-device.

Why it is used: Lets a guest drive GPUs, HBAs, or NICs at full hardware speed.

# Enable IOMMU
GRUB_CMDLINE_LINUX_DEFAULT="intel_iommu=on"

# Update
update-grub

# Reboot

# Attach
virsh attach-device vm /etc/libvirt/qemu/pci.xml --persistent

# pci.xml
<hostdev mode='subsystem' type='pci' managed='yes'>
  <source>
    <address domain='0x0000' bus='0x01' slot='0x00' function='0x0'/>
  </source>
</hostdev>

Interview talking point: “The IOMMU is what isolates device DMA from the host, so intel_iommu=on is not optional, and managed=‘yes’ in the XML lets libvirt unbind the device from the host driver automatically at attach time.”

Q1188: How do you use KVM snapshots?

What it is: Snapshots capture a VM’s state: snapshot-create-as makes one, snapshot-list shows them, snapshot-revert rolls the guest back, and snapshot-delete removes it.

Why it is used: Preserves a known-good state before risky changes or upgrades.

# Create snapshot
virsh snapshot-create-as vm --name snap1

# List
virsh snapshot-list vm

# Revert
virsh snapshot-revert vm snap1

# Delete
virsh snapshot-delete vm snap1

Interview talking point: “The practical interview question is what a snapshot covers: a revert rolls the guest back to the captured state, so anything written afterward is discarded rather than kept as a separate recoverable layer.”

Q1189: How do you use KVM vhost-net?

What it is: vhost-net moves virtio packet handling into the kernel through the vhost_net module, and the VM’s interface XML selects model virtio with driver name vhost.

Why it is used: Speeds up VM networking by shortening the path between guest and host.

# Load module
modprobe vhost_net

# Enable in VM
<interface type='network'>
  <model type='virtio'/>
  <driver name='vhost'/>
</interface>

Interview talking point: “With vhost enabled the host kernel handles the virtio rings directly instead of user-space QEMU relaying them, cutting context switches and raising throughput, which is why vhost_net is loaded on performance-minded hosts.”

Q1190: How do you use KVM/QEMU?

What it is: KVM with QEMU runs virtual machines on the host CPU; virt-install creates a VM with RAM, disk, and OS details, while virsh lists, starts, shuts down, or destroys it.

Why it is used: Consolidating multiple workloads on one physical machine.

# Install
apt install qemu-kvm libvirt-daemon

# Create VM
virt-install --name vm1 --ram 2048 --disk size=20 --os-variant ubuntu20.04 --location http://archive.ubuntu.com/ubuntu/dists/focal/main/installer-amd64/

# Manage
virsh list --all
virsh start vm1
virsh shutdown vm1
virsh destroy vm1

Interview talking point: “virsh shutdown sends a graceful shutdown request the guest must honor, while virsh destroy is an abrupt power cut, so reach for destroy only when the guest will not stop.”

Q1191: How do you use KVM?

What it is: KVM is a Linux kernel hypervisor that lets the host run virtual machines. The commands install qemu-kvm and libvirt, list guests, create a VM, start, stop, and attach to its console.

Why it is used: To run several isolated guest operating systems on one physical Linux server.

# Install
apt install qemu-kvm libvirt-daemon virt-manager

# List
virsh list --all

# Create
virt-install --name vm1 --ram 2048 --disk size=20 \
  --os-variant ubuntu20.04 --location http://archive.ubuntu.com/ubuntu/dists/focal/main/installer-amd64/

# Start/Stop
virsh start vm1
virsh shutdown vm1
virsh destroy vm1

# Console
virsh console vm1

Interview talking point: “virsh start boots a guest, virsh shutdown requests a graceful halt, and virsh destroy cuts power immediately. That makes destroy the last resort when a guest stops responding to a clean shutdown.”

Q1192: How do you use libguestfs-tools?

What it is: libguestfs-tools inspects guest disk images without booting them: guestmount mounts a partition, virt-ls lists files, virt-edit edits, virt-copy-in and virt-copy-out move files, and virt-cat reads.

Why it is used: Fixes a VM disk or harvests data offline, even when the guest will not start.

# Mount
guestmount -a disk.img -m /dev/sda1 /mnt

# List files
virt-ls -a disk.img /

# Edit
virt-edit -a disk.img /etc/passwd

# Copy in/out
virt-copy-in -a disk.img file.txt /root/
virt-copy-out -a disk.img /root/file.txt ./

# Cat
virt-cat -a disk.img /etc/passwd

Interview talking point: “The -a flag names the image for every one of these commands, and guestmount gives a live filesystem mount, the quickest route to browsing a broken guest’s disk instead of editing it one file at a time.”

Q1193: How do you use libvirt?

What it is: libvirt is a management API and toolset for virtualization, driven here through virsh. It defines storage pools, starts and autostarts them, defines virtual networks, and creates and lists disk volumes.

Why it is used: To manage storage, networking, and volumes for virtual machines through one consistent command interface.

# Pools
virsh pool-define-as default dir - - - - /var/lib/libvirt/images
virsh pool-start default
virsh pool-autostart default
virsh pool-list

# Networks
virsh net-define /etc/libvirt/qemu/networks/default.xml
virsh net-start default

# Volumes
virsh vol-create --pool default --name vm1.qcow2 --capacity 20G
virsh vol-list default

Interview talking point: “Pool, network, and volume are libvirt’s three building blocks: pools hold storage, networks hold virtual wiring, and volumes are the individual disk images created inside a pool.”

Q1194: How do you use oVirt?

What it is: oVirt is an enterprise virtualization management platform controlled through ovirt-cli. The commands authenticate as the internal admin, create a VM from a blank template in the default cluster, and start it.

Why it is used: To automate VM creation and lifecycle tasks against a central oVirt engine.

# Install
# Install engine and hosts

# Create VM
ovirt-cli --user admin@internal --password password \
  vm create --name "MyVM" --template "Blank" --cluster "Default"

# Start
ovirt-cli --user admin@internal --password password \
  vm start --name "MyVM"

Interview talking point: “oVirt commands reach a central engine rather than a local hypervisor, so the user, template, and cluster must already exist on the engine before a VM can be created or started.”

Q1195: How do you use Proxmox?

What it is: Proxmox is a virtualization platform driven by the qm command. It creates VM 100 with memory and a virtio NIC on vmbr0, starts it, lists VMs and storage volumes, and runs backups with vzdump.

Why it is used: To manage virtual machines, storage, and backups on a Proxmox host from the shell.

# Install
# Download ISO, install

# Create VM
qm create 100 --memory 2048 --name "MyVM" --net0 virtio,bridge=vmbr0
qm start 100

# List
qm list
pvesm list

# Backup
vzdump --compress --storage local 100

Interview talking point: “Proxmox splits the work by command: qm runs virtual machines, pvesm reports storage contents, and vzdump handles backups, so checking free capacity before creating a VM starts with pvesm list.”

Q1196: How do you use QEMU?

What it is: QEMU emulates hardware and runs virtual machines, while qemu-img manages disk images. The commands create a qcow2 image, convert one, resize it, print its info, and boot a machine with 2048 MB and an ISO.

Why it is used: To build, convert, and run virtual disk images and machines entirely from the command line.

# Create image
qemu-img create -f qcow2 vm.qcow2 20G

# Convert
qemu-img convert -O qcow2 source.img dest.qcow2

# Resize
qemu-img resize vm.qcow2 +10G

# Info
qemu-img info vm.qcow2

# Run
qemu-system-x86_64 -m 2048 -hda vm.qcow2 -cdrom iso.img

Interview talking point: “qemu-img operates on image files without booting a guest, so conversion and resizing happen offline, while qemu-system-x86_64 is the piece that actually executes the virtual machine.”

Q1197: How do you use Vagrant with KVM?

What it is: Vagrant drives local VMs from a Vagrantfile: the vagrant-libvirt plugin targets libvirt, the file sets memory and cpus, and vagrant up —provider=libvirt starts the box.

Why it is used: Spins up reproducible development environments on KVM rather than another hypervisor.

# Install
vagrant plugin install vagrant-libvirt

# Vagrantfile
Vagrant.configure("2") do |config|
  config.vm.box = "generic/ubuntu2204"
  config.vm.provider "libvirt" do |lv|
    lv.memory = 2048
    lv.cpus = 2
  end
end

# Commands
vagrant up --provider=libvirt
vagrant destroy

Interview talking point: “The provider block decides which hypervisor Vagrant talks to, so the same Vagrantfile still runs elsewhere, and vagrant destroy tears the machine down cleanly instead of leaving a stray domain in virsh list.”

Q1198: How do you use virsh console?

What it is: virsh console attaches to a guest’s serial console; the kernel must be told to use it with console=tty0 console=ttyS0,115200, and Ctrl+] detaches from the session.

Why it is used: Recovers a VM whose network or graphical console is unreachable.

# Enable console in VM
GRUB_CMDLINE_LINUX_DEFAULT="console=tty0 console=ttyS0,115200"

# Connect
virsh console vm

# Exit
# Ctrl+]

Interview talking point: “The serial console works even when the guest has no networking, which is why it is the emergency access path, and Ctrl+] to escape matters because otherwise the session just looks hung inside the guest.”

Q1199: How do you use virt-manager?

What it is: virt-manager is a graphical front end for libvirt: it manages local QEMU/KVM guests and can add connections such as QEMU/KVM over SSH to remote hosts.

Why it is used: Gives a visual console and hardware editor when command-line virsh is not convenient.

# Install
apt install virt-manager

# Run
virt-manager

# Connect to remote
# File -> Add Connection
# QEMU/KVM over SSH

Interview talking point: “Underneath it is still libvirt, so anything done in the GUI appears in virsh list and virsh dumpxml, and remote mode works because libvirt simply tunnels over the SSH connection.”

Q1200: How do you use VirtualBox?

What it is: VirtualBox is a hypervisor controlled here through VBoxManage. It creates and registers a VM, sets memory and video RAM, adds a SATA controller, attaches a disk image, and starts the machine.

Why it is used: To script VirtualBox VM creation on a workstation instead of clicking through the graphical console.

# Install
apt install virtualbox

# Create VM
VBoxManage createvm --name "MyVM" --ostype Ubuntu_64 --register
VBoxManage modifyvm "MyVM" --memory 2048 --vram 128
VBoxManage storagectl "MyVM" --name "SATA" --add sata
VBoxManage storageattach "MyVM" --storagectl "SATA" --port 0 --device 0 --type hdd --medium disk.vmdk

# Start
VBoxManage startvm "MyVM"

Interview talking point: “VBoxManage mirrors every GUI action, and the order matters: storagectl must create the SATA controller before storageattach can attach a disk to it on a given port.”

Q1201: How do you use Xen?

What it is: Xen is a bare-metal hypervisor managed with the xl toolstack. Commands install it, create a guest from a config file, list running guests, start, shut down, destroy, and attach to the console.

Why it is used: To run server workloads on a hypervisor that owns the hardware directly rather than a host OS.

# Install
apt install xen-hypervisor

# Create VM
xl create /etc/xen/vm1.cfg

# List
xl list

# Start/Stop
xl create vm1.cfg
xl shutdown vm1
xl destroy vm1

# Console
xl console vm1

Interview talking point: “xl create both defines and starts the guest from its config file, while xl shutdown asks the guest to halt gracefully and xl destroy tears it down without waiting.”

Q1202: How do you optimize Linux for virtualization?

What it is: Linux virtualization tuning with CPU pinning, memory overcommit, para-virtual networking, VirtIO storage, and per-VM monitoring.

Why it is used: Reduces virtualization overhead and keeps VMs predictable under load.

# Linux virtualization optimization
# 1. CPU
# Pinning

# 2. Memory
# Overcommit

# 3. Network
# Para-virtual

# 4. Storage
# VirtIO

# 5. Monitoring
# Per-VM

Interview talking point: “VirtIO and para-virtual drivers cut emulation overhead by letting the guest cooperate with the host, which is why they appear for storage and network. CPU pinning trades flexibility for consistent performance on busy hosts.”

Q1203: How do you optimize Linux for virtualization?

What it is: Linux virtualization optimization involves tuning CPU pinning, memory overcommit, para-virtualized networking, VirtIO storage, and per-VM monitoring.

Why it is used: These tunings help maximize resource efficiency and performance for virtual machines.

# CPU
# Pinning

# Memory
# Overcommit

# Network
# Para-virtual

# Storage
# VirtIO

# Monitoring
# Per-VM

Interview talking point: “CPU pinning and per-VM monitoring give fine control, while VirtIO and para-virtual networking reduce overhead for I/O and network traffic.”

HA Tooling (Linux)

Q1204: How do you configure Corosync/Pacemaker?

What it is: Corosync provides membership and messaging between cluster nodes while Pacemaker manages resources on top. pcs authenticates hosts, sets up the cluster, and creates resources like a virtual IP and a web service with constraints binding them together.

Why it is used: To build a two-node active-passive cluster where services and IPs move together on failure.

# Install
apt install pacemaker corosync pcs

# Configure corosync
# /etc/corosync/corosync.conf
totem {
    version: 2
    cluster_name: mycluster
    transport: udpu
    interface {
        ringnumber: 0
        bindnetaddr: 192.168.1.0
        mcastport: 5405
    }
}

nodelist {
    node {
        ring0_addr: node1.example.com
        nodeid: 1
    }
    node {
        ring0_addr: node2.example.com
        nodeid: 2
    }
}

quorum {
    provider: corosync_votequorum
    expected_votes: 2
}

# Configure cluster
pcs host auth node1 node2
pcs cluster setup mycluster node1 node2
pcs cluster start --all
pcs cluster enable --all

# Add resources
pcs resource create VirtualIP ocf:heartbeat:IPaddr2 \
  ip=192.168.1.100 cidr_netmask=24 op monitor interval=30s
pcs resource create WebService systemd:apache2 \
  op monitor interval=30s
pcs constraint colocation add WebService VirtualIP

Interview talking point: “The colocation constraint is what keeps the web service on the same node as the virtual IP; without it the two could land on different machines after failover.”

Q1205: How do you configure Corosync?

What it is: Corosync is the cluster layer providing node membership and quorum through a totem ring, configured in corosync.conf and brought up with pcs commands.

Why it is used: It forms the communication and quorum foundation that a Pacemaker cluster runs on.

# Install
apt install pacemaker corosync pcs

# Configure corosync
# /etc/corosync/corosync.conf
totem {
    version: 2
    cluster_name: mycluster
    transport: udpu
}

nodelist {
    node {
        ring0_addr: node1.example.com
        nodeid: 1
    }
    node {
        ring0_addr: node2.example.com
        nodeid: 2
    }
}

quorum {
    provider: corosync_votequorum
    expected_votes: 2
}

# Setup cluster
pcs host auth node1 node2
pcs cluster setup mycluster node1 node2
pcs cluster start --all
pcs cluster enable --all

Interview talking point: “The quorum block sets expected_votes to 2, so with two nodes a single survivor lacks quorum, and transport udpu means unicast UDP instead of multicast on the network.”

Q1206: How do you configure fencing?

What it is: Fencing, also called STONITH, forcibly powers off a failed cluster node using fence agents such as IPMI before another node takes over its resources.

Why it is used: To guarantee a suspected-dead node cannot keep running and corrupt shared data.

# Install fence-agents
yum install fence-agents-all

# Configure STONITH
# /etc/cluster/cluster.conf
<cluster name="mycluster" config_version="1">
  <fence_daemon post_fail_delay="0" post_join_delay="3"/>
  <fence_method name="method" device="name">
    <fence_level level="1" id="level1">
      <device name="device" action="on" port="node1"/>
    </fence_level>
  </fence_method>
  <nodes>
    <node name="node1" nodeid="1" fences="method"/>
    <node name="node2" nodeid="2" fences="method"/>
  </nodes>
</cluster>

# IPMI fence
stonith -a ipmi -t "ipmilan=user:pass@node1" -v

# Test fence
pcs stonith fence node1
pcs stonith history node1

Interview talking point: “pcs stonith history shows past fence operations, and fencing is deliberately brutal because cutting power is the only reliable way to prove a node is truly offline.”

Q1207: How do you configure keepalived?

What it is: keepalived implements VRRP where instances declare state, priority, and a virtual IP, with the higher priority node holding the VIP and a lower priority backup.

Why it is used: To give a service a floating IP that fails over automatically between two hosts.

# Install
apt install keepalived

vrrp_instance VI_1 {
    state MASTER
    interface eth0
    virtual_router_id 51
    priority 100
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass mypassword
    }
    virtual_ipaddress {
        192.168.1.100 dev eth0 label eth0:vip
    }
}

# BACKUP config
# priority 90

# Enable service
systemctl enable keepalived
systemctl start keepalived

Interview talking point: “The master sits at priority 100 and the backup at 90, so adjusting priority is what hands the virtual IP over when the master stops advertising.”

Q1208: How do you configure Keepalived?

What it is: Keepalived implements VRRP so a virtual IP address floats between a master and backup router. A tracked script can reduce priority when a service check fails, which triggers failover.

Why it is used: To remove a single point of failure by failing a shared IP address over automatically.

# Install
apt install keepalived

vrrp_instance VI_1 {
    state MASTER
    interface eth0
    virtual_router_id 51
    priority 100
    advert_int 1

    authentication {
        auth_type PASS
        auth_pass mysecret
    }

    virtual_ipaddress {
        192.168.1.100
    }

    track_script {
        check_apache
    }
}

vrrp_script check_apache {
    script "pkill -0 apache2"
    interval 2
    weight 2
}

# On backup server
# priority 90
# state BACKUP

Interview talking point: “The virtual router id must match on both nodes while priority decides who wins, and a track script that fails on the master demotes it so the backup assumes the virtual IP.”

Q1209: How do you configure Keepalived?

What it is: Keepalived implements VRRP failover, where a MASTER node holds a virtual IP, a BACKUP with lower priority stands by, and a shared password authenticates the exchange.

Why it is used: So a service keeps its IP reachable even when the active node or its interface dies.

# Install
apt install keepalived

# Configure
# /etc/keepalived/keepalived.conf
vrrp_instance VI_1 {
    state MASTER
    interface eth0
    virtual_router_id 51
    priority 100

    virtual_ipaddress {
        192.168.1.100/24
    }

    track_interface {
        eth0 weight -20
    }

    authentication {
        auth_type PASS
        auth_pass secret123
    }
}

# Backup configuration
# state BACKUP
# priority 90

Interview talking point: “priority 100 versus 90 elects the master, and track_interface with weight -20 subtracts from that priority, so losing eth0 can demote the node and hand the virtual IP to the backup.”

Q1210: How do you configure Pacemaker/Corosync?

What it is: Pacemaker with Corosync forms a cluster configured through pcs: authenticate hosts, set up the cluster, add resources like an IP with a monitor operation, and view status.

Why it is used: To run highly available services that move between nodes when one fails.

# Install
apt install pacemaker corosync pcs

# Authenticate nodes
pcs host auth node1 node2

# Create cluster
pcs cluster setup mycluster node1 node2

# Start cluster
pcs cluster start --all

# Add resource
pcs resource create VIP ocf:heartbeat:IPaddr2 \
  ip=192.168.1.100 cidr_netmask=24 op monitor interval=30s

# Configure failover
pcs constraint location VIP rule score=200 \
  pingd 100

# View status
pcs status

Interview talking point: “The resource creation line defines an IPaddr2 resource with op monitor interval=30s, and that monitor operation, not the crash itself, is what triggers the periodic health checks.”

Q1211: How do you configure Pacemaker?

What it is: Pacemaker is the resource manager; pcs creates resources such as a virtual IP and an Apache service, then binds them with constraints and metadata.

Why it is used: To place, order, and automatically recover cluster services across nodes.

# Create resource
pcs resource create VirtualIP ocf:heartbeat:IPaddr2 \
  ip=192.168.1.100 cidr_netmask=24 op monitor interval=30s

pcs resource create WebService ocf:heartbeat:apache \
  configfile=/etc/apache2/apache2.conf \
  op monitor interval=30s

# Constraints
pcs constraint colocation add WebService VirtualIP INFINITY
pcs constraint order VirtualIP then WebService

# Stickiness
pcs resource meta WebService resource-stickiness=100

# Failover
pcs constraint location WebService prefers node1=50

# View status
pcs status
pcs resource show

Interview talking point: “Colocation keeps WebService with the virtual IP, order starts the IP first, resource-stickiness discourages relocation, and location prefers node1 with a score of 50.”

Q1212: How do you configure VRRP?

What it is: VRRP, configured here with keepalived, lets routers share a virtual IP: one instance is MASTER by priority, tracks an interface, and holds 192.168.1.100 until it fails.

Why it is used: Provides automatic failover of a gateway address between two hosts.

# keepalived config
vrrp_instance VI_1 {
    state MASTER
    interface eth0
    virtual_router_id 51
    priority 100

    virtual_ipaddress {
        192.168.1.100 dev eth0
    }

    track_interface {
        eth0
    }
}

Interview talking point: “Only one node owns the virtual IP at a time and priority decides the winner, so a backup takes over when track_interface sees eth0 drop; this is exactly how floating gateway setups remove the single point of failure.”

Q1213: How do you configure VRRP?

What it is: Keepalived implements VRRP to provide a floating virtual IP shared between redundant routers or servers.

Why it is used: Ensure high availability of critical network services like gateways or load balancers.

# Install keepalived
apt install keepalived

vrrp_instance VI_1 {
    state MASTER
    interface eth0
    virtual_router_id 51
    priority 100

    virtual_ipaddress {
        192.168.1.100/24 dev eth0
    }

    track_interface {
        eth0 weight -20
    }

    authentication {
        auth_type AH
        auth_pass secret123
    }

    notify_backup "/usr/local/bin/backup.sh"
    notify_fault "/usr/local/bin/fault.sh"
    notify_master "/usr/local/bin/master.sh"
}

# Backup configuration
# priority 90
# state BACKUP

Interview talking point: “The node with highest priority becomes master for the virtual IP; interface tracking can demote priority on link failure.”

Q1214: How do you use Corosync?

What it is: Corosync is the cluster messaging and membership layer. Its config sets totem protocol version, cluster name, transport, a node list with addresses and node IDs, and a quorum provider.

Why it is used: To provide heartbeat and quorum so cluster software knows which peers are alive.

totem {
    version: 2
    cluster_name: mycluster
    transport: udpu
}
nodelist {
    node {
        ring0_addr: 192.168.1.10
        nodeid: 1
    }
    node {
        ring0_addr: 192.168.1.11
        nodeid: 2
    }
}
quorum {
    provider: corosync_votequorum
}

Interview talking point: “Corosync decides who is in the cluster while Pacemaker decides what runs where; the votequorum provider stops the cluster acting when too few nodes remain to form quorum.”

Q1215: How do you use keepalived?

What it is: keepalived implements VRRP so several hosts share one virtual IP. This instance declares MASTER state on eth0, a virtual router ID, priority, advertisement interval, a password, and the virtual IP to hold on eth0.

Why it is used: To give a service one stable IP address that survives failure of any single node.

vrrp_instance VI_1 {
    state MASTER
    interface eth0
    virtual_router_id 51
    priority 100
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass password
    }
    virtual_ipaddress {
        192.168.1.100 dev eth0 label eth0:vip
    }
}

Interview talking point: “The virtual_ipaddress block is what clients connect to: whichever node currently holds VRRP master state owns that address, and it moves automatically when the master stops advertising.”

Q1216: How do you use Pacemaker?

What it is: Pacemaker is a cluster resource manager. pcs authenticates the nodes and forms a cluster, a virtual IP resource is created with a monitoring operation every 30 seconds, a location constraint is added, and pcs status reports state.

Why it is used: To keep services running by failing them over between nodes in a cluster.

# Setup
pcs host auth node1 node2
pcs cluster setup mycluster node1 node2

# Resources
pcs resource create VIP ocf:heartbeat:IPaddr2 ip=192.168.1.100 cidr_netmask=24 op monitor interval=30s

# Constraints
pcs constraint location VIP rule score=200 pingd 100

# Status
pcs status

Interview talking point: “The op monitor interval makes Pacemaker health-check the resource actively, so a lost virtual IP is detected and relocated within that cadence instead of waiting for someone to notice.”

Security Policy Authoring

Q1217: How do you configure seccomp?

What it is: seccomp filters which syscalls a process may make. Docker accepts a profile through —security-opt seccomp, either the default or a JSON file whose defaultAction is SCMP_ACT_ERRNO with a syscall list.

Why it is used: To restrict container syscalls so a compromised container cannot reach dangerous kernel entry points.

# Default profile
docker run --rm -it --security-opt seccomp=default hello-world

# Custom profile
cat /etc/docker/seccomp.json
{
    "defaultAction": "SCMP_ACT_ERRNO",
    "architectures": ["SCMP_ARCH_X86_64"],
    "syscalls": []
}

docker run --security-opt seccomp=/etc/docker/seccomp.json nginx

Interview talking point: “The sample profile sets defaultAction SCMP_ACT_ERRNO, so unlisted syscalls fail instead of running, and architectures pins x86_64. Passing seccomp=default applies Docker’s shipped profile while seccomp=/etc/docker/seccomp.json applies your own file.”

Q1218: How do you implementgrsecurity?

What it is: grsecurity is a hardened kernel patch set paired with PaX and the gradm policy manager. PaX flags are set per binary, and gradm loads compiled access policies.

Why it is used: To add exploit mitigations such as memory protection and randomization beyond the stock kernel.

# Install grsecurity
# Option 1: Use precompiled kernel
wget https:// kernels.org/pub/linux/kernel/v4.x/linux-4.14.12-grsec.tar.xz

# Option 2: Use PaX/gradm
apt install paxctl gradm

# PaX flags
paxctl -C /usr/bin/nginx
paxctl -m /usr/bin/nginx  # Enable MPROTECT
paxctl -s /usr/bin/nginx   # Enable SEGMEXEC
paxctl -r /usr/bin/nginx   # Enable RANDEX

# gradm configuration
# /etc/gradm/admin
# admin:password:0:0

# Enable learning mode
gradm -L /etc/gradm/learning
# Run application in learning mode
gradm -L /etc/gradm/learning -E /usr/bin/nginx

# Compile rules
gradm -F -O /etc/gradm/default.policies

# Enable
gradm -e nginx

Interview talking point: “gradm learning mode observes an application real accesses to generate a policy that you compile and enable later, instead of guessing every allowed path by hand.”

Q1219: How do you use seccomp?

What it is: seccomp filters which syscalls a process may make; Docker applies a JSON profile where defaultAction SCMP_ACT_ERRNO blocks everything except the listed allowed calls.

Why it is used: To shrink the kernel attack surface of containers by denying unused syscalls.

# Default profile
docker run --rm -it --security-opt seccomp=default hello-world

# Custom profile
cat > /etc/docker/seccomp.json <<EOF
{
    "defaultAction": "SCMP_ACT_ERRNO",
    "architectures": ["SCMP_ARCH_X86_64"],
    "syscalls": [
        {"names": ["read", "write"], "action": "SCMP_ACT_ALLOW"}
    ]
}
EOF

Interview talking point: “The custom profile sets defaultAction to SCMP_ACT_ERRNO and only allows read and write, so any syscall not named fails, which is deny-by-default rather than a deny list.”

Q1220: How do you configure secure boot?

What it is: UEFI Secure Boot verifies signed kernels; mokutil —import enrolls a Machine Owner Key, and mokutil —list-enrolled plus sbverify confirm what is trusted.

Why it is used: To ensure only signed kernels and boot loaders run on the machine.

# Install signed kernel
# Use UEFI secure boot

# Enroll keys
# MOK (Machine Owner Key)
mokutil --import key.der

# Verify
mokutil --list-enrolled
sbverify --list /boot/vmlinuz

Interview talking point: “MOK exists so users can enroll their own keys, letting a locally built kernel boot while Secure Boot stays enabled.”

Q1221: How do you harden the Linux kernel?

What it is: Kernel hardening tunes sysctl settings in /etc/sysctl.conf to close risky network behavior: enabling TCP SYN cookies, turning on reverse-path filtering, ignoring broadcast pings, and refusing to accept or send redirects.

Why it is used: To reduce the attack surface of a public-facing host against spoofing, floods, and redirect attacks.

# Network hardening
net.ipv4.tcp_syncookies = 1
net.ipv4.conf.all.rp_filter = 1
net.ipv4.conf.default.rp_filter = 1
net.ipv4.icmp_echo_ignore_broadcasts = 1
net.ipv4.conf.all.accept_redirects = 0
net.ipv4.conf.default.accept_redirects = 0
net.ipv4.conf.all.send_redirects = 0
net.ipv4.conf.default.send_redirects = 0

# Apply
sysctl -p

Interview talking point: “Editing sysctl.conf does nothing until sysctl -p reloads it, and the settings come in pairs like conf.all and conf.default so both existing and future interfaces are covered.”

Q1222: How do you implement secure bootstrapping?

What it is: Secure bootstrapping involves using TPM, secure boot, measured boot, remote attestation, encrypted secrets, sealed secrets, and HSMs for key protection.

Why it is used: These mechanisms ensure nodes and workloads boot in a trusted state with verified integrity from the start.

# Use TPM
# Secure boot
# Measure boot
# Use remote attestation

# Kubernetes
# Use encrypted secrets
# Use sealed secrets
# Use HSM for keys

Interview talking point: “Combining measured boot with remote attestation allows the control plane to verify a node’s integrity before allowing it to join the cluster.”

Q1223: How do you implement secure bootstrapping?

What it is: Secure bootstrapping ensures nodes and workloads start in a trusted state.

Why it is used: It establishes integrity from initialization through runtime operations.

# Use TPM
# Secure boot

# Measure boot
# Remote attestation

# Encrypted secrets
# Sealed secrets

# Use HSM
# Key management

Interview talking point: “Using TPM for measurements, secure boot, remote attestation, HSMs, and encrypted secrets like Sealed Secrets helps verify trust.”

Q1224: How do you implement TPM security?

What it is: tpm2-tools talks to the TPM chip: tpm2_createprimary makes a primary key, and tpm2_unseal releases sealed data bound to that key.

Why it is used: To keep keys and secrets in hardware so they never sit on disk as plain files.

# Install TPM tools
apt install tpm2-tools

# Create key
tpm2_createprimary -G rsa -o primary.key

# Seal
tpm2_unseal -c primary.key -o unsealed.key -i sealed.data

# Unseal
tpm2_unseal -c primary.key -i sealed.data

Interview talking point: “Sealing binds data to TPM state, so sealed material only unseals when the platform is in the expected condition.”

Q1225: How do you secure boot process?

What it is: Boot security covers UEFI secure boot key enrollment, a GRUB password hash, disabling USB boot, kernel parameters, and TPM measured boot with tpm2 tools.

Why it is used: To ensure only expected code runs from power-on and to detect tampering.

# UEFI secure boot
# Check status
mokutil --sb-state

# Enroll keys
mokutil --import key.der

# GRUB password
# Generate hash
grub-mkpasswd-pbkdf2
# Add to /etc/grub.d/40_custom
set superusers="admin"
password_pbkdf2 admin grub.pbkdf2.sha512...hash...

# Rebuild GRUB
update-grub

# Disable USB boot
# /etc/modprobe.d/blacklist-usb.conf
install usb-storage /bin/true

# Boot kernel parameters
# /etc/default/grub
GRUB_CMDLINE_LINUX="secure boot=1"

# TPM measured boot
# Install
apt install tpm2-tools

# Measure boot
tpm2_pcrread

# Verify
tpm2_quote -c -k key.file -g sha256 -f quote.out -q "my quote"

Interview talking point: “mokutil —sb-state reports whether secure boot is on, while grub-mkpasswd-pbkdf2 produces the hash you paste into 40_custom so editing the boot menu requires a password.”

Q1226: How do you secure kernel parameters?

What it is: sysctl settings tune kernel behavior at runtime and are persisted in /etc/sysctl.conf, then applied with sysctl -p or sysctl —system.

Why it is used: To harden the network stack and kernel against spoofing, redirects, and information leaks.

# Network security
net.ipv4.conf.all.rp_filter=1
net.ipv4.conf.default.rp_filter=1
net.ipv4.icmp_echo_ignore_broadcasts=1
net.ipv4.icmp_ignore_bogus_error_responses=1
net.ipv4.conf.all.accept_redirects=0
net.ipv4.conf.default.accept_redirects=0
net.ipv4.conf.all.send_redirects=0
net.ipv4.conf.default.send_redirects=0
net.ipv4.conf.all.accept_source_route=0
net.ipv4.conf.default.accept_source_route=0
net.ipv4.tcp_timestamps=0

# Kernel security
kernel.dmesg_restrict=1
kernel.kptr_restrict=2
kernel.yama.ptrace_scope=2
kernel.sysrq=0

# Memory protection
vm.mmap_min_addr=65536
vm.swappiness=10

# Apply
sysctl -p
sysctl --system

Interview talking point: “net.ipv4.conf.all.rp_filter=1 drops packets whose source would not route back, guarding against spoofed sources, while kptr_restrict and dmesg_restrict hide kernel pointers from unprivileged users.”

Q1227: How do you secure Linux kernel?

What it is: /etc/sysctl.conf holds tunable kernel parameters, and sysctl -p or sysctl —system applies them without rebooting. The shown settings enable SYN cookies, reverse-path filtering, and ignore ICMP redirects and broadcast echoes.

Why it is used: To harden the network stack against SYN floods, spoofed packets, and ICMP redirect attacks.

# Network
net.ipv4.tcp_syncookies = 1
net.ipv4.conf.all.rp_filter = 1
net.ipv4.conf.default.rp_filter = 1
net.ipv4.icmp_echo_ignore_broadcasts = 1
net.ipv4.conf.all.accept_redirects = 0
net.ipv4.conf.default.accept_redirects = 0

# Apply
sysctl -p
sysctl --system

Interview talking point: “sysctl -p reloads just the main config file while sysctl —system reads every sysctl.conf and sysctl.d drop-in in order, so use the latter after adding files under /etc/sysctl.d. These are runtime values that reset on reboot until written to disk.”

Q1228: How do you secure the kernel?

What it is: Kernel hardening in sysctl: install lines block unused filesystem modules, and network settings enable reverse path filtering and ignore broadcast ICMP echo requests.

Why it is used: Shrinks attack surface and hardens kernel networking behavior.

# Disable unused filesystems
install cramfs /bin/true
install squashfs /bin/true
install vfat /bin/true

# Network hardening
net.ipv4.conf.all.rp_filter = 1
net.ipv4.conf.default.rp_filter = 1
net.ipv4.icmp_echo_ignore_broadcasts = 1

Interview talking point: “rp_filter set to 1 enables reverse path filtering, dropping packets whose source would not be reachable on that interface, a common defense against spoofed sources.”

Q1229: How do you secure the Linux kernel?

What it is: Kernel hardening sets sysctl values that restrict dmesg and kernel pointer access, limit ptrace, disable sysrq, and tighten reverse path and redirect handling. modprobe install rules can neutralize unused filesystems, and IPv6 can be disabled.

Why it is used: To reduce the kernel attack surface on hosts that do not need those features.

# Kernel hardening
kernel.dmesg_restrict=1
kernel.kptr_restrict=2
kernel.yama.ptrace_scope=2
kernel.sysrq=0
net.ipv4.conf.all.rp_filter=1
net.ipv4.conf.default.rp_filter=1
net.ipv4.icmp_echo_ignore_broadcasts=1
net.ipv4.conf.all.accept_redirects=0
net.ipv4.conf.default.accept_redirects=0

# Disable unused filesystems
# /etc/modprobe.d/disable-filesystems.conf
install squashfs /bin/true
install udf /bin/true

# Disable IPv6 if not needed
# /etc/sysctl.conf
net.ipv6.conf.all.disable_ipv6=1
net.ipv6.conf.default.disable_ipv6=1

Interview talking point: “Setting kptr_restrict to 2 hides kernel pointers from even privileged readers, and the install rule pointing a filesystem at a stub binary makes loading that module a no-op instead of merely blacklisting it.”

Performance Profiling (advanced)

Q1230: How do you use flame graphs?

What it is: Flame graphs visualize stacked profiles: perf record samples a process, then stackcollapse-perf.pl and flamegraph.pl turn those samples into a flame.svg.

Why it is used: To see where a process spends CPU time at a glance across call stacks.

# Install
git clone https://github.com/brendangregg/FlameGraph.git

# Generate
# CPU profile
perf record -F 99 -p PID -g
perf script | ./FlameGraph/stackcollapse-perf.pl | ./FlameGraph/flamegraph.pl > flame.svg

# Memory
./flamegraph.pl --title="Memory Profile" < profile.txt > memory.svg

Interview talking point: “Sampling at -F 99 with -g records call graphs at about 99 hertz, giving representative stacks while adding little overhead to the target.”

Level 16: Production/SRE Design

Production Environment and SRE

Q1231: How do you handle production incidents?

What it is: A six-step incident workflow from detection through triage, communication, mitigation, resolution, and post-mortem, with a worked example of a downed database using systemctl and pg_ctl promote.

Why it is used: To respond to outages with a repeatable process instead of ad hoc panic.

# Incident response workflow
# 1. Detection
# - Alerts from monitoring
# - User reports

# 2. Triage
# - Assess severity (SEV1-4)
# - Identify impact
# - Determine if customer-facing

# 3. Communication
# - Create incident channel
# - Update status page
# - Notify stakeholders

# 4. Mitigation
# - Stop bleeding (rollbacks, traffic shift)
# - Apply fix

# 5. Resolution
# - Verify fix
# - Confirm recovery

# 6. Post-mortem
# - Document timeline
# - Identify root cause (5 whys)
# - Action items

# Example incident
# Database down
# 1. Check status
# systemctl status postgresql
# 2. Attempt restart
# systemctl restart postgresql
# 3. If failed, promote replica
# pg_ctl promote -D /var/lib/postgresql/data
# 4. Verify
# psql -c "SELECT 1"
# 5. Document

Interview talking point: “Triage assigns a severity such as SEV1 before work starts, which drives who gets paged and how fast. If restarting Postgres fails, pg_ctl promote turns the replica into primary, trading data loss for availability.”

Q1232: How do you implement auto-remediation?

What it is: Auto-remediation detects issues from alerts, classifies severity, triggers a runbook action, executes automation scripts, and verifies the fix afterward.

Why it is used: To resolve common incidents without waiting for a human responder.

# Auto-remediation
# 1. Detection
# Alerts

# 2. Classification
# Severity

# 3. Action
# Runbook

# 4. Automation
# Scripts

# 5. Verification
# Confirm fix

Interview talking point: “Verification is a distinct final step, because an automated fix that silently fails is worse than no action if it still closes the alert.”

Q1233: How do you implement auto-remediation?

What it is: Auto-remediation detects issues and automatically applies corrective actions based on runbooks.

Why it is used: It reduces mean time to recovery by fixing common issues without manual intervention.

# Detection
# Alerts

# Classification
# Severity

# Action
# Runbook

# Automation
# Scripts

# Verification
# Confirm fix

Interview talking point: “Classifying alerts and verifying fixes after remediation prevents automation from masking underlying problems.”

Q1234: How do you implement developer experience?

What it is: Developer experience covers local environments with Docker Compose and localstack, OpenAPI specs in Swagger UI, IDE language servers, fast unit and integration tests, and simple kubectl or ArgoCD deploys.

Why it is used: To shorten feedback loops so developers ship code faster with fewer blockers.

# Developer experience
# 1. Local development
# Docker Compose
# localstack

# 2. Documentation
# OpenAPI specs
# Swagger UI

# 3. IDE integration
# LSP servers
# Debugging

# 4. Testing
# Fast feedback
# Unit tests
# Integration tests

# 5. Deployment
# Simple commands
# kubectl
# ArgoCD

# Example: Developer workflow
# 1. Clone repo
# 2. Make changes
# 3. Run tests locally
# 4. Push to branch
# 5. CI runs tests
# 6. Merge to main
# 7. CD deploys

# Self-service
# Create environment
# Deploy app
# View logs
# Scale application

Interview talking point: “Tests run locally before push and again in CI, so breakage is caught before merge, and self-service extends to environment creation, viewing logs, and scaling the app.”

Q1235: How do you implement documentation?

What it is: A system document records purpose, OS, hardware, network addresses, a service table with ports and autostart, backup schedule and retention, monitoring URLs, and links to runbooks.

Why it is used: Give the team one place answering what runs where and how to recover it.

# System Documentation

## Overview
- Purpose: Production web server
- OS: Ubuntu 22.04 LTS
- Hostname: web01.example.com

## Hardware
- CPU: 4 vCPU
- RAM: 8 GB
- Disk: 100 GB SSD

## Network
- IP: 192.168.1.10
- Gateway: 192.168.1.1
- DNS: 8.8.8.8, 8.8.4.4

## Services
| Service | Port | Status | Auto-start |
|---------|------|--------|------------|
| nginx | 80, 443 | running | yes |
| php-fpm | 9000 | running | yes |
| mysql | 3306 | running | yes |

## Backups
- Schedule: Daily at 2 AM
- Retention: 30 days
- Location: /backup

## Monitoring
- Prometheus: http://monitoring:9090
- Grafana: http://monitoring:3000
- Alerts: #ops-alerts

## Runbooks
- [Service restart](runbooks/service-restart.md)
- [Disk full](runbooks/disk-full.md)
- [High CPU](runbooks/high-cpu.md)

Interview talking point: “The service table pairing port with auto-start status is what on-call reads first, and the runbook links turn the document from reference facts into executable recovery steps.”

Q1236: How do you implement platform engineering?

What it is: Platform engineering builds an internal developer platform: CI/CD pipelines, a service catalog like Backstage, Terraform and Helm templates, unified dashboards, and policy enforcement, all self-service.

Why it is used: To give developers ready-made golden paths instead of having them assemble infrastructure alone.

# Platform engineering
# Internal Developer Platform (IDP)
# Self-service

# Components
# 1. CI/CD pipelines
# GitHub Actions
# GitLab CI

# 2. Service catalog
# Backstage
# Port

# 3. Infrastructure templates
# Terraform modules
# Helm charts

# 4. Observability
# Unified dashboards

# 5. Security
# Policy enforcement

# Implementation
# Platform team builds tools
# Developers consume

# Benefits
# Faster deployments
# Consistency
# Security
# Reduced cognitive load

# Backstage
# Create catalog
# Service templates
# Documentation

Interview talking point: “The division of labor is that the platform team builds the tools and developers consume them, which cuts cognitive load while keeping deployments consistent and secure.”

Q1237: How do you implement reliability engineering?

What it is: Reliability engineering sets SLOs such as 99.9 percent availability or p99 under 200ms, spends an error budget, cuts toil, and adds canaries, circuit breakers, and self-healing.

Why it is used: To balance feature delivery speed against uptime with a measurable shared target.

# Reliability engineering
# SRE principles
# 1. SLOs (Service Level Objectives)
# - Availability: 99.9%
# - Latency: p99 < 200ms

# 2. Error budgets
# 100% - SLO = error budget
# If budget exhausted, freeze features

# 3. Toil reduction
# Automate manual tasks

# 4. Post-mortems
# Blameless
# Focus on process improvement

# 5. Releases
# Canary deployments
# Feature flags

# 6. Circuit breakers
# See earlier

# 7. Bulkheads
# Isolate failures

# 8. Self-healing
# Restart failed pods
# Replace unhealthy nodes

Interview talking point: “The error budget is 100 percent minus the SLO, and once it is exhausted features freeze, which turns availability into an objective release gate rather than a vague goal.”

Q1238: How do you implement SRE practices?

What it is: SRE practices cover error budgets, toil reduction, observability across metrics, logs, and traces, incident response, and gradual rollouts, supported by Prometheus, Grafana, Jaeger, and Loki.

Why it is used: To operate services predictably while keeping on-call work and reliability under control.

# SRE practices
# Error budgets
# https://sre.google/sre-book/availability-table/

# Toil management
# Identify
# Quantify
# Automate
# Eliminate

# Observability
# Metrics
# Logs
# Traces

# Incident management
# On-call rotation
# Runbooks
# Post-mortems

# Change management
# Canary releases
# Gradual rollouts

# SRE tools
# Prometheus
# Grafana
# Jaeger
# Loki

# On-call
# PagerDuty
# OpsGenie

# Automation
# Ansible
# Terraform
# Kubernetes

Interview talking point: “Toil management follows identify, quantify, automate, eliminate, and the three observability signals each map to a different tool, so no single tool answers every question.”

Q1239: How do you implement SRE practices?

What it is: SRE practices implementation involves defining error budgets and SLOs, reducing toil, implementing monitoring, conducting post-incident reviews, and enabling automation.

Why it is used: It balances reliability goals with feature development velocity.

# Error budgets
# Define

# SLOs
# Set

# Toil
# Reduce

# Monitoring
# Implement

# Post-incident
# Review

# Automation
# Enable

Interview talking point: “Error budgets create a shared incentive, as they quantify how much risk is acceptable before prioritizing reliability work.”

Q1240: How do you use Backstage templater?

What it is: A Backstage scaffolder Template declaring a name parameter and a fetch-template step that renders values from that parameter into the template.

Why it is used: Scaffolds new services consistently through parameterized automation steps.

apiVersion: scaffolder.backstage.io/v1beta3
kind: Template
metadata:
  name: my-template
spec:
  owner: myteam
  type: service
  parameters:
  - title: Name
    properties:
      name:
        type: string
  steps:
  - id: fetch-template
    action: fetch:template
    name: Fetch template
    input:
      url: ./template
      values:
        name: ${{ parameters.name }}

Interview talking point: “Parameters are collected up front and then referenced inside steps with the expression syntax, so one template produces different output per run with no file edits.”

Q1241: How do you use Backstage?

What it is: Backstage runs as a container on port 7000, has plugins created with backstage-cli, and loads components from catalog-info.yaml URLs registered in app-config.

Why it is used: To give teams a developer portal listing services, owners, and tooling from a central catalog.

# Install
docker run -it -p 7000:7000 backstage

# Create plugin
backstage-cli create-plugin --name my-plugin

# Add to app
# app-config.yaml
catalog:
  locations:
  - type: url
    target: https://github.com/org/repo/catalog-info.yaml

Interview talking point: “The catalog section lists a URL target pointing at catalog-info.yaml in a repo, so registering a service is a file addition rather than a database entry, and create-plugin extends the portal itself.”

High Availability and Failover Design

Q1242: How do you configure health checking?

What it is: Health checking removes unhealthy backends: HAProxy probes a /health endpoint with rise and fall counts, a shell script exits non-zero on failure, and keepalived adjusts priority by weight.

Why it is used: To keep traffic away from failing instances automatically and restore them when they recover.

# HAProxy health check
backend servers
    balance roundrobin
    option httpchk GET /health
    http-check expect status 200
    server s1 192.168.1.10:8080 check inter 2000 rise 2 fall 3
    server s2 192.168.1.11:8080 check inter 2000 rise 2 fall 3 slowstart 10s

# Custom health check script
# healthcheck.sh
#!/bin/bash
curl -sf http://localhost:8080/health || exit 1

# Using monitor-ptr in nginx
# nginx.conf
upstream backend {
    server 192.168.1.10:8080;
    server 192.168.1.11:8080;
    server 192.168.1.12:8080 backup;
}

# Keepalived health check
vrrp_script check_service {
    script "/usr/local/bin/healthcheck.sh"
    interval 5
    weight -10
}

Interview talking point: “inter 2000 rise 2 fall 3 means two good probes bring a server up and three bad ones take it out, while slowstart 10s ramps its traffic back gradually.”

Q1243: How do you implement health checks?

What it is: Kubernetes probes check containers: liveness restarts a sick pod, readiness gates traffic, and startup allows slow booting with a longer failure budget.

Why it is used: To keep broken pods out of service and restart hung ones automatically.

# Liveness probe
livenessProbe:
  httpGet:
    path: /healthz
    port: 8080
  initialDelaySeconds: 30
  periodSeconds: 10
  failureThreshold: 3

# Readiness probe
readinessProbe:
  httpGet:
    path: /ready
    port: 8080
  initialDelaySeconds: 5
  periodSeconds: 5
  failureThreshold: 3

# Startup probe
startupProbe:
  httpGet:
    path: /healthz
    port: 8080
  failureThreshold: 30
  periodSeconds: 10

Interview talking point: “A failing liveness probe restarts the container while a failing readiness probe only removes it from endpoints, so the two probes answer different questions about the same pod.”

Q1244: How do you test high availability?

What it is: HA testing verifies failover by stopping the cluster on the active node and confirming the virtual IP moved. pcs resource move relocates a service deliberately, and status commands show where resources currently run.

Why it is used: To prove the failover path actually works before a real outage depends on it.

# Test failover
# Node 1: Primary
pcs status

# Stop cluster on node 1
pcs cluster stop node1

# Verify IP moved to node 2
ip addr show
pcs status

# Test resource
pcs resource move WebService node2
pcs status

# Failover test script
#!/bin/bash
echo "Starting failover test..."
CURRENT=$(crm_mon -1 | grep -A1 "Master/Slave Set:" | tail -1 | awk '{print $3}')
echo "Current primary: $CURRENT"

if [ "$CURRENT" == "node1" ]; then
    pcs cluster stop node1
    sleep 10
    echo "Checking failover..."
    pcs status | grep "VirtualIP"
else
    pcs cluster stop node2
    sleep 10
    pcs status | grep "VirtualIP"
fi

echo "Failover test completed"

Interview talking point: “Stopping node1 and then checking ip addr and pcs status confirms the VIP followed, and a scripted test that stops whichever node currently holds the resource can be rerun routinely.”

Q1245: How do you configure high availability?

What it is: High availability uses a load balancer like HAProxy or NGINX, clustering with Pacemaker and Corosync, Keepalived for a virtual IP, and replicated databases.

Why it is used: To keep the service running when individual servers or a database node fail.

# Load balancer
# HAProxy, NGINX

# Clustering
# Pacemaker + Corosync
# Keepalived for VIP

# Database
# Master-slave or multi-master
# Connection pooling

Interview talking point: “Keepalived floats a virtual IP between nodes, so clients keep one address while the machine behind it changes without any client reconfiguration.”

Q1246: How do you implement data replication?

What it is: Replication options include GlusterFS replica volumes created across bricks, and Ceph pools where ceph osd pool set size controls the replica count.

Why it is used: To survive node or disk loss by keeping multiple copies of every piece of data.

# Use GlusterFS
gluster volume create replica 2 \
  node1:/brick1/brick \
  node2:/brick1/brick

# Use Ceph
ceph osd pool create mypool 100 100
ceph osd pool set mypool size 3

# Use Rook
# Configure in CephCluster CRD
spec:
  storage:
    useAllNodes: true
    useAllDevices: true
    deviceFilter: sd*

Interview talking point: “GlusterFS sets replication at volume creation with replica 2, while Ceph adjusts it later with pool set size 3, so the knob timing differs.”

Q1247: How do you implement high availability?

What it is: High availability building blocks: load balancers like HAProxy and NGINX, clustering with Pacemaker, Corosync and Keepalived, replicated databases, stateless design, and session affinity.

Why it is used: Removes single points of failure so service continues when components fail.

# Load balancer
# HAProxy, NGINX

# Clustering
# Pacemaker/Corosync
# Keepalived

# Database
# Master-slave
# Multi-master

# Application
# Stateless design
# Session affinity

Interview talking point: “Stateless application design makes failover simple, while session affinity is the concession you make when application state cannot be removed from the tier.”

Q1248: How do you implement split-brain prevention?

What it is: Split-brain prevention uses corosync quorum voting with expected_votes and two_node settings, a third tie-breaker node, and recovery commands to stop one side.

Why it is used: To stop two halves of a cluster from both claiming authority after a network partition.

# Quorum configuration
quorum {
    provider: corosync_votequorum
    expected_votes: 2
    two_node: 1
}

# Ping node for quorum
quorum {
    provider: corosync_votequorum
    expected_votes: 3
    wait_for_all: 1
}

# Tie-breaker
totem {
    interface {
        ringnumber: 0
        transport: udpu
    }
}

nodelist {
    node {
        ring0_addr: node1
        nodeid: 1
    }
    node {
        ring0_addr: node2
        nodeid: 2
    }
    node {
        ring0_addr: node3
        nodeid: 3
    }
}

# Split-brain recovery script
# /usr/local/bin/split-brain-recovery.sh
pcs cluster stop node1
pcs cluster start node2

Interview talking point: “With two nodes you enable two_node because a plain majority is unreachable during a partition, and adding a third node restores a real quorum that can break the tie.”

Q1249: How do you optimize Linux for high availability?

What it is: Linux high availability tuning with Keepalived, HAProxy, configured health checks, comprehensive monitoring, and regular testing.

Why it is used: Keeps services reachable when a node or upstream fails.

# Linux HA optimization
# 1. Keepalived
# Configure

# 2. HAProxy
# Tune

# 3. Health checks
# Configure

# 4. Monitoring
# Comprehensive

# 5. Testing
# Regular

Interview talking point: “Keepalived handles virtual IP failover while HAProxy distributes traffic to healthy backends, so they solve different halves of availability. Health checks are what connect the two, since failover only works if failure detection is accurate.”

Q1250: How do you optimize Linux for high availability?

What it is: Linux high availability optimization configures Keepalived, tunes HAProxy, sets up health checks, enables comprehensive monitoring, and performs regular testing.

Why it is used: To minimize downtime and maintain consistent service availability.

# Keepalived
# Configure

# HAProxy
# Tune

# Health checks
# Configure

# Monitoring
# Comprehensive

# Testing
# Regular

Interview talking point: “Health checks and monitoring detect failures quickly, while HAProxy tuning and regular testing ensure failover works as intended.”