Engineering Distributed Systems in a Local Environment

Module 5 – Multi-Node Lab Design

One Server Is Not a System

Production environments are not single machines.

They are:

Application nodes
Database nodes
Monitoring nodes
Load balancers
Bastion hosts

If your lab only has one VM, you are not learning distributed behavior.

This module introduces controlled multi-node design inside your virtualized environment.

1. Architecture Philosophy

We will build:

Node 1 → Application Node
Node 2 → Database Node
Node 3 → Monitoring Node (optional extension)

Each node has:

Defined responsibility
Controlled network exposure
Segmented communication

This mirrors real-world infrastructure.

2. Network Design for Multi-Node Systems

You configured segmentation in Module 2.

Now we apply it.

Recommended design:

Node 1:

NAT + Host-only

Node 2:

Host-only only

Node 3 (optional):

NAT + Host-only

Why?

NAT allows updates and package installs
Host-only allows internal communication
Database node does NOT require internet

This models private subnet design.

3. Node Role Separation

Define responsibilities clearly.

Application Node

Purpose:

Runs web server
Hosts API
Serves traffic

Should:

Accept HTTP/HTTPS
Connect internally to database

Should NOT:

Store persistent database files

Database Node

Purpose:

Stores structured data
Internal-only service

Should:

Accept traffic only from application node
Never be exposed via Bridged adapter

Database exposure is a common production mistake.

Monitoring Node (Optional Advanced Step)

Purpose:

Collect logs
Monitor CPU/memory/disk
Observe system health

Separating monitoring increases operational clarity.

4. VM Cloning Strategy

Do not reinstall OS for each node.

Use snapshot from Module 3:

03-hardened-base

Clone the VM:

VirtualBox → Clone → Linked Clone (for lab efficiency)

Rename:

app-node-01
db-node-01
monitor-node-01

This ensures consistent configuration across nodes.

Consistency reduces unpredictable behavior.

5. Internal Communication Validation

After booting nodes:

Check host-only IPs:

ip a

Test connectivity:

From app-node:

ping <db-node-host-only-ip>

From db-node:

ping <app-node-host-only-ip>

If nodes cannot communicate internally, segmentation is broken.

6. Simulating Real Service Flow

Example setup:

Install web server on app-node:

sudo dnf install nginx -y
sudo systemctl enable --now nginx

Install database on db-node:

sudo dnf install mariadb-server -y
sudo systemctl enable --now mariadb

Do not expose database externally.

Bind database service to host-only IP only.

This enforces internal-only communication.

7. DNS & Name Resolution

Instead of using IP addresses directly, configure /etc/hosts.

On each node:

192.168.56.10 app-node-01
192.168.56.11 db-node-01

This simulates internal DNS.

Production systems do not rely solely on IPs.

8. Resource Planning Across Nodes

Monitor resource distribution:

free -h
htop

Multiple VMs stress your host system.

If host CPU spikes:

Reduce VM cores
Adjust RAM allocation
Balance node distribution

Multi-node design requires hardware awareness.

9. Failure Simulation

Now test system resilience.

Simulate:

Stop database service
Kill web server
Disable host-only adapter temporarily

Observe:

How failures propagate
What breaks first
What remains functional

Distributed systems fail differently than single-node systems.

Understanding propagation is critical.

10. Security Reinforcement

Ensure:

Database port not exposed via NAT
Firewall rules restrict internal traffic
SSH access controlled per node

Check open ports:

sudo ss -tulnp

Multi-node architecture increases attack surface.

Hardening must scale with topology.

11. Snapshot Strategy in Multi-Node Design

Take snapshots at:

Clean base clone
After role-specific configuration
Before major changes

Example naming:

app-node-01-configured
db-node-01-configured

Version each node separately.

Infrastructure versioning must remain controlled.

12. Lab Assignment

Clone hardened base VM.
Create:
- app-node-01
- db-node-01
Configure:
- NAT + Host-only for app
- Host-only only for db
Install web server on app.
Install database on db.
Verify internal connectivity.
Simulate failure by stopping database.
Document observed behavior.

Deliverable:

Explain:

Why database node should not have NAT.
What happens if database is exposed via Bridged.
How this design maps to cloud private subnets.

If you cannot explain role separation, you are not designing architecture.

13. Production Reflection

Consider:

How does this compare to AWS public/private subnet design?
Where would a load balancer fit?
What happens under horizontal scaling?
How would you introduce redundancy?

Distributed systems require intentional topology.

Module Completion Criteria

You are ready for Module 6 when:

Nodes communicate internally.
Roles are separated clearly.
Services are not overexposed.
Failure propagation is understood.
Snapshots are versioned per node.

→ Module 6 – Observability Foundations