Modernizing and Optimizing High-Traffic Moodle Infrastructure for Peak Exam Concurrency

Overview

Scaling a Moodle Learning Management System (LMS) to withstand sudden, high-density traffic spikes—such as thousands of concurrent students concurrently launching an exam, navigating quiz modules, and executing rapid auto-saves—requires a carefully balanced infrastructure.

During a high-stakes exam event serving roughly 2,000 concurrent users generating over 137,000 interactions within a 24-hour window, a single-server Linux/Docker architecture running a standard PHP-FPM, MySQL, and Redis stack was systematically tuned. This article outlines the real-time operational challenges encountered, the metrics observed, the critical bottlenecks identified (including a catastrophic disk I/O death spiral), and the final stabilized configuration blueprint.

---

The Infrastructure Stack

The architecture utilized a 32GB RAM host machine partitioning workloads across isolated Docker environments:

• Application Node: PHP 8.x + FPM + Nginx + Redis client.
• Database Node: Dedicated MySQL Instance utilizing LVM (Logical Volume Manager) storage allocations.
• Session & Cache Layer: Redis-server caching session state to prevent database and disk bloat.
• Analytics Layer: Matomo real-time monitoring to gauge user actions and request depth.

---

Phase 1: Handling the Initial Surge (Optimizing Throughput)

At the onset of the exam period, Matomo analytics recorded rapid user engagement scaling to 1,383 visits and 31,201 actions every 30 minutes, with students executing an average of 28 actions per visit (heavily concentrated inside Moodle’s interactive /mod/quiz engine).

1. PHP-FPM Process Management Configuration

To counteract process-spawning latency—where the CPU wastes cycles dynamically generating worker threads during a massive traffic wave—the PHP-FPM pool manager was locked into static mode. This kept workers permanently "warmed up" in RAM.

pm = static
pm.max_children = 150
pm.max_requests = 1000
Copy

• Result: System CPU overhead (%sy) remained low at 11.0% to 15.3%, proving that pre-forking workers eliminated thread management overhead.

2. OPcache Tuning

To ensure the application server didn't continuously recompile PHP code scripts across hundreds of thousands of page views, aggressive OPcache values were implemented:

opcache.memory_consumption = 512
opcache.max_accelerated_files = 25000
opcache.validate_timestamps = 0  ; Disabled during exams to eliminate disk stats
Copy

• Result: Achieved a 99.95% OPcache hit rate, insulating the CPU from repetitive code compilation.

3. Database Connection Headroom

With 150 static workers handling multiple waves of 1,500 active students, real-time MySQL monitors showed active database connections surging from 310 up to 470+ concurrent connections. The database boundary was raised dynamically to maintain buffer capacity:

SET GLOBAL max_connections = 800;
Copy

• Result: Prevented critical "Too Many Connections" errors, leaving a safe buffer above the 470-connection peak.

---

Phase 2: The Cascading Failure (Memory Exhaustion & I/O Wait Death Spiral)

As the exam day progressed into a second peak window, individual Moodle PHP processes grew larger than calculated, consuming more memory per worker due to heavy data-payload processing. The system hit a critical failure threshold characterized by:

• Total Memory Exhaustion: Physical RAM plummeted to 206.8 MiB free.
• Kernel Thrashing (kswapd): The kernel threads kswapd0 and kswapd1 pinned CPU capacity trying to force active memory pages out to the disk.
• I/O Wait Death Spiral: The host hit an alarming 53.0% wa (I/O Wait). The CPU sat paralyzed, waiting for a slow physical disk swap partition to read/write memory data.
• Uninterruptible Sleep (D State): Critical php-fpm workers entered the dreaded D execution state, locked by the kernel until disk I/O requests cleared.
• Storage Exhaustion: Concurrently, an LVM data partition (/mnt/docuseal/www/lms06data) hit 100% capacity (49GB used), preventing Moodle from compiling updated local style caches (localcache), causing the application UI's CSS to break entirely.

---

Phase 3: The Recovery Action Plan

To rescue the server without losing student exam states, a sequential recovery protocol was executed:

1. Breaking the Swap Storm

The application container was forcefully isolated and restarted to clear out the blocked D state worker backlogs and release the saturated physical RAM.

docker kill moodle-php8-redis-1
docker start moodle-php8-redis-1
Copy

• Result: Free physical RAM immediately normalized back to 27.5 GB free.

2. Resizing the Saturated Volume

Using LVM flexibility, storage space was hot-allocated away from an underutilized volume group directly into the choked Moodle data mount without introducing infrastructure downtime:

sudo lvextend -L +20G /dev/ubuntu-vg/lmsdata
sudo resize2fs /dev/ubuntu-vg/lmsdata
Copy

3. Clearing Cache and Trash Bloat

Once file write operations were restored by the volume expansion, Moodle's internal filesystem was aggressively cleaned via the Command Line Interface (CLI) to restore clean state execution:

docker exec -u www-data moodle-php8-redis-1 php /var/www/html/admin/cli/empty_trash.php
docker exec -u www-data moodle-php8-redis-1 php /var/www/html/admin/cli/purge_caches.php
Copy

4. Resetting the Worker Safety Margin

To ensure the server would not slide back into memory saturation, the static pool size was dialed down to match the physical RAM threshold:

pm.max_children = 100  ; Safeguarded a 5GB to 8GB operational RAM buffer
Copy

---

Summary Architectural Checklist for Future Scaling

The operational lessons distilled from this high-traffic event provide a rigid blueprint for future Moodle exam infrastructure deployment:

Optimization Layer	Metric / Target	Action Taken / Core Configuration
Session State	Redis Server	Offload sessions completely to RAM via Redis; prevents disk lockouts during high action-per-minute user paths.
PHP Execution	pm = static	Pre-allocate workers. Scale the worker count tightly (max_children = 100) to guarantee execution remains entirely within physical RAM.
Storage (LVM)	Monitor Disk Space	Ensure localcache and temp directory structures have ample scaling capacity; ensure standard log-purging routines are operational.
Database Limit	max_connections = 800	Scale connection ceilings well beyond the absolute count of active PHP workers to prevent query starvation.
Horizontal Scaling	Target for >1,500 Users	When single-node workloads sustain CPU load spikes exceeding normal hardware thresholds, transition the PHP processing tier behind an Nginx load balancer across multi-node application clusters.