# Markatty VPS — Investigation, Changes & Stabilization Plan

Living record of the July 2026 CPU-saturation investigation, the fixes applied, every
change made to the production server (with backup locations + revert commands), and the
plan to make the platform stable under real concurrency.

Server: **AlmaLinux 9.8, cPanel 136, 8 cores, 15 GB RAM, no swap.**
Real IP: **160.153.179.163** (the WHM hostname `163.179.153.160.host.secureserver.net`
embeds the *reverse* octet order — the literal `163.179.153.160` is a DIFFERENT machine).
SSH: `ssh markatty` (user `mofathy`, wheel + NOPASSWD sudo, key `~/.ssh/id_ed25519_markatty`).

Three cPanel accounts share the box: `markattystore` (410 domains, the busy one),
`markattyinfo`, `markattybiz`. Each runs a **separate plain-file deployment** with its own
database (`markattystore_merged`, `markattyinfo_1`, `markattyinfo_12`). Not git repos on
the server — deploys are file copies.

---

## 1. Root cause of the 100% CPU (SOLVED)

**Infinite recursion in `Webkul\Core\Core::getCurrentChannel()`.** Any request for a domain
with no matching `company` row recursed until the PHP-FPM worker died, pinning one core the
whole time. Proven by a php-fpm slowlog stack trace, not by guesswork:

```
getCurrentChannel() -> Cache::remember('channel:host:<md5>') MISS -> closure
  -> Company::getCurrent() -> 'domain_not_found'
    -> trans() (building the error response)
      -> CustomTranslationServiceProvider::get() -> getCurrentChannel()
        -> same key, STILL a miss (remember() hasn't written yet) -> loop forever
```

- Onset ~**Jul 8**; escalated to 14+ hours/day above 90% by Jul 13–14. Before Jul 8 the box
  never exceeded 90% in a month of `sar` history.
- Manifested as: 512 MB memory-exhausted fatals, 30s/600s max-execution fatals, and
  core-pinning runaways — all the same recursion hitting different limits.
- **Fixed** by a re-entrancy guard (`self::$resolvingChannel`) — commit `acb105ec7e`,
  deployed to `markattystore` **2026-07-16 01:22 UTC**. Verified: the 04:00–08:00 window
  (previously pegged) stayed <2% user / ~98% idle; zero recursion in the slowlog since.

### Disproven theories (do not revisit)
- PM2 `bonik`/`balmy` EADDRINUSE crash loop as the chronic cause — it was a 25-min
  today-only event, though those Node apps *were* burning ~2 cores/day and are now stopped.
- AI import as the primary cause — Jul 8–14 were saturated with 0–1 import requests.
- Traffic volume — 19h served 8,828 req at ~3% CPU vs 14h's 10,328 at ~95%. Flat traffic,
  50× CPU swing. It was never a capacity problem; it was the bug.
- Wrong-class trap: recursion runs through `Webkul\Core\Core::getCurrentChannel()`.
  `SAASCustomizer\Company::getCurrentChannel()` is a DIFFERENT method that does NOT recurse.

### Still vulnerable
`markattyinfo` and `markattybiz` run older separate copies WITHOUT the recursion fix. Bounded
by the 600s cap (below) + their low traffic, so deferred. Fix = same guard applied to
`/home/markattyinfo/public_html/prod/Code/html2/.../Core.php` and the biz equivalent
(biz uses a direct-query version, not `Cache::remember`, but the guard still applies).

---

## 2. Code fixes — all on branch `hot-fix` (GitLab `m7moodali88/markatty`)

| Commit | Fix | Deploy status |
|---|---|---|
| `acb105ec7e` | Recursion guard in `Core::getCurrentChannel()` | **Deployed to store 01:22** |
| `3a058cfe26` | Hoist FTP connection out of per-image loop in `uploadImagesToCdn()` | In git (author's) |
| `f4b8495de1` | Guard `getAdsMetaData()` against null channel (fixes error-page 500) | In git — deploy |
| `606538e573` | Guard `StoreDesignController` ad-count calls against null channel | In git — deploy |
| `0bedda7761` | Fix admin orders API 500 from shipments w/ orphaned customer morph | In git — deploy |
| `60e75a705c` | Show parent product name + variation & main SKU for configurable items | In git — deploy |
| `dfb64024db` | Rate-limit public company auth/registration routes (throttle) | In git — **staging-test then deploy** |
| `59c99d4664` | Cache getSuperConfigData() — removes per-page DB queries platform-wide | In git — deploy |

Confirm what's live before assuming: `grep -c resolvingChannel <path>/Core.php` (4 = fixed).

---

## 3. Server changes made — with BACKUPS and REVERT commands

**FPM tuning is now DURABLE (2026-07-20).** The slowlog/cap/capacity settings previously
lived in direct `.conf` edits that a cPanel rebuild wiped. They are now baked into cPanel's
own config layer, so `php_fpm_config --rebuild` regenerates them instead of erasing them:
- **Per-domain YAMLs** `/var/cpanel/userdata/{markattystore,markattyinfo,markattybiz}/<domain>.php-fpm.yaml`
  carry `pm_max_children: 15`, `pm_max_requests: 500`, `request_terminate_timeout: 600s`,
  `request_slowlog_timeout: 10s`, `slowlog: /var/log/php-fpm-slow/<domain>_.log`.
  (Confirmed: cPanel passes these custom directives through to the generated `.conf`.)
- **System defaults** `/var/cpanel/ApachePHPFPM/system_pool_defaults.yaml` sets
  `pm_max_children: 15`, `pm_max_requests: 500`, `pm_process_idle_timeout: 10`,
  `request_terminate_timeout: 600s` — so ANY newly-created pool is protected by default.
- Rebuild after editing YAMLs: `sudo /usr/local/cpanel/scripts/php_fpm_config --rebuild`
  then `sudo systemctl reload ea-php80-php-fpm ea-php82-php-fpm`.
- **YAML backup (pre-durability):** `/root/fpm-yaml-backup-20260720-101843/` (291 files).
- Verified 2026-07-20: all **291** pools have max_children=15, terminate_timeout, slowlog.
- Also: `meragluten.markatty.store` was flagged `php_fpm=1` but had no pool conf (half-
  provisioned); re-enabled via `whmapi1 php_set_vhost_versions vhost=meragluten.markatty.store
  version=ea-php80 php_fpm=1`, now has the full tuned pool.

### 3.1 Slowlog + terminate-timeout (all 288 pools)
Added to every pool conf:
```
request_slowlog_timeout = 10s
slowlog = /var/log/php-fpm-slow/<pool>.log
request_terminate_timeout = 600s      # kills any runaway at 10 min (was unlimited)
```
- **Backup:** `/root/fpm-backup-20260715-225745/` (266 files)
- **Revert:**
  ```bash
  sudo cp -a /root/fpm-backup-20260715-225745/. /opt/cpanel/ea-php80/root/etc/php-fpm.d/
  sudo systemctl reload ea-php80-php-fpm ea-php82-php-fpm
  ```

### 3.2 PHP-FPM capacity (markattystore pools) — Phase A
```
pm.max_children = 15     # was 5 -> 10 (15:17) -> 15 (15:51, after swap added)
pm.max_requests = 500    # was 20
```
- **Backups (chronological):**
  - `/root/fpm-pm-backup-20260716-151724/` — captures the original 5/20 state
  - `/root/fpm-pm-backup-20260716-155108/` — captures the 10/500 state
- **Revert to original 5/20:**
  ```bash
  sudo cp -a /root/fpm-pm-backup-20260716-151724/. /opt/cpanel/ea-php80/root/etc/php-fpm.d/
  sudo systemctl reload ea-php80-php-fpm ea-php82-php-fpm
  ```

### 3.3 Stopped PM2 apps (markattystore) — bonik/balmy Next.js storefronts
Were crash-looping / burning ~2 cores/day. Only served
alleya/timetravelertest/fantasylandtest.markatty.store (not live customers).
- **Revert:** `sudo -u markattystore bash -lc "pm2 start bonik balmy && pm2 save"`

### 3.4 SSH access created
- User `mofathy` (uid 1005), wheel group, key-only. Sudoers: `/etc/sudoers.d/90-mofathy`
  (`mofathy ALL=(ALL) NOPASSWD: ALL`). Key file is root-equivalent — guard it.
- **Revert:** `sudo userdel -r mofathy; sudo rm /etc/sudoers.d/90-mofathy`

### 3.6 Swap (4 GB) — added 2026-07-16
`/swapfile` (4 GB, dd-created, XFS-safe), persisted in `/etc/fstab`, `vm.swappiness=10`
(in `/etc/sysctl.conf`). OOM safety net; unlocks `max_children` 15–20.
- **Revert:**
  ```bash
  sudo swapoff /swapfile && sudo rm /swapfile
  sudo sed -i '\#^/swapfile #d' /etc/fstab
  sudo sed -i '/^vm.swappiness/d' /etc/sysctl.conf
  ```

### 3.7 Blocked IP — scraper (2026-07-21)
`157.143.3.35` — headless-Chrome scraper (AS8758 datacenter, spoofed Android UA).
7,410 requests in 11h; a 1,485-request burst in 20 min drove CPU to **95% (04:00–04:20 UTC)**
— the first real saturation since the recursion fix. Not malicious (no vuln probing), but it
hammered `api.markatty.store` catalog/homepage endpoints, which are expensive because the
CDN JSON cache is broken (see the `.com`/`.store` bug in §7).
- Blocked via `iptables -I INPUT 1 -s 157.143.3.35 -j DROP`.
- **Persistence:** no csf/firewalld/iptables-services on this box, so rules do NOT survive
  reboot on their own. Re-applied by `/root/blocked-ips.sh` (idempotent) via a `@reboot`
  root cron. Add future blocks to that script.
- Root crontab backed up to `/root/root-crontab-backup-<timestamp>.txt`.
- **Revert:** `sudo iptables -D INPUT -s 157.143.3.35 -j DROP` and remove the line from
  `/root/blocked-ips.sh`.
- NOTE: the DROP counter was still 0 at time of writing — the bot went idle ~11:25, before
  the rule was added. Rule is correct but not yet empirically observed dropping traffic.

### 3.5 Archived files (non-destructive moves, restore if needed)
- `.save` pool-conf snapshots → `/root/fpm-conf-save-archive-20260715-235535/` (272 files)
- Slowlog evidence (recursion + FTP traces) → `/root/slowlog-evidence-20260716/`

---

## 4. Why it's still architecturally fragile (the real problem)

Even with the recursion gone, the platform can't absorb normal concurrency. It's not mainly
CPU — it's **worker starvation from synchronous blocking I/O**:

- `pm.max_children` (now 10, was 5) = hard concurrency ceiling per tenant.
- A product edit does a **synchronous FTP upload to the CDN** (~9s, sometimes minutes),
  holding a worker the whole time doing nothing but waiting on the network.
- A few concurrent slow requests exhaust the pool → the tenant appears frozen, even at low CPU.

Confirmed config smells (all in `markattystore` `.env` / config):
| Setting | Current | Effect |
|---|---|---|
| `QUEUE_DRIVER` / `QUEUE_CONNECTION` | `sync` | Every job runs inline in the web request |
| `CACHE_DRIVER` / `SESSION_DRIVER` | `file` | Every request takes `flock` — serializes under load |
| `pm.max_children` | 10 | Small ceiling; slow requests hold slots for full duration |
| PHP version | 8.0 (EOL) | Slower, unsupported |
| Swap | none | A memory spike OOM-kills instead of degrading |

Existing assets to build on: Redis is **running** (`PONG`, no PHP client yet); 5 `ShouldQueue`
job classes already exist and pass tenant context via constructor params (correct pattern),
running inline only because the driver is `sync`. It's a single Laravel install serving all
410 domains (one DB, one cache dir) — so cache keys are already tenant-namespaced.

---

## 5. Stabilization plan (dependency-ordered)

**Phase A — FPM headroom + swap** · low risk · *max_children/max_requests DONE (3.2); swap pending.*
Add 4 GB swap (safe, reversible) to unlock `max_children` 15–20. Currently capped at 10 due
to 4.6 GB free / no swap.

**Phase B — Cache + sessions → Redis** · low risk · best win/risk ratio.
`composer require predis/predis` (pure-PHP, PHP 8.0-safe), then `CACHE_DRIVER=redis` /
`SESSION_DRIVER=redis` on dedicated Redis DBs. Removes flock serialization. Cutover cost:
users re-login once.

**Phase C — Turn on the real queue** · medium risk.
`QUEUE_DRIVER=redis` + supervised worker (`systemd`/supervisor `queue:work`). Gotcha: the 5
existing jobs + closure dispatch start running host-less — audit each for hidden reliance on
`request()`/host/session (esp. the `->afterResponse()` ones). Staging test required.

**Phase D — Product image upload → job** · medium risk · *the actual "edits hang" fix.*
The 3 product-save call sites (`ProductRepository:102`, `ProductController:543`,
`ProductService:1005`) ignore the upload's return value → cleanly backgroundable. New
`App\Jobs\UploadProductImagesToCdn`. Two complications found:
1. Uploaded files are ephemeral — request must stage them to `storage/app/cdn-staging/{uuid}`
   and pass paths to the job.
2. `uploadImagesToCdn` converts to **WebP** and returns a `.webp` path; the separately-saved
   `ProductImage` record must agree with the final CDN filename, or images 404. Make the URL
   deterministic up front.
(The other 41 `uploadImagesToCdn` callers USE the return value — theme/content/super config,
lower frequency, out of scope for the stability fix.)

**Phase E — later:** PHP 8.0→8.3/8.4; FTP→S3-style API (FTP is slow & chatty even hoisted);
cache the product datagrid + SSR pages; `Access-Control-Max-Age` for CORS preflights
(~50% of API calls are preflights re-booting Laravel).

---

## 6. Monitoring — the standing tripwire

Any request >10s dumps a stack trace under `/var/log/php-fpm-slow/`. Health check:
```bash
find /var/log/php-fpm-slow -type f -size +0        # empty = healthy
sudo cat /var/log/php-fpm-slow/<file>              # read the top frame if not
```
| Top frame | Meaning | Action |
|---|---|---|
| `sync_db.php` | Manual DB-migration script (slow by nature) | ignore |
| `Ftp.php` / `flysystem` / `Intervention\Image` | CDN image upload (slow, bounded) | ignore (Phase D fixes) |
| `getCurrentChannel` / `FileStore` on a **`.store`** pool | recursion regressed | **investigate** |
| `getCurrentChannel` on an **info/biz** pool | expected (unpatched copies) | note only |

Useful evidence sources: `sar -u -f /var/log/sa/saNN` (per-10-min CPU, ~30 days),
`/var/log/dcpumon/2026/<Mon>/<DD>` (cPanel daily per-user CPU + top processes),
app's own `storage/app/saas-monitor/captured/*.json` (slow-request captures; note it CANNOT
capture non-terminating requests — use the FPM slowlog for those).

---

## 7. Open security / hygiene debt (unfixed)

- **CDN FTP password is hardcoded in plaintext** in `Core.php`, `OrderRepository.php`,
  `SitemapCDNUploader.php` — committed to git history (3+ commits). **Rotate it** (not just
  move to `.env`; history keeps it). Also `'ssl' => false` (plaintext FTP) and
  `Http::withoutVerifying()` in the AI importer.
- **`sync_db.php`** sits in the web root, publicly reachable, no auth, copies DB tables.
  Lock down or delete when the migration work is done.
- Superadmin container exposed on `0.0.0.0:8090` over plain HTTP.
- 301 sites on outdated/EOL PHP (WHM warning) — tied to the PHP 8.0 upgrade in Phase E.