This runbook gives an incoming owner a safe first response. It does not authorize production data mutation. Use reviewed, reversible procedures for repairs.
- Record the time, reporter, affected role, branch, order/job reference and exact action.
- Determine whether the frontend, backend, database or an external integration is failing.
- Check
/api/v1/health and /api/v1/version.
- Record the frontend version shown by the application.
- Check the latest Vercel and DigitalOcean releases independently.
- Check backend logs around the exact request time using a correlation identifier if one exists.
- Do not retry a state-changing action until persisted effects are known.
| Severity |
Example |
Response |
| Critical |
Dispatch unavailable, widespread login failure, data loss, incorrect accounting output, privacy exposure |
Stop releases, name an incident owner, preserve evidence and communicate immediately |
| High |
One branch cannot operate, payments/refunds cannot be verified, integration blocks a core flow |
Triage immediately and establish a manual fallback |
| Medium |
Degraded report, isolated user/record problem, non-critical integration failure |
Scope affected records and schedule a controlled fix |
| Low |
Cosmetic defect or documentation mismatch |
Record and prioritize normally |
Check in order:
- Is shared UtopiaSpace authentication working?
- Does the request carry a valid token with the expected issuer and expiry?
- Does the person have an active local
app_users record?
- Does the local role grant the required module/level?
- Is the requested branch within the person's scope?
- Is the cookie domain/SameSite/secure policy correct for the environment?
- If silent SSO fails, does direct OAuth still work?
Do not solve an access incident by loosening middleware or granting a broad role without identifying the failed layer.
Dispatch is the highest-risk recovery path.
- Inspect the order status and audit trail.
- Check whether an order number was minted.
- Check whether stock was issued.
- Check whether company ownership was frozen.
- Check whether a job log or notification was created.
- If any effects exist, do not click dispatch again. Reconcile the partial result first.
- If no effects exist and the order remains a draft, inspect validation and permission failures before retrying.
A 500 after commit can still mean the core transaction succeeded and a synchronous best-effort integration failed afterward.
- Record order, job log, line reference and snapshot IDs.
- Separate the Indoor baseline from each job's submitted lines.
- Compare current order items with snapshot lines and audit history.
- Check whether the problem affects imported pre-cut-off data or live post-cut-off work.
- Do not reconstruct history from descriptions alone when stable line references exist.
- Preserve signed quantities and amounts, including zero and negative lines.
- Confirm whether the row is pending, verified, issue-found, cancelled or posted.
- Check source payment, proof, amount, branch/payee, verification actor/time and accounting reference.
- Do not edit a verified or posted fact through an uncontrolled database update.
- A failed response does not prove that the payment write failed; inspect the row before resubmitting.
- Record consumer, token scope, query filters and requested time range.
- Confirm whether imported rows are intentionally included.
- Preserve negative/contra lines and marker rows according to the feed contract.
- Compare order date and payment date filters; the finance payment feed can include a row when either falls within its defined year boundary.
- Check signed proof URL expiry separately from record existence.
- Confirm whether a row was already posted and under which document reference.
- Check whether the Hub Edge Function is deployed and reachable.
- Verify both sides have the same vehicle-listing secret without exposing it.
- A
503 not_configured, 401 unauthorized or 502 query_failed has a different corrective owner.
- Do not accept a partial fleet as success; missing rows can be interpreted as deactivated vehicles.
- Run
fleet:sync only after the upstream contract and target environment are verified.
- Confirm that
lg:generate-invoices exists in the deployed backend version.
- Confirm a production scheduler or external trigger actually ran on the 20th at 02:00.
- Review logs and existing invoice rows before manual execution.
- The command is intended to be idempotent, but capture pre/post counts and affected partner orders.
Every repair plan must include:
- exact environment;
- affected IDs and record count;
- pre-cut-off versus live-data split;
- query used to identify records;
- backup or captured pre-state;
- transaction and rollback method;
- dry-run output;
- reviewer and executor;
- post-repair checks and audit entry.
The repository exposes health/version routes and Laravel logging configuration, but the inspected documentation does not identify a complete production dashboard, alert policy, retention period, on-call rota or correlation-ID standard. These remain blocking handover items until owners and evidence are recorded.