Stripo Plugin is down

Incident Report for Stripo

Postmortem

SUMMARY

On 8 October 2026, between 22:20 and 23:18 UTC, the Stripo Plugin was unavailable: the editor did not load or respond. The outage lasted 58 minutes. No customer data was lost, and other Stripo products were not affected.

WHAT HAPPENED

The database server behind the Plugin lost time synchronization. Two time-sync services were running on the server at once: the standard daemon, which adjusts the clock gradually, and a legacy scheduled job that steps the clock abruptly. A misconfigured schedule made the legacy job run almost every minute. The repeated clock jumps destabilized the server until the Plugin's database became unavailable.

TIMELINE (UTC, 8 October 2026)

21:44 - Clock instability begins on the Plugin database server.
22:13 - Our monitoring raises the first alert for the server.
22:20 - The Plugin stops loading for customers.
22:40 - Monitoring reports the Plugin database as unavailable.
23:00 - First customer reports are received; the on-duty engineer is called in.
23:18 - Service restored. The Plugin is fully operational.

RESOLUTION

Our engineers identified the time-sync conflict and restored the server. The Plugin returned to normal operation at 23:18 UTC and has been under extended monitoring since.

WHAT WE ARE DOING TO PREVENT THIS

  • Removing the legacy time-sync job so that this server, and every other production server, uses a single time source.
  • Adding monitoring for clock drift so that this kind of fault is caught before it affects service.
  • Changing alert escalation. Our monitoring alerted seven minutes before the outage, but the alert did not page an engineer. Critical alerts on production database servers will now page the on-duty engineer immediately, at any hour.

We apologize for the disruption and thank you for your patience. If you were affected and have questions, please contact our support team.

Posted Oct 09, 2026 - 09:15 UTC

Resolved

The Plugin is fully operational again. DevOps restored the Plugin DB host; we are monitoring and have seen no further errors since 23:18 UTC.

A postmortem with root cause, timeline, and action items is being prepared and will be published shortly.
Posted Oct 08, 2026 - 23:18 UTC

Investigating

Started: 2026-10-08 22:20 UTC

Impact: Plugin is unavailable for customers — the editor does not load / does not respond. We have received reports from several customers.

What happened: At 22:13 UTC Zabbix reported the agent on the Plugin DB host (pg-plugins.stripo.email, 172.31.13.64) as unavailable; at 22:40 UTC the PostgreSQL monitor for live-plugins failed as well. The Plugin stopped working for customers at ~22:20 UTC.

DevOps is investigating. Next update here as soon as we have more information.
Posted Oct 08, 2026 - 22:20 UTC
This incident affected: Stripo Plugin (API Gateway Service (plugin)).