What Happened
In the early hours of February 12th, a power supply failure caused a cascading infrastructure issue across our systems. The initial power event led to hardware damage on a critical network component, which took down internal connectivity between our servers. This meant our storage cluster couldn't communicate properly, which brought down the services that depend on it — including the API, the web app, and all processing endpoints.
Timeline (UAE Time, GMT+4)
07:56 — Power supply failure triggered infrastructure disruption
~08:30 — Network adapter failure identified; troubleshooting began
~14:00 — Filesystem corruption discovered on a secondary node (caused by the power event)
~16:30 — Filesystem repaired, cluster connectivity restored on backup link
17:40 — All services confirmed operational, storage cluster healthy
What Was Affected
All Maskr.io services were impacted during the outage window:
Background removal (web + API)
Image upscaling
All other processing endpoints
Web app access
No data was lost. All user data, credits, and account information remained intact throughout the incident. The storage cluster fully recovered with no data corruption.
Current Status
All services are back online and operating normally. We are currently running on a backup network link while replacement hardware is in transit. Full network capacity will be restored once the replacement arrives (expected February 18th).
Final Note
This one hurt. 10 hours of downtime is not acceptable, and we know that. Replacement hardware has been ordered and we're treating this as a priority.
If you experienced any issues during the outage that haven't resolved, reach out to us on X or drop a message in the community.
We appreciate your patience.
— The Maskr.io Team