Platform performance issues in EMEA environment

Incident Report for QikServe

Postmortem

A planned database migration to AWS on Monday morning (24 Jan 2022) resulted in degraded service for our mobile web and app customers, including a short period of complete outage.

The migration initially appeared successful, but as traffic increased later that morning, our monitoring systems detected capacity issues.

We successfully took various actions throughout the day to increase the performance of the platform. However, further capacity was still needed, and later on Monday, we determined it was necessary to carry out unscheduled maintenance and made the decision to temporarily suspend our applications for 30 minutes. This window allowed us to put in place additional resource and successfully complete the migration.

All services were fully restored on Monday and are operating as we expect.

We are very sorry for the inconvenience this caused our customers and their guests, please contact Support if you would like further details.

Warm regards,

QikServe Support

Posted Jan 31, 2022 - 17:09 GMT

Resolved

The incident has now been resolved and all services, including the Datasym and Sharp POS integrations, are now functioning normally.

Please contact us at support@qikserve.com if you're still experiencing any issues, or if you need any more help.

Posted Jan 24, 2022 - 22:52 GMT

Monitoring

The required changes have now successfully been deployed; the Platform is functioning normally, and service has been fully restored. The exception to this continues to be Sharp and Datasym POS, which we expect to be restored by 4am tomorrow.

We will continue to monitor the situation throughout the day, but we don't expect any further interruptions at this point.

If you continue experiencing issues, please contact support@qikserve.com.

Posted Jan 24, 2022 - 16:19 GMT

Identified

Unfortunately due to ongoing issues on our Platform today, our Engineering team will have to urgently deploy some changes to the Platform. During this time, the Platform will be completely unavailable.
We aim to complete these changes within the next 2 hours.

We are very sorry for the inconvenience this is causing customers and guests. Please contact support@qikserve.com if there's anything we can help with.

Posted Jan 24, 2022 - 15:00 GMT

Investigating

We are currently investigating this issue.

Posted Jan 24, 2022 - 12:48 GMT

Update

We are continuing to monitor for any further issues.

Posted Jan 24, 2022 - 12:44 GMT

Monitoring

Monitoring - All services are back up and running as expected with the exception of Sharp and Datasym POS. We will keep monitoring the situation, but please get in touch with support@qikserve.com should you continue to experience any issues.

Posted Jan 24, 2022 - 10:48 GMT

Update

We continue investigating the issue. Our sincere apologies for the inconvenience this may be causing our customers and their guests.

Posted Jan 24, 2022 - 09:44 GMT

Investigating

We are currently seeing degraded performance in the EMEA environment of our platform. We are investigating the issue.

Posted Jan 24, 2022 - 08:42 GMT

This incident affected: Common Services - EMEA platform (Store Dashboard - https://manage.withqikserve.com/, Expo Screen - https://expo.withqikserve.com, Email Services- EMEA), Guest Facing Applications - EMEA platform (Online Ordering - https://order.withqikserve.com, Mobile Applications - EMEA, Pay at Table Services - https://pay.withqikserve.com, Open Check Services - https://tab.withqikserve.com), and Integrations (Point of Sale Integrations, Payment Integrations, Other Integrations).