ROOT CAUSE ANALYSIS: An automatic forced upgrade of an AWS OpenSearch instance (1/18/2022) caused restarts of a linked cluster in AWS Fargate and analytics reporting with AWS Athena. The restart of Athena encountered an issue with a new version released 1/17/2022.
“At Mon, 17 Jan 2022 22:21:51 GMT, Amazon Athena upgraded your workgroup [primary] to use Athena engine version 2 in your AWS account.”
REMEDIATION: The Fargate cluster was restarted to adjust logging and the Athena analytics tables were rebuilt to point to the new location.
Posted Jan 20, 2022 - 11:46 CST
Resolved
Licensing (on AWS Fargate) also experienced a forced upgrade, and the did not come back up cleanly with logging. The service has been restarted and services are operating as normal.
Posted Jan 19, 2022 - 16:37 CST
Update
Engineers are still researching the latency following the ElasticSearch issue. This channel will continue to house updates as they are available.
Posted Jan 19, 2022 - 15:06 CST
Identified
Intermittent login and search latency have continued to be seen. The DevOps team is refreshing network resources to address.
Posted Jan 19, 2022 - 10:24 CST
Investigating
We have reports of continued slow login and search experience after yesterdays restart. The DevOps team is investigating.
Posted Jan 19, 2022 - 09:24 CST
This incident affected: Desktop Client and Web Portal.