From vulnerability discovery to solution deployment in 2 weeks.
It’s never fun to find out your software has a security flaw, but if you’re in the software business and your products are installed on millions of devices, it’s likely you haven’t found them yet.
This case study tells the story of a team effort that proves that clients and solution providers, big tech competitors and device manufacturers can work together in harmony to protect the end-user.
In early October 2021, MCE Systems received an email report from the Microsoft Security Vulnerability Research (MSVR) team detailing possible security vulnerabilities in our SDK. Within minutes a response team headed by our CTO, CIO, CISO, Chief Software Architect, and VP of Delivery was mobilized to understand the implications of the report. The good news is that, while there were potential vulnerabilities, our solution is incredibly robust, and, with enhanced data protection, exploitation of these flaws would be incredibly difficult.
The report’s overall findings were that there are no signs of active exploits.
An investigation followed and mapped which of our clients retained the potential to be affected, and to what extent. It was great to reconfirm that the automated processes, procedures, and accreditation’s (ISO/IEC 27701, 27001, 27017) MCE has in place not only comply with the privacy and security standards required for Tier-1 Enterprise SaaS delivery, but also allowed us to assuredly respond to queries about the level of risk and possible impact on our customers.
Understanding the potential level of risk
As an omnichannel, hybrid solution, delivered as a platform, product, or SDK – and available on multiple operating systems, including iOS, iPadOS, MacOS, Android, Windows and Web – identifying the code responsible and altering it to patch the vulnerability was not enough. In fact, the fixes themselves were fairly simple and straightforward, being completed within 24 hours. Moreover, we knew that our biggest challenge was to make sure every end-user who uses the app is updated to the new patched version (and that the update would not degrade their experience in any way).
Implementing the patch
We started by applying a configuration patch to our cloud services, which are responsible for all of our solution endpoints. The solution architecture allows for remote customization and configuration that are controlled from our cloud core services, which allowed us to resolve some of the vulnerabilities simply, and with the flick of a switch (probably, that’s an oversimplification of the process for a system that serves millions of users globally, but it did feel like it).
Collaboration with Microsoft
A quick online meeting with the team at Microsoft that reported on the vulnerabilities has been set up to learn even more about the findings and how they were found (to assure that no exploits have been detected). We were happy to find out that the Microsoft team chose not only to thoroughly explain the vulnerabilities they found, but also offered to assist us to test that these had been completely eliminated. They also offered to assist us in supporting our operator customers to push these patch updates out – as they correctly identified deployment would be one of the major challenges.
Eradication of vulnerabilities
We were pleased to learn that the Microsoft team not only took the time to thoroughly explain and record the vulnerabilities they discovered, but also volunteered to help us test that we had entirely eradicated them. They also promised to help us communicate to our customers the importance of delivering these upgrades in an efficient and timely manner.
Synergy of Operators, OEMs, and MCE Systems
Other vulnerabilities were addressed using proprietary software that was distributed to the millions of devices that our apps and services are installed upon. Some were downloaded by end-users, while others were pre-installed by the OEMs.
We informed our customers about the results and emphasized that, while we don’t believe this is a major issue, they should move quickly and be prepared to assist us in releasing the app update to their users.
The challenges of updating an app
Let’s take a moment to discuss how difficult it is to ensure that these apps are updated fast and why.
Our customers are mostly Tier 1 carriers, which requires us to either fully integrate our solutions into their applications and services, or white-lablel our solutions so that their customers have a consistent experience across all services. MCE has minimal direct influence over app store updates (Apple App Store and Android’s Google Play Store) because our clients submit the apps to these online marketplaces. This necessitates us going through the carriers’ change management processes, which are designed to be foolproof and always slower than they (and we) desire.
Millions of people have our apps or apps enabled by our SDK and platform preloaded on their mobile devices. It is a necessity to mandate a Maintenance Release (MR) from the device OEM in order to update these. You can understand that OEMs have their own protocols that make this a difficult task, particularly when time is of the essence.
On-going collaboration
Microsoft worked alongside MCE and its customers to not only to explain and deliver the fixes but also to monitor them and provide support to Operators where necessary. The level of commitment and urgency was above and beyond what we expected.
When we identified a slower conversion rate into the latest version, we reached out to customers directly using in-app messaging and notifications, urging them to update their app. We have allowed enough time for this and after a while have blocked access to older versions of the app.
In conclusion
“Ultimately, we were able to safeguard our clients’ customers in a matter of weeks, without a single exploit of the vulnerabilities. Thank you to the Microsoft MSVR team for quickly discovering and mitigating the vulnerabilities. Thank you to our customers (the Telcos) for acting quickly and prioritizing the end-user. Thank you, Google for assisting us in ensuring that Android users have a double safety net. Thank you to our security, architects, developers, and customer success managers for quickly mobilizing – designing, implementing, and monitoring”.
– Liran Weiss, MCE’s CCO.
