Health Card Validation (HCV)/Demographic Search Issues

Incident Report for CabMD

Postmortem

We want to extend our sincerest apologies for the recent issues you may have experienced with Health Card Validation (HCV) on CabMD since April 1st. We understand how vital this service is to your operations, and we deeply regret any inconvenience these disruptions may have caused.

What Happened?

Recently, our HCV service implementation has faced significant challenges, primarily due to factors beyond our immediate control. Firstly, we believe the shutdown of the MOHLTC’s IMS service at the end of March led to an increased volume of HCV requests on the MOHLTC’s platform, putting unprecedented stress on our systems as service degraded. Additionally, the transition from GO Secure to OPS BPS Secure (Ontario Public Service Broader Public Service Secure) by the Ministry of Health (MOH) and Long-Term Care (MOHLTC) involved an unexpected policy of expiring passwords close to the migration date. Unfortunately, this change was not communicated effectively, leading to widespread password resets and a spike in failed communications with the MOH. These factors combined have highlighted several areas in our connection processes with the MOH which needed to be corrected.

Our Response

Upon identifying the problem, which unfortunately took longer than we had hoped, our team took actions to mitigate the impact on your experience as we hope you have noticed in the last week:

Increased Resilience: We’ve enhanced our software to better handle interruptions and ensure smoother communication with MOH services, improving stability even during periods of increased demand.
Enhanced Logging: We are now capturing detailed logs of all transactions related to HCV. These logs are crucial as they provide us and the MOH with clear insights into the issues, aiding faster resolution. We’ll be adding this information to the CabMD status page so that it’s visible, and to aid in automating communication when services related to HCV drop below expected performance.
Industry Collaboration: We have reached out to and begun working closely with other companies affected by similar issues. This collaboration aims to advocate for quicker resolutions and share best practices for handling such disruptions.
Continuous Optimization: We have implemented several optimizations to improve the speed and reliability of the HCV service. More enhancements are planned and will be rolled out as part of our ongoing commitment to excellence.

Looking Forward

Our commitment to resolving these issues is unwavering. We have been and will continue to be in communication with affected parties including the MOH to ensure a comprehensive and long-lasting resolution. Additionally, we will continue to update our systems and processes to prevent such occurrences in the future whenever possible.

We appreciate your patience and understanding as we work through this challenge. If you have any concerns or need further assistance, please do not hesitate to contact our support team.

Thank you for your continued trust in CabMD.

Posted Apr 30, 2024 - 11:37 EDT

Resolved

This incident has been resolved.

Posted Apr 30, 2024 - 11:11 EDT

Monitoring

A fix has been implemented and we are monitoring the results.

Posted Apr 25, 2024 - 13:15 EDT

Update

The technical issues that CabMD has been experiencing related to Health Card Validation (HCV) appear to correlate with the recent timeline for the shutdown of the IMS services from the Ministry of Health and Long-Term Care (MOHLTC). We understand the inconvenience this may cause and want to assure you that we are actively working with the MOHLTC to identify and address these challenges.

The IMS (Information Management System) services, provided by the MOHLTC, were essential technological components used by healthcare systems for HCV. These services facilitated real-time access and validation of health card data, ensuring that healthcare services were promptly and correctly administered to patients. The shutdown of these IMS services and subsequent increased usage of the Electronic Business Systems (EBS) HCV systems may have disrupted these processes, potentially leading to delays in health service delivery and significant challenges in patient data verification. This transition seems to be heavily affecting the efficiency and reliability of all users reliant on these systems including CabMD.

We are constantly communicating with MOHLTC representatives and await further details from them. Our team is committed to resolving these issues swiftly and will keep you updated as new information becomes available.

Thank you for your patience and understanding as we navigate this difficult situation.

Related bulletin regarding the shutdown of IMS services: https://www.ontario.ca/document/ohip-infobulletins-2024/bulletin-240305-final-reminder-end-ims-connectlistener-health-card

Posted Apr 18, 2024 - 12:55 EDT

Investigating

We want to update you on the ongoing issues some of you may be experiencing with our Health Card Validation (HCV) feature. We have identified several areas for improvement, have deployed several of these changes over the last two weeks, and are actively working to enhance the reliability of this service.

Currently, we are encountering intermittent slow responses from the Ministry of Health and Long-Term Care (MOHLTC) HCV web service. In some instances, even when the response times are normal, the demographic information necessary for filling out medical billing claims is missing.

Please rest assured that we are in ongoing communication with the Ministry to address these issues as effectively as possible. We are committed to providing them with all necessary information to help pinpoint and resolve the problems.

We understand the importance of this feature for your day-to-day operations and appreciate your patience as we work through these challenges. We will continue to keep you informed of our progress and are here to assist you with any questions or concerns.

Thank you for your understanding and cooperation.

Posted Apr 18, 2024 - 09:24 EDT

This incident affected: Website Backend (https://my.cab.md) and MOHLTC's HCV Service.