Metrc API Outage Post-Mortem

Summary of what happened:

The California Metrc API (CCTT-Metrc) experienced consistent outages for 17 days in a row, from February 16, 2021 - March 5, 2021 causing all third party integrators serious metrc-sync issues for packages to assemblies to transfers and more.

During this outage, we carefully tracked the performance of the Metrc API closely down to the minute. Here are Metrc’s response times/performance over the last month:

‍

‍

Why Did This Happen and What Did Metrc Do About It?

According to updates we received from Metrc in API Bulletin 89 and Bulletin 90, they attributed the outages to certain API endpoints (such as the sales receipt endpoint) that they were deprecating, but third party integrators did not do the migration. The migration was presented as an option to migrate, but most integrators did not make the migration.

As a result, the end point--not optimized for mass usage--ended up consuming significant memory and causing the API to crash.

Metrc first noticed the outage on February 17th and took the following measures to fix:

1. Metrc attributed "the volume and pace" of API calls in the state of California as a potential cause of the outage. They upgraded their servers in hopes that it would lessen the effects of the outage (however, it seemed that the core issue has yet to have been addressed).
2. Metrc released Phase 3 of Rate Limiting on March 5th in an attempt to help out with the outage. This limits the amount of calls each integrator can make to Metrc per second in an effort to even the playing field.

How Did Distru Respond?

It’s important to note this was the longest running Metrc API outage of all time. Typically outages last a few hours at most, and we resolve this by working with our customers to manually push syncs to ensure everything gets through.

This outage, on the other hand, lasted 17 days. By day 3, we suspected it might continue on, as Metrc didn’t know the root cause of the issue. Since we track and understand the API behavior so closely, our engineering team quickly implemented several ways to alleviate failed syncs. In other words, we didn’t take the band-aid approach, and instead created scalable ways to tackle outages in the future.

Here’s a Timeline of What We Implemented:

02-16-2021: Manual pushes to Metrc [ongoing]

This was the first day we noticed major downtime with the Metrc API. Anytime there is a Metrc outage, our engineers spend time manually pushing stale syncs through. We have engineers in Spain and Brazil, so they’re able to get a jump on pushing info to Metrc when California is asleep and the system isn’t bogged down with requests.

02-19-2021: Auto-reconciliation of packages [new]

We've implemented auto-reconciliation of packages, which will auto-repair all inventory discrepancies that we can during this time of outage. This should help keep your Distru inventory in sync with Metrc if you need to operate in Metrc for the time being.

03-01-2021: Auto-repairing of packages [new]

Package auto-repairing was implemented. Moving forward, every 10 minutes Distru will find all package discrepancies and auto-repair them. Any package that cannot be auto-repaired will keep showing up in the Inventory Audit feed. Company admins will get an email every night with the list of packages from their company that were auto-repaired.

03-02-2021: Auto-reconciliation of assemblies and API status indicator [new]

1. We've implemented auto-reconciliation of assemblies. This will attempt to auto-match any assemblies that have been double entered in both Distru and Metrc during a Metrc outage, given that the output has the same Metrc Item, Unit, Quantity and Package Label in both systems.

2. We released a Metrc API status indicator in the Metrc control panel. When you click the Metrc shield and see the message “The Metrc API is down”, that means Distru hasn't been able to fetch packages or transfers from Metrc for your license(s) for more than 20 minutes. Here’s what it looks like:

03-05-2021: Metrc API status indicator redesign, API Outage Q&A Session and meeting with Metrc [new]

1. We redesigned the Metrc API status indicator in the Metrc control panel. Now, when you click the Metrc shield, you’ll see a Metrc Status section where you’ll be able to see the exact Metrc API sync status for your license(s). You'll see either green, orange or red status next to your license(s). Hover over the status to see what it means. Here’s what it looks like:

2. We hosted a Metrc API Outage Q&A for our customer to ask questions about the outage. We answered these questions:

• Why does the API work sometimes and other times it doesn’t?
• What does Distru do if my sync doesn’t go through?
• How does Distru monitor outages?
• How do I know if the API outage is affecting me?
• How do I remove an assembly that’s stuck?
• What does rate limiting mean and who does that affect?

‍

3. Distru met with Metrc on a third party integrator meeting to discuss the outage.  The general takeaway is that Metrc thinks the issue has been resolved.

‍

How Can We Prevent This in The Future?

We are continuing to monitor the Metrc API in case of any outages so that we can warn customers promptly. We are also exploring the following:

1. We’re experimenting with a feature that would allow you to create assemblies in Distru that push to Metrc even during an outage
2. We’re working on a way for syncing packages to be added to Sales Orders. This will enable you to print out invoices with package information even during a Metrc outage.

In addition, we're working closely with the CDA to meet with the heads of the CDFA to discuss further matters on this. Part of this includes:

1. Rewording the state contract to ensure that API downtimes are properly worded, right now they aren't and are not even explicitly state in the contract
2. We're a part of a third-party integrator group that meets with Metrc monthly to bring the concerns of us (and the operators) we serve to Metrc directly

‍

Here’s how You Can Help:

We suggest contacting the BCC directly. If they hear enough complaints, it could help. Here's what to do:

Fill out this BCC "Notification and Request Form" with the following:

1. License name, number and expiration date.
2. Under Section B. REQUIRED NOTIFICATIONS, check the box next to “Connectivity to Track and Trace is Lost - 5050(b)”
3. On page 2 you have the option of adding a note where it says, “Use space below for additional information, as needed.”
4. Add a message with more info. Feel free to use this: Metrc’s API is not functioning yet again. The track and trace system’s API is fundamentally broken, in violation of California Business and Professions Code SEC. 51. Section 26068 which mandates a functional API for licensees to report commercial cannabis activity, as required by regulations. Our industry’s ability to generate tax revenues for the state is severely disrupted when the track and trace system mandated for licensees does not work as required by law.

Email the form to the BCC (may as well copy CDFA and Metrc so there's documentation across the board):

• BCCTrackandTrace@dca.ca.gov
• BCC@dca.ca.gov
• calcannabis@cdfa.ca.gov
• api-info@metrc.com

‍

‍