The California Metrc API (CCTT-Metrc) experienced consistent outages for 17 days in a row, from February 16, 2021 - March 5, 2021 causing all third party integrators serious metrc-sync issues for packages to assemblies to transfers and more.
During this outage, we carefully tracked the performance of the Metrc API closely down to the minute. Here are Metrc’s response times/performance over the last month:
According to updates we received from Metrc in API Bulletin 89 and Bulletin 90, they attributed the outages to certain API endpoints (such as the sales receipt endpoint) that they were deprecating, but third party integrators did not do the migration. The migration was presented as an option to migrate, but most integrators did not make the migration.
As a result, the end point--not optimized for mass usage--ended up consuming significant memory and causing the API to crash.
Metrc first noticed the outage on February 17th and took the following measures to fix:
It’s important to note this was the longest running Metrc API outage of all time. Typically outages last a few hours at most, and we resolve this by working with our customers to manually push syncs to ensure everything gets through.
This outage, on the other hand, lasted 17 days. By day 3, we suspected it might continue on, as Metrc didn’t know the root cause of the issue. Since we track and understand the API behavior so closely, our engineering team quickly implemented several ways to alleviate failed syncs. In other words, we didn’t take the band-aid approach, and instead created scalable ways to tackle outages in the future.
02-16-2021: Manual pushes to Metrc [ongoing]
This was the first day we noticed major downtime with the Metrc API. Anytime there is a Metrc outage, our engineers spend time manually pushing stale syncs through. We have engineers in Spain and Brazil, so they’re able to get a jump on pushing info to Metrc when California is asleep and the system isn’t bogged down with requests.
02-19-2021: Auto-reconciliation of packages [new]
We've implemented auto-reconciliation of packages, which will auto-repair all inventory discrepancies that we can during this time of outage. This should help keep your Distru inventory in sync with Metrc if you need to operate in Metrc for the time being.
03-01-2021: Auto-repairing of packages [new]
Package auto-repairing was implemented. Moving forward, every 10 minutes Distru will find all package discrepancies and auto-repair them. Any package that cannot be auto-repaired will keep showing up in the Inventory Audit feed. Company admins will get an email every night with the list of packages from their company that were auto-repaired.
03-02-2021: Auto-reconciliation of assemblies and API status indicator [new]
1. We've implemented auto-reconciliation of assemblies. This will attempt to auto-match any assemblies that have been double entered in both Distru and Metrc during a Metrc outage, given that the output has the same Metrc Item, Unit, Quantity and Package Label in both systems.
2. We released a Metrc API status indicator in the Metrc control panel. When you click the Metrc shield and see the message “The Metrc API is down”, that means Distru hasn't been able to fetch packages or transfers from Metrc for your license(s) for more than 20 minutes. Here’s what it looks like:
03-05-2021: Metrc API status indicator redesign, API Outage Q&A Session and meeting with Metrc [new]
1. We redesigned the Metrc API status indicator in the Metrc control panel. Now, when you click the Metrc shield, you’ll see a Metrc Status section where you’ll be able to see the exact Metrc API sync status for your license(s). You'll see either green, orange or red status next to your license(s). Hover over the status to see what it means. Here’s what it looks like:
2. We hosted a Metrc API Outage Q&A for our customer to ask questions about the outage. We answered these questions:
3. Distru met with Metrc on a third party integrator meeting to discuss the outage. The general takeaway is that Metrc thinks the issue has been resolved.
We are continuing to monitor the Metrc API in case of any outages so that we can warn customers promptly. We are also exploring the following:
In addition, we're working closely with the CDA to meet with the heads of the CDFA to discuss further matters on this. Part of this includes:
We suggest contacting the BCC directly. If they hear enough complaints, it could help. Here's what to do:
Fill out this BCC "Notification and Request Form" with the following:
Email the form to the BCC (may as well copy CDFA and Metrc so there's documentation across the board):
Organize your deliveries, optimize your route plan, and log returns
Standardize your SOP across multiple warehouses or locations