Missing webhooks

Hello,

I just found out that some webhooks are not delivered to our servers for the Assista powerup and currently I’m trying to investigate what could be the reason for that. We found that several days ago (when Trello had outages).

Yesterday we synced actions in our DB with Trello for all boards accordingly.
Today I found 11 boards from our database missing webhooks and their respective action payloads which are present via Trello rest api.

The most strange thing is that we are receiving webhooks OK for those boards and we do have webhook ids stored in our DB even after the timestamp of the missing ones.

Because we do not have the missing webhooks in our logs nor we have exceptions related to receiving them I suspect that they have been send but our servers were not accessible. In that case as far as I know Trello will repeat delivery and we should get them eventually. Sadly that didn’t happen so far :confused:

I just investigated the missing webhooks for a single board and all missing webhooks are related to archive card actions done within ‘2021-09-29T20:39:35.792Z’ (+/- 1 sec).

I have a list of missing action ids and their respective boards if that is helpful.

Could you help me find out what could be the reason for this?

Thanks,
Boris

Hi @BorisFilipov

I’m not aware that Trello will re-try to post any events to your webhooks. I’m curious if new ones work for you though?

EDIT:
Oops. I guess they do retry, but only 3 times.

1 Like

Hello @jrfischer

… curious if new ones work for you though?

Yeah they do. Double-triple checked because this was the obvious reason. However it is not the case.

Here is simplified version of what we do and how we check for missing webhook payloads per board:

  1. we receive webhooks and store their respective payload Action.ids in our DB
  2. after a day we run a script which gets via Trello api all actions’ ids and compares them to those which we have in our DB.

If our servers receive everything correctly we shouldn’t find any mismatch between ids stored in our DB and the ones returned by Trello api. And we should have in our logs trace that we received these action ids. However we don’t have them neither in the DB nor in the logs.

I believe our servers failed for some reason to accept or store those webhook payloads but I don’t have any clues how to find that. If somebody from Trello side could check webhooks delivery status per action I can provide a not so small list of action ids (we have 19 boards and counting with missing tens of actions each).

I see. If you’re receiving webhook payloads for your existing webhooks but missing some from previous attempts then, to me, it seems those actions are not going to come back through the webhook calls per the docs. I doubt Trello keeps those around, but might still be worth contacting them directly?

Sometimes, what I have to do is look at the history in the UI, then retrigger the event by manually doing what was already done to the board/list/card.