Hi team - It looks like we have run out of quota f...
# squadsn
Nikhil Gupta
07/08/2025, 5:36 AMHi team - It looks like we have run out of quota for monitoring issues on Sentry. I have requested additional quota but not sure who received the request. It just said "Request sent" š If any of you need to give approval, could you please do so? Thanks!
m
Max
07/08/2025, 8:18 AMI just included more budget, to solve the problem.
Max
07/08/2025, 8:19 AMBut there is an important underlying issue: why did we run out of qouta?
Max
07/08/2025, 8:19 AMCleanShot 2025-07-08 at 16.02.51@2x.png
Max
07/08/2025, 8:20 AMWe accepted >129K errors in just a couple of days in the project middleware
Max
07/08/2025, 8:22 AMIf we exclude those errors, then the graph looks like this:
Max
07/08/2025, 8:23 AMPlease investigate, understand the underling issue and report back what happened during those days. (And furthermore, why did we just learn about it after running out of credits )
Max
07/08/2025, 8:30 AMJust to put the numbers into context:
ā¢ In the last 90 days we had 129K errors.
-126K of those errors happened in 5 days!!! (Jun 26 to Jun 29)
Max
07/08/2025, 8:36 AMOur plan has currently 50,000 errors per months included. If we look at historical usage we will see than on average we don't use more than 5 or 6% of that qouta.
Max
07/08/2025, 8:37 AMIn summary: we (normally) don't have a usage or quota problem. (Lets try to find out what happened and learned from it). š
n
Nikhil Gupta
07/08/2025, 10:19 AMIt seems that we have 126K errors related to the AWS permissions or something similar. @Han Wang - Would you know what caused this?
Nikhil Gupta
07/08/2025, 10:20 AMWhatever it was, it seems that someone resolved this already. It would be good to know the root cause though.
h
Han Wang
07/08/2025, 3:00 PM@Nikhil Gupta is it possible to see the error message from sentry or somewhere else?
Han Wang
07/08/2025, 3:00 PMI think it is important to understand the root cause.
Han Wang
07/08/2025, 3:01 PMAlso I think the period aligns with permission change on the AWS side, so I wonder what exact error message we have during that time?
n
Nikhil Gupta
07/09/2025, 1:53 PMHi @Han Wang - The error message can be found in this issue in Sentry. I dont see more details than that and I don't have access to AWS to look into it further. But from the message below, it might be related to the permissions issue as you mentioned.
Copy code
ClientError
An error occurred (AccessDenied) when calling the PutObject operation: User: arn:aws:iam::033515712407:user/timenet-reader is not authorized to perform: s3:PutObject on resource: "arn:aws:s3:::timenet/api/requests/request_body/692W2P587P.json" because no identity-based policy allows the s3:PutObject actionš 1