7

I have an API behind AWS Cloudfront which has functioned fine with our front end application for years. Recently, after a feature release, I've noticed some users are reporting data not being saved and the app just hanging. After a lengthy investigation, I've discovered that our Cloudfront distribution will return a 403 Forbidden error when a PUT request's JSON body is greater than 8kb. Anything less works fine, anything more returns 403. I verified this by sending PUT requests with a decreasing body data size until I got the expected 201 Created response, and just checked the size of the body sent. The JSON is properly formatted.

The error returns in about 170ms and contains the header from Cloudfront X-Cache: Error from cloudfront

I have looked for settings on size limits, I've tried disabling the WAF rules, I've tried "Compress objects" to both on/off. Would having Real-time logs enabled have an impact on the max data accepted somehow? Seems crazy but I'm kind of bewildered by this issue.

I would love to show you all some kind of log from Cloudwatch to help, but requests that don't make it past Cloudfront are not logged in Cloudwatch, and I have not been able to setup any kind of logging to get better insight into why it's barfing.

Improve this question
9
  • stackoverflow.com/questions/63761063/…
    – Devang Sanghani
    Commented Feb 3, 2022 at 7:55
  • Also, another pointer would be check what went out in the latest release.
    – Devang Sanghani
    Commented Feb 3, 2022 at 7:57
  • Sorry, that is not related to this problem. That is multi origin, I'm referring to request body size
    – sean.hudson
    Commented Feb 3, 2022 at 8:01
  • 1
    I did find a SizeRestrictions_BODY rule in the AWS managed rule set for the WAF which "Verifies that the request body size is at most 8 KB (8,192 bytes)" I've overridden that rule to "count" but the issue persists. I'll look for other places this might show
    – sean.hudson
    Commented Feb 3, 2022 at 8:02
  • It really seems to be an 8kB limit, but WAF is not even showing any DENY actions in its logs. Does anyone know where else a Cloudfront dist might have an 8kb body length limitation?
    – sean.hudson
    Commented Feb 3, 2022 at 8:18

1 Answer 1

Reset to default
5

As was the initial hunch, this turned out to be a WAF ACL rule issue.

The blocking ACL was applied to the application load balancer, so finding it in the Web ACL list either requires inspecting the region where your load balancer is (eg us-west-2), or by inspecting the load balancer's Integrate Services, where you can see any AWS WAF rules:

  • AWS > EC2 > Load Balancers > {instance} > Integrated services (tab) > AWS WAF

The specific rule was in an AWS managed rule set called AWS-AWSManagedRulesCommonRuleSet. Just turn the SizeRestrictions_BODY rule to Count instead of "Use action defined in the rule"

This obviously has impacts on what requests get through to your application, so do with that what you will.

Improve this answer
1
  • Been trying to figure out why for a few hours now, I came to think it was Cloudflare, but it ended up being this rule. Thanks
    – MrEduar
    Commented 11 hours ago

Not the answer you're looking for? Browse other questions tagged or ask your own question.