Why doesn't my AWS Transfer down scoping policy work across users?

Question

Continuing How to restrict user access in AWS Transfer? ...

I want a multi-user SFTP system running out of a single bucket with directory structure of:

/my-bucket/user/user     - user can: list 
                    /in/ - user can: list get put
                    /out/ - user can: list get

I have an S3 bucket where I create the folder for the user and the in/ out/ subdirs.

My problem now is that I can only make my policies work correctly for the first user I created. The rest end up with permissions problems, unless I remove the down scoping policy, in which case they have too many privileges.

My bucket is a default S3 with no configuration. It is not public at all.

My base role policy is this:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "s3:ListAllMyBuckets",
                "s3:GetBucketLocation"
            ],
            "Resource": "*"
        },
        {
            "Effect": "Allow",
            "Action": "s3:ListBucket",
            "Resource": "arn:aws:s3:::my-bucket"
        },
        {
            "Effect": "Allow",
            "Action": "s3:*",
            "Resource": "arn:aws:s3:::my-bucket/*"
        }
    ]
}

My downs scope policy is:

    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "AllowListingOfUserFolder",
            "Action": [
                "s3:ListBucket"
            ],
            "Effect": "Allow",
            "Resource": [
                "arn:aws:s3:::${transfer:HomeBucket}"
            ],
            "Condition": {
                "StringLike": {
                    "s3:prefix": [
                        "${transfer:HomeFolder}/*",
                        "${transfer:HomeFolder}"
                    ]
                }
            }
        },
        {
            "Sid": "InDirObjectAccess",
            "Effect": "Allow",
            "Action": [
                "s3:PutObject",
                "s3:GetObject",
                "s3:DeleteObjectVersion",
                "s3:DeleteObject",
                "s3:GetObjectVersion",
                "s3:GetObjectACL",
                "s3:PutObjectACL"
            ],
            "Resource": "arn:aws:s3:::${transfer:HomeDirectory}/in/*"
        },
             {
            "Sid": "OutDirObjectAccess",
            "Effect": "Allow",
            "Action": [
                "s3:GetObject",
                "s3:DeleteObjectVersion",
                "s3:DeleteObject",
                "s3:GetObjectVersion",
                "s3:GetObjectACL"
            ],
            "Resource": "arn:aws:s3:::${transfer:HomeDirectory}/out/*"
        }
    ]
}

The logs aren't helping much with this unfortunately. They're just saying "Permission Denied."