Mounting remote directory directly in docker using sshfs plugin as non-root user

Question

I'm encountering issues while trying to mount a remote directory directly into a Docker container using the vieux/sshfs plugin. This issue does not occur when I run the container as root user without user: '1000:1000'. The ownership of the mounted folder is 1000:1000 with permissions set to 755. Here's my docker compose:

services:
  busybox:
      image: busybox
      container_name: busybox
      user: '1000:1000'
      stdin_open: true
      tty: true
      volumes:
        - config:/config

volumes:
  config:
    driver: vieux/sshfs
    driver_opts:
      sshcmd: "[email protected]:/path"
      password: "password"
      uid: "1000" #Tried without this
      gid: "1000" #Tried without this

When I try to ls from root directory I get ./config:permission denied. I have installed the plugin with --grant-all-permissions and also tried allow_other in driver_opts.

I am able to mount it to docker by first mounting it directly to my host directory and then mounting that to my docker, but I wanted to know if the former was at all possible.