Issues with Cross-Namespace Service Discovery in Kubernetes with Spring cloud kubernetes

Question

I'm running multiple Spring Boot applications in two different Kubernetes namespaces. Here’s my configuration for all services:

SPRING_CLOUD_KUBERNETES_DISCOVERY_NAMESPACES_0: welcoming
SPRING_CLOUD_KUBERNETES_DISCOVERY_NAMESPACES_1: default
SPRING_CLOUD_KUBERNETES_ENABLED: true
SPRING_CLOUD_KUBERNETES_LOADBALANCER_MODE: SERVICE
SPRING_CLOUD_KUBERNETES_RELOAD_ENABLED: true
SPRING_CLOUD_LOADBALANCER_CACHE_ENABLED: true

When a request is made to retail-onboarding in the welcoming namespace, it uses RestTemplate to call audit-service in the default namespace. However, it returns an error stating that the service ID cannot be found, even though the Kubernetes discovery client lists the service:

2024-07-08T08:51:59.290Z DEBUG [retail-onboarding,,] 1 --- [nio-8080-exec-3] .c.k.f.d.Fabric8DiscoveryServicesAdapter : searching in namespaces : [welcoming, default] with filter : null 

2024-07-08T08:51:59.336Z DEBUG [retail-onboarding,,] 1 --- [io-8080-exec-10] o.s.c.k.f.d.KubernetesDiscoveryClient    : will return services : [retail-onboarding, audit-service, ingress-nginx-controller, ingress-nginx-controller-admission, kubernetes,]

2024-07-08T08:52:22.099Z ERROR [retail-onboarding,,] 1 --- [nio-8080-exec-9] c.b.f.i.a.RunWithoutAuthorizationAspect  : Exception thrown when invoking join point

com.backbase.audit.client.exceptions.InternalServerErrorException: java.lang.IllegalArgumentException: Service Instance cannot be null, serviceId: audit-service                                                                                                                         Caused by: java.lang.IllegalArgumentException: Service Instance cannot be null, serviceId: audit-service

This setup works correctly when I switch to using SPRING_CLOUD_KUBERNETES_DISCOVERY_ALL_NAMESPACES: true

I have also configured the following ClusterRole and ClusterRoleBindings for cross-namespace service discovery:

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: cross-namespace-service-discovery
rules:
- apiGroups: [""]
  resources: ["pods", "services", "endpoints"]
  verbs: ["get", "list", "watch"]

---

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: cross-namespace-service-discovery-binding
subjects:
- kind: ServiceAccount
  name: default
  namespace: welcoming
roleRef:
  kind: ClusterRole
  name: cross-namespace-service-discovery
  apiGroup: rbac.authorization.k8s.io

---

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: cross-namespace-service-discovery-binding
subjects:
- kind: ServiceAccount
  name: default
  namespace: default
roleRef:
  kind: ClusterRole
  name: cross-namespace-service-discovery
  apiGroup: rbac.authorization.k8s.io

What could be causing the service discovery to fail with the specific namespaces setup? How can I ensure that services in welcoming can reliably discover and call services in default namespace?

Any insights or suggestions would be greatly appreciated!