How to detect colored pixel of Sprite3D texture with perspective camera ray?

Question

I've tried to ray cast my mouse position to Sprite3D on screen but I have some problems with it.

func _input(event):
    if event is InputEventMouse:
        mouse = event.position
    if event is InputEventMouseButton and event.pressed:
        if event.button_index == MOUSE_BUTTON_LEFT:
            get_selection()

func get_selection():
    var worldspace = player.camera_3d.get_world_3d().direct_space_state
    var start = player.camera_3d.project_ray_origin(mouse)
    var end = player.camera_3d.project_position(mouse, 1000)
    
    var query = PhysicsRayQueryParameters3D.create(start, end)

    var result = worldspace.intersect_ray(query)

Brief setting of my project My character/enemy is a Sprite3D in CharacterBody3D with small collision box only on its feet. I want the mouse to detect the shape of colored pixels of my character/enemy I saw on screen but intersect_ray() always return with collider in the object, so it went through the character to the floor object instead.

Why I don't set the collision box to match the height of sprite? Camera is attached to player with 45 degree angle looking down to the player and it can zoom-in-out and tilt then it will look_at() player again at its new position. While tilting, character's visual is also tilted to parallel with camera. So I think its collision box couldn't be the same size as sprite because sprite's height will be changing all the time and collision box cannot be tilted into the ground too.

visuals.rotation.x = camera_3d.rotation.x

So my question is... Is there a way to do a manual raycasting? or to calculate colored pixels' positions from characters/enemy on screen compare to mouse position and know if it's player or enemy. Please keep in mind that camera can zoom-in or zoom-out and tilting.

Thank you in advance

Answer 1

Reading the pixels would involve getting the texture from the viewport and querying that. Which is doable, if somewhat annoying. It is similar to: How would I mask the player node out of a canvas shader?.

---

Instead I'm going to suggest to dedicate a physics layer for things you want to be able to pick up like this.

You can specify to your PhysicsRayQueryParameters3D what layers it should detect by setting its collision_mask, so it only picks objects that have the layer you want.

So you can have a second physics body (an StaticBody3D would do) which is in that dedicated layer and has a collider that matches the Sprite3D much closer.

That is, you can keep your current physics object with its collider which would have layer and mask that interact with the game world (this one could have a wide base, for example), and a second physics object that only has a layer that allows you to pick it up but won't interact with anything in the game world (so it can be thin and tall to better match the Sprite3D).

You could rotate this second physics body to look_at the get_viewport().get_camera_3d() if you need to mimic billboarding. And you can also update its collision shape in runtime if needed.

---

By the way, your physic objects have a property input_ray_pickable which enables _input_event, if that works for you, it might save you some coding (you might not need intersect_ray). If it doesn't, having it enabled is a waste, so consider turning it off.