tag | 88fb2dbc14838e45a3c61cd7de7edb73632acf22 | |
---|---|---|
tagger | Peter Hutterer <peter.hutterer@who-t.net> | Fri Dec 09 03:27:33 2022 |
object | 39f3ec007b3405198491e0932d599122f4e1ac3e |
0.4
commit | 39f3ec007b3405198491e0932d599122f4e1ac3e | [log] [tgz] |
---|---|---|
author | Olivier Fourdan <ofourdan@redhat.com> | Tue Dec 06 13:26:57 2022 |
committer | Peter Hutterer <peter.hutterer@who-t.net> | Thu Dec 08 02:37:46 2022 |
tree | 28843668e4e15f1b17e9355277d7e76037469c91 | |
parent | 8bcbc53c0f621b3d7791824c6cfb5179e424faa5 [diff] |
liboeffis: protect from interrupted system calls In various places, including the DBUS calls which can take some time, if a SIGALRM triggers, the call will fail. To prevent this from happening, use the wrappers when possible and make sure to block the SIGALRM signal when issuing DBUS calls. Signed-off-by: Olivier Fourdan <ofourdan@redhat.com>
libei is a library for Emulated Input, primarily aimed at the Wayland stack. It provides four parts:
libei
)libeis
)libreis
)liboeffis
)The communication between these parts is an implementation detail, neither client nor server need to care about the details. Let's call it the BRidge for EI, or 🥣 brei. In the future, 🥣 brei may become a stable protocol. For now, this C library is it.
For the purpose of this document, libei refers to the project, libei
/libeis
/libreis
to the libraries provided.
The API documentation is available here
In the Wayland stack, the EIS server component is part of the compositor, the EI client component is part of the Wayland client.
+--------------------+ +------------------+ | Wayland compositor |---wayland---| Wayland client B | +--------------------+\ +------------------+ | libinput | libeis | \_wayland______ +----------+---------+ \ | | +-------+------------------+ /dev/input/ +---brei----| libei | Wayland client A | +-------+------------------+
The use-cases libei attempts to solve are:
xdotool
or more generically the XTEST extensionsynergy
, for both client-emulated input as well as the forwarding of physical or logical devices.libei provides three benefits:
libei provides separation of emulated input from normal input. Emulated input is a distinct channel for the compositor and can thus be handled accordingly. For example, the compositor may show a warning sign in the task bar while emulated input is active.
The second benefit is distinction. Each libei client has its own input device set, the server is always aware of which client is requesting input at any time. It is possible for the server to treat input from different emulated input devices differently.
The server is in control of emulated input - it can filter input or discard at will. For example, if the current focus window is a password prompt, the server can simply discard any emulated input. If the screen is locked, the server can pause all emulated input devices.
As of version 0.3, libei allows a libei
context to be either a sender or a receiver. In the “sender” mode, the libei
context gets a list of devices from the EIS implementation and can emulate events on these devices. The devices usually represent virtual devices, e.g. a generic relative pointer corresponding to the cursor or per-screen absolute input devices.
In the “receiver” mode, the libei
context gets a list of devices from the EIS implementation and receives events from those. This allows for input capture, provided the EIS implementation supports it. The devices can correspond to virtual devices, e.g. a generic relative pointer corresponding to the cursor. Or they may be representations of physical devices, e.g. a tablet device getting forwarded.
A libei
context may only be in one mode. It is up to the EIS implementation to accept a sender or receiver libei
context.
We start from the baseline of: “there is no emulated input in Wayland (the protocol)”.
There is emulated input in X through XTEST but it provides neither separation, distinction nor control in a useful manner. There are however many X clients that require XTEST to work.
There are several suggestions that overlap with libei, with the main proposals being:
Emulated input is not specifically Wayland-y. Clients that emulate input generally don‘t care about Wayland itself. It’s not needed to emulate events on their own surfaces and Wayland does not provide global state. The only connection to Wayland is merely that input events are received through the Wayland protocol. So a Wayland protocol for emulating input is not a great fit, it merely ticks the convenient box of “we already have IPC through the wayland protocol, why not just do it there”.
DBus is the most prevalent generic IPC channel on the Linux desktop but it's not available in some compositors. Any other specific side-channel requires an IPC mechanism to be implemented in the sender and receiver.
The current situation looks like that neither proposal will be universally available. Wayland clients (including Xwayland) would need to support any combination of methods.
libei side-steps this issue by making the communication itself a an implementation detail and providing different negotiation backends. A client can attempt to establish a libei context through a Flatpak Portal first and fall back onto a public DBus interface and then fall back onto e.g. a named UNIX socket. All with a few lines of code only. There is only one spot the client has to care about this, the actual emulation of input is identical regardless of backend.
A pseudo-code implementation for server and client are available in the examples/
directory.
The server starts a libeis
context (which can be integrated with flatpak portals) and uses the libeis
file descriptor to monitor for client requests.
A client starts a libei
context and connects to the server - either directly, via DBus or via a portal. The server (or the portal) approves or denies the client. After successful authentication the server sends one or more seats (a logical group of devices) to the client; the client can request the creation of devices with capabilities pointer
, keyboard
or touch
, etc. in those seats.
The client triggers input events on these devices, the server receives those as events through libeis
and can forward them as if they were regular input events. The server has control of the client stream. If the stream is paused, events from the client are discarded. If the stream is resumed, the server will receive the events (but may discard them anyway depending on local state).
The above caters for the xdotool
use-case.
For a synergy
use-case, the setup requires:
synergy-client
on host A capturing mouse and keyboard events via an unspecified protocolsynergy-server
on host B requesting a mouse/keyboard capability device from the compositorsynergy-client
receives events via from compositor A it forwards those to the remote synergy-server
which sends them via libei
to the compositor B.libei does not provide a method for deciding when events should be captured, it merely provides the transport layer for events once that decision has been made.
libreis
is designed to allow a third-party that does not have a full context to manage restrictions. This is aimed at portals that only have the file descriptor to the EIS implementation but cannot initiate a full EI context.
libreis
works so that this third-party can configure the EIS implementation to restrict the abilities of an EI client (later) connected to this implementation. For example, a portal can set the client name using libreis
based on the app-id. A client cannot override this name later.
libei functionality is a superset of XTest's input emulation which consists of a single request, XTestFakeInput
. This request allows emulation of button, key and motion events, including X Input 1.x events (but not XI2). So libei can be a drop-in replacement since it supports the same functionality and more.
However, XTest is an X protocol extension and users of XTest usually obtain more information out-of-band (“out-of-band” here means “not through XTest but instead other X protocol requests”).
One example is xdotool
which does window focus and modifier mangling (see below). Window focus notification is not available to a pure libei client and would have to be obtained or handled on a separate channel, e.g. X or Wayland. Having said that, a Wayland client does not usually have acess to query or modifiy the window focus.
Modifiers in xdotool
are handled by obtaining the modifier mask from the X server, identifying any difference to the intended mask and emulating key events to change the modifier state to the intended one. For example, if capslock is on, xdotool would send a capslock key event first (thus disabling capslock) and then the actual key sequence. This is followed by another capslock key event to restore the modifier mask.
This is not possible for a pure libei client as the modifier state is maintained by the windowing system (if any). A client can obtain the modifier state on Wayland on wl_keyboard.enter
but when the client is in-focus, there is rarely a need to emulate events.
Overall, it is best to think of libei devices as virtual equivalents to a hardware device.
Where flatpak portals are in use, libei
can communicate with the portal through a custom backend. The above diagram modified for Flatpak would be:
+--------------------+ | Wayland compositor |_ +--------------------+ \ | libinput | libeis | \_wayland______ +----------+---------+ \ | [eis-0.socket] \ /dev/input/ / \\ +-------+------------------+ | ======>| libei | Wayland client A | | after +-------+------------------+ initial| handover / connection| / initial request | / dbus[org.freedesktop.portal.RemoteDesktop.ConnectToEIS] +--------------------+ | xdg-desktop-portal | +--------------------+
The current approach works so that
libeis
socket backend at $XDG_RUNTIME_DIR/eis-0
xdg-desktop-portal
provides org.freedesktop.portal.RemoteDesktop.ConnectToEIS
xdg-desktop-portal
to request emulated inputxdg-desktop-portal
authenticates a client and opens the initial connection to the libeis
socket. It restricts the capabilities available on that socket (e.g. sets the client name based on app-id
using libreis
).xdg-desktop-portal
hands over the file descriptor to the client which can initialize a libei
contextlibei
and libeis
talk directly to each other, the portal has no further influence.This describes the current implementation. Changes to this approach are likely, e.g. the portal may control pauseing/resuming devices (in addition to the server). The UI for this is not yet sorted.
Sandboxing is addressed via flatpak portals but a further level is likely desirable, esp. outside flatpak. The simplest solution is the client announcing the name so the UI can be adjusted accordingly. API wise-maybe an opaque key/value system so the exact auth can be left to the implementation.
For the use-case of fowarding input (e.g. synergy
) we need a method of capturing input as well as forwarding input. An initial idea was for libei to provide capability monitoring, i.e. a client requests all events from a specific capability. As with input emulation same benefits would apply - input can only be forwarded if the compositor explicitly does so.
However, this fails in the details. For example, for synergy
we need capability monitoring started by triggers, e.g. the client requests a pointer capability monitoring when the real pointer hits the screen edge. Or in response to a keyboard shortcut.
Some of the capabilities are distinctively display server-specific, for example the concept of a seat and a device is different between X and Wayland.
At this point, no implementation of capability monitoring is planned for libei.
The emulated input may require a specific keyboard layout, for example for softtokens (usually: constant layout “us”) or for the synergy
case where the remote keyboard should have the same keymap as the local one, even where the remote host is configured otherwise.
In libei, the server informs the client about the keymap it expects and it is up to the client to provide the correct keyboard events.
Modifier state handling, group handling, etc. is still a private implementation so even where the server supports individual keymaps. So it remains to be seen if this approach is sufficient.
There are PoC implementations of using libei
within Xwayland and connecting it to a libeis
context in the compositor (PoC with Weston). This allows Xwayland to intercept XTEST events and route those through the compositor instead.
+--------------------+ +------------------+ | Wayland compositor |---wayland---| Wayland client B | +--------------------+\ +------------------+ | libinput | libeis | \_wayland______ +----------+---------+ \ | | +-------+------------------+ /dev/input/ +---brei----| libei | Xwayland | +-------+------------------+ | | XTEST | +-----------+ | X client | +-----------+
Of course, Xwayland is just another Wayland client, so the connection between libei and libeis could be handled through a portal.
libei is not designed for short-lived fire-and-forget-type applications like xdotool
. It provides context and device negotiation between the server and the client - the latter must be able to adjust to limitations the server imposes.
The current implemtation of the protocol does not allow for a libei
client to send all requests in bulk and exit. The decision on whether to accept a device is ultimately made by the caller implementation and non-deterministic. For libei to support a batch request, someone would have to wait. It cannot be the server as the exact requirements are unknown: do we pause processing on the client altogether? We may miss a disconnect event? Do we pause processing for one device only? But then we may be re-ordering input events and cause havoc.
It could be libei
itself to implement these event queues but this too can mess with the input order. And implementing an event queue is not hard, so this issue is punted to the caller instead. Xwayland in its current implementation already does this.
uinput is a Linux kernel module that allows creating /dev/input/event
-compatible devices. Unlike XTest it is independent of a windowing system but requires write access to /dev/uinput
, usually limited to root. uinput devices are effectively identical to physical devices and will thus work on the tty and in any windowing system.
From the POV of a libei
client, uinput is a server implementation detail. The client does not need to know that the manager employs uinput to create the devices, it merely connects to an available EIS instance.
+---------+ | server |___auth channel__________ +---------+ \ | libeis |- +---------------------+ +---------+ \____brei___| libei | application | | +---------------------+ /dev/uinput/
Note that the server would likely need some authentication channel to verify which client is allowed to emulate input devices at the kernel level (polkit? config files?). This however is out of scope for libei. An example uinput server is implemented in the eis-server-demo
in the with libei repository.
In a Wayland and/or sandboxed environment, emulating input events requires going through the XDG RemoteDesktop portal. This portal is available on DBus but communication with DBus is often cumbersome, especially for small tools.
liboeffis
is an optional library that provides the communication with the portal, sufficient to start a RemoteDesktop session and retrieve the file descriptor to the EIS implementation.