How do I get xkb settings to work via xwayland? They work fine when used on Wayland-native applications

Question

My custom keyboard mapping only works through xdg_shell applications, not through xwayland. I have configured Sway to show in the title bar what's running via xwayland and what's not, using for_window [tiling] title_format "%title — $shell".

My question: Is this a bug in Sway, XWayland, or is there something I can do to my config to make it work?

Versions:

• System: Ubuntu 20.10
• Sway: v1.5
• Xwayland: package version: 2:1.20.10-2

I created an xkb keymap using xmodmap under X. Example content:

add    mod3 = Mode_switch
keycode  40 = e E U2203 U2203

(keycode 40 because I run Dvorak)

This makes my Right Alt key Mode_Switch (Mod3), and makes Mod3+e write "∃" ("there exists").

I dumped my xkb using: xkbcomp -xkb :0 test.xkb. The relevant mapping in that file is:

   key <AC03> {
        type[group1]= "ALPHABETIC",
        type[group2]= "ALPHABETIC",
        type[group3]= "ONE_LEVEL",
        type[group4]= "ONE_LEVEL",
        symbols[Group1]= [               e,               E ],
        symbols[Group2]= [           U2203,           U2203 ],
        symbols[Group3]= [        NoSymbol ],
        symbols[Group4]= [        NoSymbol ]
    };

Then in Sway I use xkb_file test.xkb to load this keyboard layout. This works fine in apps that go via xdg_shell, but not ones that go via xwayland. Via xwayland it becomes nothing.

Examples X programs where it doesn't work:

• xterm
• google chrome (started with default parameters)

Example native wayland:

• gnome-terminal
• google chrome started with -enable-features=UseOzonePlatform -ozone-platform=wayland

I'm also comparing the output of xev -event keyboard no wev -f wl_keyboard.

In both X and Wayland I see the Mode_Switch keypress, but the following e key becomes keysym 0x0, NoSymbol. In wev I see the symbol being written just fine. See below for exact outpu.

xev -event keyboard
KeyPress event, serial 28, synthetic NO, window 0xc00001,
    root 0x525, subw 0x0, time 371977355, (63,73), root:(3107,635),
    state 0x0, keycode 66 (keysym 0xff7e, Mode_switch), same_screen YES,
    XLookupString gives 0 bytes: 
    XmbLookupString gives 0 bytes: 
    XFilterEvent returns: False

KeyPress event, serial 28, synthetic NO, window 0xc00001,
    root 0x525, subw 0x0, time 371977635, (63,73), root:(3107,635),
    state 0x4000, keycode 40 (keysym 0x0, NoSymbol), same_screen YES,
    XLookupString gives 0 bytes: 
    XmbLookupString gives 0 bytes: 
    XFilterEvent returns: False

KeyRelease event, serial 28, synthetic NO, window 0xc00001,
    root 0x525, subw 0x0, time 371977787, (63,73), root:(3107,635),
    state 0x4000, keycode 40 (keysym 0x0, NoSymbol), same_screen YES,
    XLookupString gives 0 bytes: 
    XFilterEvent returns: False

KeyRelease event, serial 28, synthetic NO, window 0xc00001,
    root 0x525, subw 0x0, time 371981259, (63,73), root:(3107,635),
    state 0x4000, keycode 66 (keysym 0xff7e, Mode_switch), same_screen YES,
    XLookupString gives 0 bytes: 
    XFilterEvent returns: False

wev -f wl_keyboard
[14:     wl_keyboard] modifiers: serial: 0; group: 0
                      depressed: 00000000
                      latched: 00000000
                      locked: 00000000
[14:     wl_keyboard] key: serial: 120526; time: 372129172; key: 66; state: 1 (pressed)
                      sym: Mode_switch  (65406), utf8: ''
[14:     wl_keyboard] modifiers: serial: 1; group: 0
                      depressed: 00000000
                      latched: 00000000
                      locked: 00000000
[14:     wl_keyboard] key: serial: 120528; time: 372129674; key: 40; state: 1 (pressed)
                      sym: U2203        (16785923), utf8: '\u2203'
[14:     wl_keyboard] key: serial: 120529; time: 372129756; key: 40; state: 0 (released)
                      sym: U2203        (16785923), utf8: ''
[14:     wl_keyboard] key: serial: 120530; time: 372130300; key: 66; state: 0 (released)
                      sym: Mode_switch  (65406), utf8: ''
[14:     wl_keyboard] modifiers: serial: 0; group: 0
                      depressed: 00000000
                      latched: 00000000
                      locked: 00000000

Answer 1

You should be using libxkbcommon instead of xmodmap by creating $HOME/.config/xkb. This article goes into detail on how Wayland and XWayland works with user-specific XKB configurations.

Wayland's keyboard configuration works like this:

• the compositor decides on the RMLVO keybard layout, through an out-of-band channel (e.g. gsettings, weston.ini, etc.)
• the compositor invokes libxkbcommon to generate a KcCGST keymap and passes that full keymap to the client
• the client compiles that keymap with libxkbcommon and feeds any key events into libxkbcommon's state tracker to get the right keysyms

The advantage we have here is that only the full keymap is passed between entities. Changing how that keymap is generated does not affect the client. This, coincidentally, is also how Xwayland gets the keymap passed to it and why Xwayland works with user-specific layouts.

I'm using Sway as well and haven't had any problems with my custom layout. This is my config with its own comprehensive README. There's additional resources in there as well.