I tried asking this on network engineering but was pointed here...
Consider a network consisting of a router incapable of VLAN, some devices including multicast IPTV receivers, and a VLAN switch that does not support IGMP snooping, through which all devices are connected to the router. All multicast ethernet frames are broadcast to all devices on the switch, as it does not support IGMP snooping.
My idea: Seperate IPTV and non-IPTV devices by two different VLANs and connect both to the router, which will only see IPTV devices on a single port and only send multicast frames on that port. All devices stay DHCP mode and are given an IP by the router's DHCP server, all on the same IP network. The VLANs only exist to move the "nearest common switching point" of the two kinds of devices, to a device that is capable of sending multicast only to the right ports, but all devices stay on the same level 3 network. The double uplink is a hacky workaround to tag IPTV and non-IPTV frames. There should be no "routing" between the two VLANs because all device can find all other devices through ARP, there is no concept of the VLANs outside of this one switch.
switch router (no VLAN)
IPTV [U]====[10] [ ]
[10]=====[U]
other device [U]====[20] [ ]
[20]=====[U]
U = no VLAN
10/20 = untagged/access port for that VLAN
I tried this and it generally works, multicast frames do not reach non-IPTV-devices anymore, but all devices (including non-IPTV) experience very high packet loss and a very unstable connection, even when no multicast is being sent over the network at all. Why is this? Are there other important Layer 2 protocols, such as STP, which suffer from this loop between router and switch and are causing problems? The switch does not support STP.
EDIT: I am consciously abusing VLAN here for an unusual purpose, not to separate the broadcast domain completely.
The router acts as the multicast router and as a whole device, including its switch, does not egress multicast frames on ports other than those through which the devices have requested them. That part of the setup works.
The Router is a ISP-provided Telekom Speedport W925V, the switch in question is a Ubiquiti Flex Mini.