You're reading things correctly
The Martial Arts feature only scales your damage from attacks that are made with either unarmed strikes and monk weapons. The poison rider is an add-on effect which is only inflicted on a failed save, while the trigger for that save is due to a successful hit, it itself isn't due to a weapon's damage roll from the Martial Arts feature.
The relevant parts of the Martials Arts rules are below (emphasis mine):
You gain the following benefits while you are unarmed or wielding
only monk weapons and you aren’t wearing armor or wielding a shield:
- You can use Dexterity instead of Strength for the attack and damage rolls of your unarmed strikes and monk weapons
- You can roll a d4 in place of the normal damage of your unarmed strike or monk weapon. This die changes as you gain monk levels, as
shown in the Martial Arts column of the Monk table.
In other words, the series of events are:
- Make an attack roll using Dexterity per the Martial Arts feature.
- If it hits, roll your Martial Arts damage die, also as per the Martial Arts feature.
- From your specific character's racial feature, the opponent then rolls a Constitution save. If they fail, they are subject to the poison damage.
The cause of the poison's damage has nothing to do with the Martial Arts feature. It is only because of your racial feature that the opponent may be subject to poison damage, it has nothing to do with the Martial Arts feature and thus doesn't scale per that feature.