Originally a comment, but posting as an answer as requested.
Change how you treat/clean the pieces after rusting them.
The exact changes depend on the exact mix of chemicals you use to produce the rust, but in general most of the common options have some way you can get rid of a significant percentage of them.
As a specific example, one of my friends who is a rather active LARPer typically uses a blend of hydrogen peroxide, acetic acid (distilled white vinegar), and iodized table salt. In general with such a mix, the primary potential sources of irritation are any residual salt, any metal halide salts that formed as a result of interaction between the salt and the metal, and any residual hydrogen peroxide.
The salt and metal halides can be gotten rid of by simply rinsing thoroughly a couple of times with a lot of hot (almost boiling) distilled water (he typically uses a few liters even for a small piece). The stuff you care about (such as the iron oxides) is insoluble in even boiling water, so this should not remove any of the newly formed rust unless you also try to scrub the piece. Be careful with the water from this though, as it will probably be mildly acidic. It should be safe to pour down a regular kitchen drain, but might cause mild irritation if you get it on your skin, and you do not want to get it in your eyes.
The hydrogen peroxide is a bit trickier. It will eventually dissipate by itself by decomposing into water and atomic oxygen (and probably cause a bit more rusting in the process). You can accelerate this by either baking the piece at a 100-200 degrees Celsius for a few hours (preferably in an area with good ventilation), or washing with a solution of potassium iodide (also in a well ventilated area). The potassium iodide will in theory do a better job because it chemically catalyzes the decomposition, but it may discolor non-metallic parts because it produces elemental iodine, and should still be rinsed thoroughly with distilled water afterwards (ironically, this catalysis is actually why iodized table salt is usually included in mixes that use hydrogen peroxide). The heating approach obviously won’t work in some cases, but is a bit easier and a bit safer when it is an option.
On a side note, the skin irritation may actually be caused by two other aspects of the piece that have nothing to do with how it was treated to produce rust:
- The rust itself may actually be a source of irritation, especially if it's a particularly rough and uneven layer. A lot of people with sensitive skin will react to abrasive materials much more readily than most others, and this can be a contributing factor.
- The metal may have a high nickel content. Metallic nickel actually causes contact dermatitis for some people with prolonged exposure (among many other nasty things it can do to your body). This is actually why ‘hypoallergenic’ metal watch bands exist, they have no nickel in them. Good quality steel usually has a nickel content of about 2.25%, which is high enough to cause such a reaction. Corrosion makes this worse though, because many of the salts of nickel (including the oxide) also cause similar reactions.