There are several points to consider here.
1) Depleted Uranium is not that much "depleted".
"Depleted uranium," the byproduct of the enrichment process, has about 0.002 percent 234U, 0.2 percent 235U and 99.8 percent 238U, and about 60 percent of natural uranium's radioactivity.
-- U.S. Office of the Secretary of Defense, emphasis mine.
2) Uranium is pyrophoric -- munitions that hit a hard target will ignite.
Upon impact on a hard surface, a uranium penetrator will undergo planar fracturing, i.e. it will "self-sharpen". (This is one of the characteristics that make uranium a viable choice for armor-piercing munitions.) It also means that what comes out the other side of the armor plate is no longer a solid penetrator, but a cloud of uranium, which will then ignite as uranium is pyrophoric.
This also means that what is being left on a battlefield is not the same thing as is lying around in a US arsenal (a metallic DU penetrator), but mostly triuranium octaoxide (U3O8), plus uranium dioxide (UO2) and uranium trioxide (UO3).
-- Report of the World Health Organization Depleted Uranium Mission to Kosovo, page 6.
Which will then get into the dust you breathe, the water you drink, and the food you eat, much more readily than a couple pounds of solid DU...
3) Uranium is toxic.
Uranium is a heavy metal, in itself already quite toxic. Many of its compounds are even more so. Think "mercury" and you are not far off the mark.
4) Uranium is mostly an alpha emitter.
That's fine if you are handling uranium (that lump of metallic DU), because alpha particles don't even penetrate your outer skin, let alone gloves.
But if you get an alpha emitter inside your body (like inhaling dust, or eating contaminated food), that's actually worse than e.g. a gamma emitter, because alpha particles do much more damage to what they do hit. What might be negligible on your outer skin does become a problem in your lung tissue, or your kidneys, where radiation does not have to penetrate deep to cause damage.
And since it's next to impossible to decontaminate the insides of your lungs, or your blood stream, those alpha emissions won't stop anytime soon.
I'm looking for hard evidence from well thought out scientific studies - not hearsay, not anecdotal, not sensationalism.
British Army doctors do think so, for one...
The big problem with all of the above, as with any chemical / radiological contamination, is to prove causality. People do get cancer. Children are born with birth defects. Sometimes numbers go up, sometimes they do get down. How can you prove that they went up for one specific reason, to the exclusion of anything else?
You don't do "in vitro" studies where you feed people uranium compounds, or let them breathe dust with uranium compounds in them, to compare their health compared to a control group...
Plus, especially following a large-scale conflict, getting reliable data on e.g. cancer rates before a conflict vs. after a conflict can be hard. The WHO report I linked above, for example, came up "inconclusive" in no small part due to the difficulties in getting reliable statistical data.
We know that uranium compounds are unhealthy, both chemically and radiologically. Not just resulting in cancer, but other unfavorable conditions as well.
In aggregate the human epidemiological evidence is consistent with increased risk of birth defects in offspring of persons exposed to DU.
-- Hindin, R.; et al. (2005). "Teratogenicity of depleted uranium aerosols: A review from an epidemiological perspective"
So, dodging the question as-asked because I know there are people out there just waiting to pounce on this because I cannot come up with a conclusive study for cancer in Serbia specifically... if in doubt, stay away from places where DU munitions hit hard objects. It's definitely not beneficial for your health.