Judging from the following table consisting of compiled notations presented in offline edition of The ACS Style Guide and AMA manual of style, section 13.12 Units of Measure:
$$
\begin{array}{ll}
\hline
\text{Unit of measure} & \text{Symbol} & \text{Ref.} \\
\hline
\text{osmolar} & \text{osm (also osM, Osm)} & \text{[1, p. 191]} \\
\text{osmole} & \text{osm} & \text{[2, p. 635]} \\
\text{microosmole} & \text{µOsm} & \text{[2, p. 634]} \\
\text{milliosmole} & \text{mOsm} & \text{[2, p. 634]} \\
\hline
\end{array}
$$
symbols $\pu{Osm}$ and $\pu{osm}$ refer to the amount-of-substance of an osmotically active entity, and symbol $\pu{osM}$ refers to osmolar concentration suggesting the following relation:
$$\pu{1 osm L^-1} = \pu{1 Osm L^-1} = \pu{1 osM}$$
The letter “o” in $\pu{Osm}$ appears to be capitalized for the sake of better readability when used in conjunction with metric prefixes, and the letter “m” in in $\pu{osM}$ is likely capitalized to resemble analogy with the unit symbol for molarity (small capital letter “m”: ᴍ).
References
- The ACS Style Guide: Effective Communication of Scientific Information, 3rd ed.; Coghill, A. M., Garson, L. R., Eds.; American Chemical Society; Oxford University Press: Washington, DC; Oxford; New York, 2006. ISBN 978-0-8412-3999-9.
- AMA Manual of Style: A Guide for Authors and Editors, 11th ed.; American Medical Association, Ed.; Oxford University Press: New York, 2020. ISBN 978-0-19-024656-3.