I focus more on the question as was originally posted (and still stands to be so in the title), that what would be a rather mild condition sufficient to force $ h $ to be linear. Also, defining $ g : \mathbb R \to \mathbb R $ with $ g ( x ) = 2 h ( x ) $, the functional equation becomes
$$ g \big( g ( x ) \big) = g ( x ) + 2 x \text , \tag 0 \label 0 $$
and we can see that $ g $ is continuous/increasing/decreasing/injective/surjective/linear/… iff $ h $ is so. Thus, I only talk about $ g $ and conditions on it (because it is more convenient), as they can be easily translated to conditions on $ h $.
First, note that \eqref{0} implies injectivity of $ g $, since if $ g ( x ) = g ( y ) $ for some $ x , y \in \mathbb R $, then
$$ x = \frac { g \big( g ( x ) \big) - g ( x ) } 2 = \frac { g \big( g ( y ) \big) - g ( y ) } 2 = y \text . $$
Next, observe that $ g ( 0 ) = 0 $ is consequence of \eqref{0}. To see this, put $ x = 0 $ in \eqref{0} to get $ g \big( g ( 0 ) \big) = g ( 0 ) $, and use injectivity. So, both conditions were redundant in the original question.
For any nonnegative integer $ n $, denote the $ n $-th iteration of $ g $ by $ g ^ n $; i.e. $ g ^ 0 $ is the identity function on $ \mathbb R $ and for any nonnegative integer $ n $, $ g ^ { n + 1 } = g ^ n \circ g $.
Lemma 1. For any $ x \in \mathbb R $ and any nonnegative integer $ n $, we have $ g ^ n ( x ) = a _ n g ( x ) + b _ n x $, where $ a _ n = \frac { 2 ^ n + ( - 1 ) ^ { n + 1 } } 3 $ and $ b _ n = \frac { 2 ^ n + 2 ( - 1 ) ^ n } 3 $.
This can be proved by mathematical induction. The base case $ n = 0 $ is easy to verify. Assuming
$$ g ^ n ( x ) = a _ n g ( x ) + b _ n x \tag 1 \label 1 $$
for all $ x \in \mathbb R $, we can substitute $ g ( x ) $ for $ x $ in \eqref{1} and use \eqref{0} to get $ g ^ { n + 1 } ( x ) = ( a _ n + b _ n ) g ( x ) + 2 a _ n x $. It's straightforward to verify that $ a _ { n + 1 } = a _ n + b _ n $ and $ b _ { n + 1 } = 2 a _ n $.
Lemma 2. If $ g $ is decreasing, then $ g ( x ) = - x $ for all $ x \in \mathbb R $.
If $ g $ is decreasing, then we must have
$$ ( y - x ) \big( g ( y ) - g ( x ) \big) \le 0 \tag 2 \label 2 $$
for all $ x , y \in \mathbb R $. Letting $ y = g ^ n ( x ) $ in \eqref{2} for any positive integer $ n $, we get
$$ \big( a _ n g ( x ) + ( b _ n - 1 ) x \big) \big( ( a _ { n + 1 } - 1 ) g ( x ) + b _ { n + 1 } x \big) \le 0 $$
by Lemma 1. This can be written as
$$ \left( g ( x ) + \frac { b _ n - 1 } { a _ n } x \right) \left( g ( x ) + \frac { b _ { n + 1 } } { a _ { n + 1 } - 1 } x \right) \le 0 \text , $$
which by taking the limit as $ n $ tends to infinity, gives $ \big( g ( x ) + x \big) ^ 2 \le 0 $, and the result follows.
Lemma 3. If $ g $ is increasing and surjective, then $ g ( x ) = 2 x $ for all $ x \in \mathbb R $.
As $ g ( 0 ) = 0 $ and $ g $ is increasing, we will have $ g ( x ) < 0 $ for negative $ x $, and $ g ( x ) > 0 $ for positive $ x $. Therefore, we can define the function $ c : \mathbb R \setminus \{ 0 \} \to \mathbb R ^ + $ with $ c ( x ) = \frac { g ( x ) } x $. For any $ x \in \mathbb R \setminus \{ 0 \} $, \eqref{0} gives
$$ \frac { g \big( g ( x ) \big) } { g ( x ) } \cdot \frac { g ( x ) } x = \frac { g ( x ) } x + 2 \text , $$
which can be rewritten as
$$ c \big( g ( x ) \big) = f \big( c ( x ) \big) \text , \tag 3 \label 3 $$
where $ f : \mathbb R ^ + \to \mathbb R ^ + $ is defined with $ f ( y ) = 1 + \frac 2 y $. Note that $ f ( y ) > 1 $ for all $ y \in \mathbb R ^ + $, and thus by \eqref{3} and surjectivity of $ g $, we get $ c ( x ) > 1 $ for all $ x \in \mathbb R \setminus \{ 0 \} $. Again, since $ y > 1 $ implies $ f ( y ) < 3 $, by a similar argument, we get $ c ( x ) < 3 $ for all $ x \in \mathbb R \setminus \{ 0 \} $. Generally, we can use mathematical induction to prove that for any nonnegative integer $ n $ and any $ x \in \mathbb R \setminus \{ 0 \} $,
$$ c ( x ) > f ^ { 2 n } ( 1 ) = 2 - \frac 3 { 2 ^ { 2 n + 1 } + 1 } \quad \text {and} \quad c ( x ) < f ^ { 2 n + 1 } ( 1 ) = 2 + \frac 3 { 2 ^ { 2 n + 2 } - 1 } \text . $$
As $ \left( f ^ { 2 n } ( 1 ) \right) _ { n = 0 } ^ \infty $ is an increasing sequence convergent to $ 2 $, we have $ c ( x ) \ge 2 $ for all $ x \in \mathbb R \setminus \{ 0 \} $. Similarly, since $ \left( f ^ { 2 n + 1 } ( 1 ) \right) _ { n = 0 } ^ \infty $ is a decreasing sequence convergent to $ 2 $, we have $ c ( x ) \le 2 $ for all $ x \in \mathbb R \setminus \{ 0 \} $. The result follows.
Lemma 4. If $ g $ is continuous, then it is monotonous and surjective.
Define $ d : \left\{ ( x , y ) \in \mathbb R ^ 2 \big| x < y \right\} \to \mathbb R $ with $ d ( x , y ) = g ( y ) - g ( x ) $. Because $ g $ is continuous, $ d $ is a composition of continuous functions, and thus continuous itself. Since the domain of $ d $ is connected, its range must be an interval (possibly unbounded). As $ g $ is injective, $ d $ never takes the value zero. This means that the value of $ d $ is either always negative or always positive, which means that $ g $ is either decreasing or increasing; i.e. monotonous.
In case $ g $ is decreasing, it is surjective by Lemma 2. In case $ g $ is increasing, $ g ( x ) $ has the same sign as $ x $, for all $ x \in \mathbb R $. This, by \eqref{0}, proves that $ \big| g \big( g ( x ) \big) \big| > | g ( x ) | $ for all $ x \in \mathbb R \setminus \{ 0 \} $. Since $ g $ is continuous, $ g ^ 2 $ is continuous, too, and thus its range is an interval (possibly unbounded). By the previous result, this interval contains the range of $ g $. Conversely, the range of $ g ^ 2 $ is contained in the range of $ g $ (as is true for any function on $ \mathbb R $), which means that the ranges of $ g $ and $ g ^ 2 $ are the same. Hence, because $ \require {color} { \color { red } g } $ is injective and $ g ^ 2 = { \color { red } g } \circ { \color { blue } g } $, therefore $ { \color { blue } g } $ must be surjective.
Theorem. Assume that a function $ g : \mathbb R \to \mathbb R $ satisfies \eqref{0} for all $ x \in \mathbb R $. Then, the following are equivalent.
- Either $ g ( x ) = - x $ for all $ x \in \mathbb R $, or $ g ( x ) = 2 x $ for all $ x \in \mathbb R $.
- $ g $ is linear.
- $ g $ is monotonous.
- $ g $ is continuous.
So, having any of these additional conditions, you would only need to rule out $ x \mapsto - x $, which can be done for example by any of these:
- $ g $ is not decreasing.
- There is some $ x \in \mathbb R $ with $ g ( x ) \ne - x $.
- $ g ( 1 ) \ne - 1 $.
- $ g \left( - \sqrt 2 \right) < g ( \pi ) $.
- …
Whether being monotonous or continuous is considered to be a mild condition is a matter of taste. But I can't see any milder conditions that guarantee $ g $ to be linear. It seems that there are uncountably many functions $ g : \mathbb R \to \mathbb R $ satisfying \eqref{0}. This comes from noting that choosing a point $ x _ 0 $ and a value for $ f ( x _ 0 ) $, \eqref{0} seems to force the value of $ f ( x ) $ only for countably many $ x $. Choosing $ x _ 1 $ to be a point other than those, it seems that we're able to choose the value of $ f ( x _ 1 ) $ almost freely. This comes from the observation that the previous step seems to exclude at most countably many values. It seems that using axiom of choice and transfinite recursion, we can continue this process, and get uncountably many different solutions. A condition that rules out this many wild solutions and keeps only two tame ones, should be a harsh one; shouldn't it? It seems that continuity is mild enough in this context.
