As it is often the case the issue lies in conventions. For mathematicians the Lie Algebra generators are any basis that can span the algebra as a vector space, for physicist we usually require the generators themselves to be hermitian (e.g. think about the Pauli matrices), because of their interpretation as observables.
Let us also make clear the difference between Algebra and Group. The Lie algebra is a vector space and an algebra thanks to the Lie bracket (the commutator), it is denoted with small cap fraktur letters. So for this case: $\mathfrak{u}(N)$ is the Lie Algebra of the group of unitary matrices of dimension $N\times N$. The group is denoted by $U(N)$ and is a group under usual matrix multiplication, whose elements are indeed unitary (thus complex entries) matrices.
The algebra condition on the elements of the algebra is obtained by differentiating the unitarity condition,
$$A A^\dagger = 1,$$
which gives
$$ A + A^\dagger = 0$$
which is nothing else than the anti-hermitian condition. This means the Lie algebra is the vector space of all anti-hermitian matrices of dimension $N\times N$. So for a matrix $A\in \mathfrak{u}(N)$ the exponentiation does give you an element of $U(N)$, and it can be shown that all elements in the vicinity of the identity of $U(N)$ can be described by exponentiation of some element of $\mathfrak{u}(N)$. At this point we are free to describe the matrix $A$ as we wish. As physicist we want to pick a basis of hermitian elements (notice that if $A$ is anti-hermitian then $-iA$ is Hermitian, or vice versa if $\sigma$ is Hermitian then $i\sigma$ is anti-hermitian) so let us have a basis for $\mathfrak{u}(N)$ of hermitian elements multiplied by $i$ and voila
$$A = \sum_n c_n i \sigma_n \in \mathfrak{u}(N)$$
and exponentiating
$$\exp(A) = \exp\left(\sum_n c_n i \sigma_n \right) \in U(N)$$
For the case of $SU(N)$ it is the $\det = 1$ condition that when differentiated forces the matrices of its algebra to be traceless.
So far we have spoken implicitly of Lie algebras as real vector spaces, that is the $c_n$ above are real and don't change the hermitian properties of $A$. However one can also complexify the algebra (build a new algebra), so one obtains a vector space over the complex numbers, thus allowing for anti-hermitian and hermitian matrices. So we have
\begin{align}
\mathfrak{u}(N) &= \text{anti-hermitian matrices}\\[7pt]
\mathfrak{u}_\mathbb{C}(N) &= gl(N,\mathbb{C}) = \text{complex matrices size} N\times N
\end{align}