Is there any 'general' topological invariant to tell the difference between $M$ and $N$, where $M$ has homotopy type of a closed manifold and $N$ has homotopy type of a manifold with boundary.
I meant something like homotopy group/ homology group can 'detect' the obstruction of being homotopic to a closed manifold.
If $M$ is a closed manifold, then you can always cross it with an interval to get a manifold with non-empty boundary that is homotopic to $M$. So the distinction between the two possibilities (closed or non-closed) is not so well-defined in the homotopy category.
On the other hand, suppose that we fix the dimension $n$ of the manifolds we want to consider. Then, the most basic difference between closed manifolds and those with boundary is the top dimensional homology, namely: a closed orientable connected manifold of dimension $n$ has $H_n(M) = {\mathbb Z}$, while on the other hand, if $N$ has non-empty boundary, then $H_n(N) = 0$.
For simplicity I want to stick to the compact, orientable case. Further i want to assume, that $N$ has the structure of a CW-complex.
It might still be interesting to consider the case, where the dimension of $M$ and the dimension of $N$ are different; for example $D^2\simeq pt$ and $[0;1]\times S^1\simeq S^1$.
To find the dimension of $M$ one has to consider the structure of the (co-)jhomology and the $\cap$-product. The dimension of $M$ is the dimension of the highest non-vanishing homology $H_*(N)$. Futhermore $H_*(N)$ and $H^*(N)$ have to satisfy Poincare duality, so that the highest non-vanishing homology has to be $\mathbb{Z}$ and if one picks a generator, the $\cap$-product with this generator has to give isomorphisms. If this is satisfied, $N$ is called a "Poincare complex". (One can consider some simple examples of manifolds with boundaries like a disk bundle over a manifold or $S^1\times S^1\setminus D^2$).
The question whether a given Poincare complex is homotopy equivalent to a manifold (the other way round) is one classical problem in surgery theory.
There is a involved obstruction process coming, which is very roughly described in the wikipedia. More details can be found for example in the books mentioned there.
There should also be twisted versions of Poincare duality int the nonorientable case, which should also give a complete answer.
I do not know, whether the additional assumption, that the Poincare complex $N$ is indeed a manifold with boundary has any consequences for the surgery obstructions.
Perhaps the fundamental algebraic difference between a compact manifold with boundary and one without boundary is whether it satisfies Poincare duality or Lefschetz duality. Indeed, in algebraic surgery theory one defines the boundary of a chain complex as a "correction term" to a "Poincare duality" which does not hold, to turn it into a Lefschetz duality which does hold. This definition is fundamental to the discipline.
In more detail, let R be a ring with involution and consider a bounded chain complex over R
$C_\bullet:=C_0\longrightarrow C_{1}\longrightarrow\cdots\longrightarrow C_n.$
The dual cochain complex is
$C^\bullet:=C^n\longrightarrow C^{n-1}\longrightarrow\cdots\longrightarrow C^0,$
where by definition $C^i:= Hom_R(C_i,R)$.
Poincare duality PD:Hk(X)$\longrightarrow$ Hn-k(X) is induced by taking the cap product with a fundamental class $[X]\in C_n$. On a chain level, the cap product is given by the composition
$C^\bullet\overset{\Delta}{\longrightarrow }(C^\bullet)^{op}\otimes C^\bullet(X)\overset{\setminus}{\longrightarrow} Hom_R(C^\bullet,C_\bullet),$
where the first map is induced from the diagonal map via the Eilenberg-Zilber theorem, and the second is the slant map. The traditional name for $\setminus\Delta[X]$ is $\varphi_0$, which is a chain map from $C^\bullet$ to $C_\bullet$. Algebraic surgery theory also defines chain maps $\varphi_1,\ldots,\varphi_n$, which fit together to form a collection {$\varphi_0,\ldots,\varphi_n$} called a symmetric structure on the chain complex $C_\bullet$. The boundary of a chain complex with symmetric structure is defined as the mapping cone of $\varphi_0$. That's your invariant- no boundary if and only if that mapping cone is chain homotopic to zero.
This leads to definitions of L-groups, algebraic cobordism, algebraic surgery, and all kind of good and deep mathematics which goes along with all of that.
If you want to know more, the canonical reference for this sort of stuff perhaps, which has sort of a cult status, is Andrew Ranicki's ATS1.
