I know that many kind of groups can be represented by matrices; for example: rotation groups can be represented by matrices. Especially all elements of rotation groups can be represented by orthogonal matrices with determinant positive one. Also Permutation group can be represented by matrices.
But I need to know:
are there any kind of group which can not be represented by matrices?
Or is there any kind of group which does not have a representation?
Can you show me sample?
There are infinite groups which cannot be represented by finite-dimensional matrices over any commutative ring. If a group can be represented by matrices in such a way then it is called linear$^{\ast}$. Not all groups are linear.
A group $G$ is residually finite if for every element $g\in G$ there exists a homomorphism $\phi_g: G\rightarrow F_g$ onto a finite group $F_g$ such that $\phi_g(g)$ is non-trivial. Equivalently, for every element $g\in G$ there exists an action of $G$ on a finite object such that the action of $g$ is non-trivial. It is a rather famous result of Malc'ev that finitely generated linear groups are residually finite. This allows you to conjure up non-linear groups almost at will!
Non-Hopfian groups One way of constructing finitely-generated, non-residually finite groups is to construct finitely generated groups which have a surjective endomorphism $G\rightarrow G$ which is not an isomorphism. Such groups are called non-Hopfian, and it is a result of (again) Malc'ev that these groups are non-residually finite. See this Math.SE answer of mine for the proof, and this one for examples of such groups (the main example is the group $\langle a, b; b^{-1}a^2b=a^3\rangle$).
Simple groups A second way of constructing finitely-generated, non-residually finite groups is to construct finitely generated infinite simple groups. Examples of such groups are Thompson's group's $T$ and $V$ (these can be realised as groups acting on the unit interval in very natural ways - see these notes or this answer of mine) and Tarski monster groups.
Higman's group The group $G=\langle a, b, c, d; a^{-1}ba=b^2, b^{-1}cb=c^2, c^{-1}dc=d^2, d^{-1}ad=a^2\rangle$ was the first example of a finitely generated, infinite group with no finite quotients. This clearly implies that $G$ is not residually finite. Higman's paper is a joy to read$^{\dagger}$, and in it he points out that $G$ can easilly be used to construct finitely generated, infinite simple groups (his paper was pre-Thompson's groups, and pre-Tarski monsters) - taking a maximal normal subgroup $N$ of $G$, $G/N$ must be simple!
$^{\ast}$Perhaps this definition really requires field not commutative ring, but everything in this answer works for the more general commutative ring definition.
$^{\dagger}$Higman, Graham (1951), A finitely generated infinite simple group, J.Lon. Math. Soc.
A representation of a group $G$ is a morphism $\rho:G\rightarrow\text{GL}(V)$ where $V$ is a vector space over a field $K$.
There are always at least the following two representations:
the trivial representation, where $\rho(g)=\text{id}_V$ for all $g\in G$.
the regular representation, where $V$ is the space of $K$-valued functions on $G$ and $\rho(g)\phi(x)=\phi(xg)$ for all $\phi\in V$ and for all $x, g\in G$.
EDIT: Also note that whenever $H$ is normal in $G$ any representation of $G/H$ can be lifted to a representation of $G$ by composing with the quotient homomorphism $G\rightarrow G/H$.
In particular if the quotient $G/H$ is abelian this gives rise to lots of representations of $G$ since abelian groups have many representations, starting with characters (i.e. $1$-dimensional representations)
As you stated, every (finite) permutation group can be represented by matrices (take $G=S_n$ and represent the permutations by permutation matrices (the permutations act naturally on a vector space of dimension $n$ over some field $K$)). Subsequently, every finite group can be embedded in a permutation group (a Theorem of Cayley), since it acts on itself by left (or right) multiplication.
