Structure group:
Given a fiber bundle with total space $E$, base space $M$, model fiber $F$ and projection $\pi$, the local trivialization condition says that given any $u\in E$ such that $u$ projects to $x$ ($\pi(u)=x$; $u$ is an element of the fiber above $x$) it is possible to map this element to $\varphi(u)=(x,f)$ where $\varphi:\pi^{-1}(U)\rightarrow U\times F$ ($U\subset M$ open) and $f\in F$.
Ignoring the question of domains, let $\varphi':\pi^{-1}(U)\rightarrow U\times F$ be a different local trivialization. It maps $u$ to $\varphi'(u)=(x,f')$.
The structure group of the fiber bundle is $G$ if for any two such overlapping local trivializations, $f'=g\cdot f$, where $g\in G$ and "$\cdot$" is a left action of $G$ on $F$. In this case $g$ is called a transition function (function because normally $g$ depends on $x$, but I just had $x$ fixed).
So an element of a fiber bundle can be represented as a pair $(x,f)$ where different $f$s corresponding to different representations are related by left actions by $G$.
Structure group of vector bundles:
A real (complex) vector bundle of rank $k$ is a fiber bundle for which $F=\mathbb{R}^k\ (F=\mathbb{C}^k)$ and $G=\text{GL}(k,\mathbb{R})\ (G=\text{GL}(k,\mathbb{C}))$ (with $G$ acting on the left via the fundamental vector representation).
If we have these two then the fibers are automatically vector spaces. We can see that by letting $u,v\in E$ be elements of the same fiber, so $\pi(u)=\pi(v)=x$, and taking two local trivializations $\varphi(u)=(x,f)$ and $\varphi(v)=(x,h)$ while $\varphi'(u)=(x,f')$ and $\varphi'(v)=(x,h')$.
Then let us define $u+v$ as $$ u+v=\varphi^{-1}(x,f+h), $$ which is the same as $\varphi'^{-1}(x,f'+h')$ because $$ \varphi'^{-1}(x,f'+h')=\varphi'^{-1}(x,gf+gh)=\varphi'^{-1}(x,g(f+h))=(\varphi'^{-1}\circ\varphi'\circ\varphi^{-1})(x,f+h)=u+v. $$
Because the tangent bundle of an $n$ dimensional manifold has $n$ dimensional fibers, and the tangent bundle is a vector bundle, the structure group is $\text{GL}(n,\mathbb{R})$ or $\text{GL}(n,\mathbb{C})$ depending on whether your manifold is real or complex.
Heuristics:
If this is confusing, consider that given a manifold $M$, and a local frame $(U,e_a)$, you can describe a tangent vector at $x\in M$ as $v=\sum_a v^ae_a$, so you can identify $v$ with the set of $n$ real numbers $v^a$. If you perform a frame transformation as $e'_{a'}=\sum_a\Lambda^a_{\ \ a'}e_a$,then the components $v^a$ will change as $v'^{a'}=\sum_a(\Lambda^{-1})^{a'}_{\ \ a}v^a$, where $\Lambda$ is a $\text{GL}(n,\mathbb{R})$-valued local function on the manifold.
Here we can consider the frame $(U,e_a)$ to provide a local trivialization (indeed for vector bundles, local trivializations = local frames), and $\Lambda$ is a transition function.
Reduction of the structure group:
Given a fiber bundle with structure group $G$, let $H$ be subgroup of $G$. If there is such a sequence of local trivializations $(U_\alpha,\varphi_\alpha)$ (where $\alpha$ is an element of some indexing set), such that the transition functions are all $H$-valued, instead of more generally $G$-valued, then we say that the structure group is reducible to $H$.
Example of reduction:
Consider the tangent bundle of $M$. Let $q\in\Gamma(T^*M\otimes T^*M)$ be a Riemannian metric on $M$. Let $(U^{(\alpha)},E_a^{(\alpha)})_{\alpha\in\mathbb{A}}$ be an open cover of $M$ by local trivializations of $TM$ ($\mathbb{A}$ is an indexing set), where a local trivialization is identified with a local frame.
Let $(U^{(\alpha)},e^{(\alpha)}_a)$ be another local frame for each $\alpha$, obtained from $(U^{(\alpha)},E_a^{(\alpha)})$ by applying the Gram-Schmidt ortogonalization process to each $E_a$. You are welcome to check that the GS-process is valid and gives smooth frames (if the original frames were smooth).
What we now have is a system of orthonormal frames $(U^{(\alpha)},e^{(\alpha)}_a)_{\alpha\in\mathbb{A}}$ covering $M$. Any such two frames differ as $e^{(\beta)}_a=\sum_b O^b_{\ a}e^{(\alpha)}_b$, where $O\in\text{O(n)}$ an orthogonal matrix. So we have achieved a reduction of the structure group from $\text{GL}(n,\mathbb{R})$ to $\text{O}(n)$.