Below is a list that tries to be exhaustive about the usage of square brackets. I tried to arrange them so that more common usages come first. Maybe such a list can never be complete, but are there uses other than these below?
Closed Intervals like $[a,b] = \{x\in\Bbb R\mid a\leqslant x\leqslant b\}$. Open or half-open intervals are usually written with parentheses at the respective end, i.e. $(a,b)$, $(a,b]$ or $[a,b)$. Sometimes reverse brackets are used: $]a,b[$, $]a,b]$ and $[a,b[$ which might have non-matching brackets, a feature that should be avoided, IMHO.
$R[x]$ or $R[x,y]$ for polynomials in $x$ resp. in $x$ and $y$ with coefficients in $R$. In most cases, $R$ is a ring, an integral domain or even a field, and $R[x]$ and $R[x,y]$ etc. denote Polynomial Rings. Similar notation is $R[[x]]$ to denote (formal) Power Series over $R$.
$E[X]$ for the Expected Value of a random variable or probability distribution $X$, but notations like $EX$ and $E(X)$ are also used. Frequently $\Bbb E$ is used instead of $E$. $E[X|Y]$ denotes expectation value for conditional probability. Sometimes completely different notation is used like $\bar X$ or $\overline X$ in physics.
With sub- and superscript for the difference of respective two function values like in $$\int_a^b\!\! f(x)\,dx = \big[F(x)\big]_{x=a}^{x=b} = F(b)-F(a)$$
With subscript used to indicate that a (complicated) function or expression is evaluated at that specific point, like in $$\left[\frac{\partial}{\partial x} f(x, \dot x, t)\right]_{t=1}$$
To denote the Equivalence Class of elements that are equivalent to an element $x$: $$[x] = \{y\mid y\sim x\}$$ where $\sim$ is an equivalence relation.
To denote the Homogeneous Coordinates of, say, a point in projective space $P\Bbb R^2$ as $[x:y:z]$ or $(x:y:z)$.
Stirling Numbers of the 1st Kind as $\begin{bmatrix}n\\k\end{bmatrix}$
For Simple Continued Fractions $$[a_0;a_1,a_2,\ldots] ~=~ a_0+ \underset{i=1}{\overset{\infty}{\Large\text{K}}}\,\frac1{a_i} ~=~ a_0+\cfrac{1}{a_1+\cfrac{1}{a_2+\cdots}}$$
$[a,b]$ for the Least Common Multiple, similar to the notation $(a,b)$ for the Greatest Common Divisor.
$f[x]$ for a function that's defined on a discrete set like on $\Bbb Z$ or $\Bbb N$ or for a Time-Discrete Signal, for example in the context of signal analysis and Z-Transform like ${\cal Z}\{x[n]\}$ for the ${\cal Z}$-Transform of time-discrete signal $x$.
For the Value of a Functional at a specific place like ${\cal F}[f]$ for the Fourier transform of $f$. More common are notations ${\cal F}\{f\}$, ${\cal F}(f)$ or just ${\cal F}f$.
$[a,b]$ for the Lie Bracket in a Lie algebra or Lie ring.
The Lie Bracket of a vector field. Like the Lie bracket in a Lie algebra it's a binary operation that's bilinear, anti-symmetric and obeys the Jacobi identity $[x,[y,z]] + [z,[x,y]] + [y,[z,x]] = 0$.
$[a,b]=ab-ba$ for the Commutator in Group and Ring theory that measures the degree of non-commutativity of an operation. In a group (where there's only one operation) it is $[a,b] = a^{-1}b^{-1}ab$.
$[X]$ for the Iverson Bracket, a generalization of Kronecker δ. For some expression / predicate $X$ that bracket evaluates to $1$ when $X$ is true, and to $0$ when $X$ is false. Kronecker δ represents as $\delta_{ij}=[i=j]$ for example.
$[n]_q$ for the q-Analog of $n$, also called q-bracket or q-number.
As Gauss Bracket $[x]$ to denote the greatest integer not greater than $x$, in programming sometimes called floor function. Iverson's notation $\lfloor x\rfloor$ is clearer and removes ambiguity due to the sheer number of different usages of $[~]$.
$[a,b,c] = (ab)c-a(bc)$ for the Associator that measures the degree of non-associativity of an operation.
For matrices and vectors. Some authors use $\begin{bmatrix} a&b\\c&d\end{bmatrix}$ instead of $\begin{pmatrix} a&b\\c&d\end{pmatrix}$ and $\begin{bmatrix} x\\y\end{bmatrix}$ instead of $\begin{pmatrix} x\\y\end{pmatrix}$ etc.
Instead of parentheses $()$ in order to to "override" the conventions for Precedence of Operations and operators and to determine in which order to evaluate an expression, like in $p(x)=x[1 + x(1+x)]$ instead of $p(x)=x(1 + x(1+x))$. Sometimes even mixed with braces $\{\}$ to add more confusion.
I am not really sure about the "Matrices and Vectors" point and that they really mean the same. So is that just an author's preference, some typographic consideration or even different semantics?