How would you explain a tensor to a computer scientist?

Question

How would you explain a tensor to a computer scientist? My friend, who studies computer science, recently asked me what a tensor was. I study physics, and I tried my best to explain what a tensor is, and I said something along the lines of "a mathematical object that is described in between the mappings of vector spaces", and he wasn't quite about that definition. I understood why, since it is a pretty wordy definition I gave, so I decided to give a more, down to earth, definition, describing a tensor as some array, with n-dimensions. However, he was still kind of confused by this. Can anyone synthesise a decent definition, tailored to a computer scientist's understanding?

Answer 1

A sound mathematical definition of a tensor product is not trivial, certainly not trivial to understand. Then working with tensors in addition also requires to grasp the concept of a vector space and its dual, and so on. That is not an obvious, or productive, path to take for a computer scientist, even though only an understanding of all this will reveal the why behind tensor operations that can be found in software libraries.

From a CS perspective I would explain that tensors are, indeed, multi-dimensional arrays plus a number of basic operations on them that have turned out to be very useful in a lot of applications. For example, most of the operations in a popular library as numpy can be constructed from the following building blocks (not an exhaustive list):

• Reordering of indices.
• Applying a binary operator $\star$ to arrays with equal dimensions: $(A \star B)[I] = A[I] \star B[I]$.
• Free tensor product of arrays: $(A\otimes B)[I,J] = A[I] \cdot B[J]$.
• Introducing additional indices of any dimension: $A’[I,J] = A[I]$.
• Contraction of two indices with equal dimension: $A’[I] = \sum_k A[k, k, I]$.

Of course, there are also some not so trivial operations on certain tensors, such as inversion or $QR$ factorization, and so on. However, understanding how a library supports the operations above will make it much easier to understand. The often countless functions that a library offers are mostly just clever and optimized combinations of these basic operations. For example, a matrix multiplication is a combination of a free tensor product and a contraction. The trace of a square matrix is a contraction. And so on.

Answer 2

If your friend is a computer scientist, then he should be well acquainted with the concept of scalars, vectors, and matrices. I see a tensor as a generalization of scalars, vectors, and matrices to higher dimensions. In other words, a tensor is a multi-dimensional array.

A scalar is a single number. You can view a scalar as a special case of a vector, that is, a vector with a single dimension of size $1$. So, a vector is a generalization of a scalar.

A vector is an ordered collection of numbers in a single dimension. You can view a vector as a special case of a matrix, that is, a matrix with a single dimension of any size $n \ge 1$. So, a matrix is a generalization of a vector.

But a matrix is an array restricted to two dimensions, each with its own size. A tensor extends the notion of a matrix analogous to how a vector extends the notion of a scalar and a matrix extends the notion of a vector. A tensor can have any number of dimensions, each with its own size. A $3$-dimensional tensor can be visualized as a stack of matrices, or a cuboid of numbers having any width, length, and height. A $4$-dimensional tensor can be visualized as a sequence of cuboids, each having a fixed width, length, and height. A $5$-dimensional tensor can be visiualized as a matrix in which every entry is a cuboid, and so on. I think the below image is a nice summary...

Notice a $1$-dimensional tensor is the familiar vector, and a $2$-dimensional tensor is the familiar matrix.

Finally, in the same way scalars, vectors, and matrices can be multiplied by one another to map objects between vector spaces (or to map objects in the same vector space), multiplication with a tensor is no different. There are additional nuances and ideas related to the use and functionality of tensors (see response of WimC for a concise summary), but I think the explanation I provided will get your friend off to a good start.

Answer 3

When telling a computer scientist about a mathematical object, it may be helpful to first specify the type of this object. If the object is a function, what are the types of its inputs and outputs? If the object is a tuple, what are the types of its coordinates?

What might make answering this question difficult is that there are several different accepted definitions of tensors, and these definitions yield different answers to the question "What is the type of a tensor?" However, one can prove that there is a natural correspondence between these definitions.

As an analogy, think about how, given an $n$-dimensional vector space $V$ over a field $F$, there is a natural correspondence between linear transformations from $V$ to $V$ and $n \times n$ matrices of elements of $F$. Because of this correspondence, we often think about linear transformations and matrices interchangeably.

Note that this "natural correspondence" depends on a choice of basis for $V$; that is, we do not have a natural way to identify linear transformations with matrices unless we have specified a basis for $V$.

Two definitions of tensors are the following:

• Given a vector space $V$ over a field $F$, a $(p,q)$-tensor is a multilinear map $T: \overbrace{V^* \times \dots \times V^*} \times \overbrace{V \times \dots \times V }\to F$, where $V^*$ denotes the dual space of $V$, and a multilinear map is a map that is linear in each coordinate.

• Given an $n$-dimensional vector space $V$ over a field $F$ with basis $B$, a $(p,q)$-tensor is a $\overbrace{(p+q) \times \dots \times (p+q)}^n$ array of elements of $F$.

Just as one can prove that there is a natural correspondence between linear transformations and matrices, one can prove that there is a natural correspondence between multilinear maps and multidimensional arrays, which means that these definitions are essentially interchangeable.

There are applications of tensors to computer science. I do not know much about these applications, but I believe they arise when data is stored in a multidimensional array and one wants to study the multilinear-algebraic properties of this array.

Answer 4

For a computer scientist:

A tensor is to a vector (or matrix) what a factory[1] is to an object.

An n-dimensional tensor accepts the coordinate system of your screen, and returns an n-dimensional array that represents a geometric quantity that is independent of which coordinate system you chose. This geometric quantity is always the "same" quantity, but its representation changes because your coordinate system changed.

So all tensors look like n dimensional arrays (vectors, matrices, etc), but to qualify as a tensor the numbers in that array need to change consistently when you chose a different coordinate system.

Example:

Consider the 1-dimensional tensor that returns a vector of length one pointing straight up from the bottom of your screen. If you choose the coordinate system where the y axis increases from the bottom of the screen to the top, then this tensor will return the representation $$[0, 1]$$. However, if you choose the coordinate system where the y axis decreases from the bottom of the screen to the top, then this tensor will return $$[0, -1]$$ instead. Different coordinate systems mean different numbers in your array, but they represent the same geometric thing! So it's a valid tensor.

As a counter example, consider a "factory" that always returns $$[0, 1]$$ no matter which coordinate system it is given. In the first coordinate system mentioned above this represents a vector that points upwards. But in the second coordinate system it represents a vector that points downwards. It's not a tensor because the array of numbers it gives for different coordinate systems have a different geometric meaning (the values in the array failed to transform correctly).

For 2d examples, you can consider:

• a clockwise rotation of 90 degrees. In different coordinate systems it'll have different matrix representations,but they'll always do the same thing to vectors (namely rotate them 90 degrees clockwise). This is a covariant tensor because if you scale your coordinate system units by a factor of k the entries in your matrix will also scale by the same factor of k
• a measurement of the length of a vector. It looks like a symmetric 2d matrix. It's a contravariant tensor because if you scale your coordinate system units by a factor of k the entries in your matrix will scale by a factor of $\frac{1}{k}$
• a transformation which projects vectors onto the x axis and then rotates them towards the positive y axis. This will look like a 2d matrix but it's not a tensor, since applying it to the same vector in different coordinate systems will yield different results.

Answer 5

(Answer from a software engineer who has followed a video series on tensors and who would share his personal experience).

I think it is a mistake to assume that computer scientists only think in terms of concrete data structures like arrays. Although a tensor can be represented by a multi-dimensional array, we know that isn't what one is.

Computer languages have plenty of methods for representing abstract behaviours (e.g., interfaces and abstract classes) and many pieces of software contain complex abstractions and behaviours that are comparable to what you might see in mathematics. The difference is that software tends to be far more verbose and descriptive. Calling a function ƒ in mathematics is normal. Calling a function f in your program (even ASCII f) will get you a comment in the code review, plus a request to specify the parameters, return value and types and a lengthy structured comment that can generate API documentation.

(And before you downvote - I don't mean to suggest that mathematics is not rigorous; simply that the form in which it is presented on a whiteboard in a video lecture is far more concise than what a programmer would be used to seeing in their IDE).

So, I have often wondered whether/which mathematical concepts can be redrafted using programming languages. Maybe it won't work for everything (summing to infinity is not recommended) but there must be many areas where it is possible.

I haven't thought about this further (sorry - this isn't a full answer) but in general, the computer scientist brain can handle abstractions, but is used to seeing things in a far more verbose form than the mathematician with far fewer implied or inferred constraints.

Answer 6

Tensors are composed of vectors and they have Ranks

A Tensor of rank $\left[0\right]$: is a Scalar because it has no direction, and no components.

A Tensor of rank $\left[1\right]$: has $1$ index and $1$ basis vector, per component. An example of a rank $\left[1\right]$ Tensor in $3D$ would be: $$A= \left[A_x, A_y, A_z\right]$$

The indexes $x,y,z$ could represent directions of forces.

A Tensor of rank $\left[2\right]$: in $3D$ has $2$ indexes and $9$ sets of $2$ basis vectors giving $9$ total components:

$$A = \left[A_{xx}, A_{xy}, A_{xz}, A_{yx}, A_{yy}, A_{yz}, A_{zx}, A_{zy}, A_{zz}\right]$$

You might use a rank $2$ Tensor to represent the forces inside a solid object. The $1{st}$ index would represent the surface area pointing in the $x$, $y$, or $z$ direction. The $2_{nd}$ index would represent the force on that surface in the $x$, $y$, or $z$ direction.

A Tensor of rank $\left[3\right]$: in $3D$ has $3$ indexes and $27$ sets of $3$ basis vectors giving $27$ total components:

$$A = \left[A_{xxx}, A_{xyx}, A_{xzx}, A_{yxx}, A_{yyx}, A_{yzx}, A_{zxx}, A_{zyx}, A_{zzx},...,A_{zzz}\right] $$

Notice that Tensors can fit nicely in a multi-dimensional array, where indexing by pointers is natural.

Answer 7

The answer is in using multi-array, but in a specific way. By using the array dimensions to represent contravariant and covariant indices, you can effectively distinguish between different types of tensors and their corresponding transformations.

Start with

$\operatorname{int}[1,3]$: This represents a 1-rank tensor with contravariant indices (row indices)

$\operatorname{int}[3,1]$: This represents a 1-rank tensor with covariant indices (column indices)

Then up

$\operatorname{int}[1,3][1,3]$: This represents a 2-rank tensor with contravariant indices (row indices) for both dimensions.

$\operatorname{int}[1,3][3,1]$: This represents a 2-rank tensor with contravariant indices (row indices) for the first dimension and covariant indices (column indices) for the second dimension.

$\operatorname{int}[3,1][1,3]$: This represents a 2-rank tensor with covariant indices (column indices) for the first dimension and contravariant indices (row indices) for the second dimension.

$\operatorname{int}[3,1][3,1]$: This represents a 2-rank tensor with covariant indices (column indices) for both dimensions.

And so on for higher ranks/dimensions. (All examples use size 3, but you can have any size, and of course any type not just integers.)

This notation aligns with the mathematical representation of tensors and allows for easy interpretation of the tensor's structure and transformation properties.

It's a concise and intuitive way to represent tensors in, for example, C#, and it provides clarity regarding the contravariant and covariant nature of tensor indices.