How To Convert World Space Normal To Tangent Space?

Question

This question is a continuation of this post on How To Make Seamless Custom CubeMap?

The idea is to create a cube map with procedurally generated noise, extract the noise and a normal map of the noise for all six sides via a render target and a orthographic camera, download it as an image, and then read it back as a texture.

The problem is when reading back the normal map it produces seams, so the recommendation is to use noise that also produces derivatives along with the height called Analytical Derivatives. These derivatives are world space normals. I'm trying to convert the world space normal noise to tangent space before writing each side of the cube to its own render target.

I know that to compute the tangent space, you need three vectors: the tangent, bitangent, and normal, to form the TBN matrix, as stated in OpenGL Tutorial 13 : Normal Mapping

The library I'm using is Three.js, which has a method that computes the tangent for you, already defined here and sets it as a attribute to be passed to the shader. This makes my life a lot easier. I can then compute the TBN matrix like this:

vertex shader

attribute vec4 tangent;
varying   mat3 TBN;

...
vec3 _tangent = tangent.xyz;
vec3 bitangent = normalize(cross(_tangent,normal));
TBN = mat3(_tangent,bitangent,normalize(normal));

fragment shader

varying  mat3 TBN;
// IQ's version returns noise value in x, gradient in yzw.
float height = noise;
vec3 gradient = grad;
// Zero out component perpendicular to the sphere.
vec3 onSphere = gradient - dot(sampleDir, gradient) * sampleDir;
// Normal vector tilts away from "uphill" direction.
vec3 normal = normalize(sampleDir - onSphere * .1);//<---- worldspaceNormal 

normal = normalize(TBN*normal); //<---- tangentSpaceNormal 
gl_FragColor = vec4(vec3(normal), 1.0);

When outputting the tangent space results, there seems to be no difference between the world space and tangent space, besides a change in the direction of the green color.

world space normal:

TBN * world space normal:

I'm under the assumption that if I did it correctly, my normal map would look like the image on the left for the entire object. Instead, I still have my normals in world space.

let camera,scene,mesh,renderer

// Create a custom shader material
const vertexShader = `
attribute vec4 tangent;
varying  mat3 TBN;
varying vec3 grad;
varying float noise;
varying vec3 sampleDir;
varying vec2 vUv;
uniform vec3 center;

${mod289()}
${taylorInvSqrt()}
${permute()}
${snoise()}

vec3 orthogonal(vec3 n){
 return normalize(
 abs(n.x)>abs(n.z) ? vec3(-n.y,n.x,0.) : vec3(0.,-n.z,n.y)
 );
}


void main() {
  vUv = uv;
   
  vec4 worldPosition = modelMatrix * vec4(position, 1.0);
  vec3 sphere  = 50.*normalize(position-center)+center;  // This transforms cube into a sphere. 
  sampleDir    = normalize(worldPosition.xyz-center);    // Need to use world space so the noise on each quad can align. 
  vec3 shift   = vec3(0.,0.,0.);                         // this just offsets the noise in any direction.
  vec3 shiftedSample = sampleDir + shift;
  vec4 noiseGrad = snoise(shiftedSample,vec3(0.));
  noise = noiseGrad.x;                               
  grad  = noiseGrad.yzw;

 //compute the TBN matrix
   vec3 _tangent = tangent.xyz;
   vec3 bitangent = normalize(cross(_tangent,normal));
   TBN = mat3(_tangent,bitangent,normalize(normal));

  vec3 sphereD = sphere + 5.0 * noise * normalize(position-center); // add displacement to the sphere.
  gl_Position  = projectionMatrix * modelViewMatrix * vec4(sphereD, 1.0);
}
`;


const fragmentShader = `
varying  mat3 TBN;

uniform vec3 center;
varying vec3 grad;
varying float noise;
varying vec3 sampleDir;
varying vec2 vUv;





float lightv2(vec3 normalMap, vec3 lightPosition, vec3 cP) {

    vec3 lightDirection = normalize(lightPosition - normalMap.xyz);
    vec3 viewDirection = normalize(cP - normalMap.xyz);
    vec3 ambientColor = vec3(0.2, 0.2, 0.2);  // Ambient light color
    vec3 diffuseColor = vec3(0.2, 0.2, 0.2);  // Diffuse light color
    vec3 specularColor = vec3(0.2, 0.2, 0.2); // Specular light color
    float shininess = 0.0;  // Material shininess factor

    // Ambient lighting calculation
    vec3 ambient = ambientColor;

    // Diffuse lighting calculation
    float diffuseIntensity = max(dot(normalMap.xyz, lightDirection), 0.0);
    vec3 diffuse = diffuseColor * diffuseIntensity;

    // Specular lighting calculation
    vec3 reflectionDirection = reflect(-lightDirection, normalMap.xyz);
    float specularIntensity = pow(max(dot(reflectionDirection, viewDirection), 0.0), shininess);
    vec3 specular = specularColor * specularIntensity;

    // Final lighting calculation
    vec3 finalColor = ambient + diffuse + specular;
    return clamp(dot(normalMap.xyz, lightDirection), 0.0, 1.0) * max(max(finalColor.r, finalColor.g), finalColor.b);
}





void main() {

// IQ's version returns noise value in x, gradient in yzw.
float height = noise;
vec3 gradient = grad;
// Zero out component perpendicular to the sphere.
vec3 onSphere = gradient - dot(sampleDir, gradient) * sampleDir;
// Normal vector tilts away from "uphill" direction.
vec3 normal = normalize(sampleDir - onSphere * .1);//<---- worldspaceNormal 

normal = normalize(TBN*normal); //<---- tangentSpaceNormal 

vec3 lightDirection = vec3(0.,0.,100.);
vec3 cameraPosition = vec3(0.,0.,0.);
float finalColor = lightv2(normal,lightDirection,cameraPosition);

gl_FragColor = vec4(vec3(normal), 1.0);

}
`;


// animation
init()


function init(){
//-----------Basic setUp
renderer = new THREE.WebGLRenderer( { antialias: true } );
renderer.setSize( window.innerWidth, window.innerHeight );
renderer.setAnimationLoop( animation );
document.body.appendChild( renderer.domElement );
renderer.setClearColor( 'white' )

camera = new THREE.PerspectiveCamera( 70, window.innerWidth / window.innerHeight, 0.01, 1000 );
camera.position.z = 100;
var controls = new THREE.OrbitControls(camera, renderer.domElement);
scene = new THREE.Scene();


/********
- set creat mesh 
- set transfroms 
- set unifroms for undoing of transfoms
*********/
let widthHeight = 10

//------------front
let frontUnifrom = {ignoreFront:{value:0}}
let front = createPlaneMesh(0,0,0,0,0,0,frontUnifrom)
scene.add( front );
//-----------back
let bz    = -widthHeight
let bry   = Math.PI
var undorotationMatrix = new THREE.Matrix4();
undorotationMatrix.makeRotationY(-bry);
let backUnifrom = {rm:{value:undorotationMatrix},undoPoition:{value:new THREE.Vector3(bz*2,0,bz)},ignoreFront:{value:1}}
let back  = createPlaneMesh(0,0,bz,0,bry,0,backUnifrom)
scene.add( back );
//---------right
let rz    = -(widthHeight)/2;
let rx    =  (widthHeight)/2;
let rry   =  Math.PI/2;
var undorotationMatrix = new THREE.Matrix4();
undorotationMatrix.makeRotationY(-rry);
let rightUnifrom = {rm:{value:undorotationMatrix},undoPoition:{value:new THREE.Vector3(-rx,0,-rz)},ignoreFront:{value:1}}
let right = createPlaneMesh(rx,0,rz,0,rry,0,rightUnifrom)
scene.add( right );
//---------left
let lz    =  -(widthHeight)/2;
let lx    =  -(widthHeight)/2;
let lry   =  -Math.PI/2;
var undorotationMatrix = new THREE.Matrix4();
undorotationMatrix.makeRotationY(-lry);
let leftUnifrom = {rm:{value:undorotationMatrix},undoPoition:{value:new THREE.Vector3(-lx,0,-lz)},ignoreFront:{value:1}}
let left  = createPlaneMesh(lx,0,lz,0,lry,0,leftUnifrom)
scene.add( left );
//--------top
let tz    =  -(widthHeight)/2;
let ty    =  (widthHeight)/2;
let trx   =  -Math.PI/2;
var undorotationMatrix = new THREE.Matrix4();
undorotationMatrix.makeRotationX(-trx);
let topUnifrom = {rm:{value:undorotationMatrix},undoPoition:{value:new THREE.Vector3(0,-ty,-tz)},ignoreFront:{value:1}}
let top  = createPlaneMesh(0,ty,tz,trx,0,0,topUnifrom)
scene.add( top );
//---------bottom
let boz   =  -(widthHeight)/2;
let boy   =  -(widthHeight)/2;
let borx  =  Math.PI/2;
var undorotationMatrix = new THREE.Matrix4();
undorotationMatrix.makeRotationX(-borx);
let boUnifrom = {rm:{value:undorotationMatrix},undoPoition:{value:new THREE.Vector3(0,-boy,-boz)},ignoreFront:{value:1}}
let bo    = createPlaneMesh(0,boy,boz,borx,0,0,boUnifrom)
scene.add( bo );
}


//--------build mesh
function createPlaneMesh(x, y, z, rotationX, rotationY, rotationZ, uniforms) {
// Create a plane geometry
const planeGeometry = new THREE.PlaneGeometry(10, 10, 150, 150);
planeGeometry.computeTangents ()

uniforms.center = {value:new THREE.Vector3(0,0,-5)}
const planeMaterial = new THREE.ShaderMaterial({
    uniforms: uniforms,
    vertexShader: vertexShader,
    fragmentShader: fragmentShader,
});
// Create the plane mesh
const planeMesh = new THREE.Mesh(planeGeometry, planeMaterial);
// Set the position of the mesh
planeMesh.position.set(x, y, z);
// Set the rotation of the mesh
planeMesh.rotation.set(rotationX, rotationY, rotationZ);
planeMesh.frustumCulled = false
return planeMesh;
}


//-----------noise functions

function mod289(){
  return  `
vec3 mod289(vec3 x) {
  return x - floor(x * (1.0 / 289.0)) * 289.0;
}

vec4 mod289(vec4 x) {
  return x - floor(x * (1.0 / 289.0)) * 289.0;
}
  `
 }

function permute(){
  return  `
vec4 permute(vec4 x) {
     return mod289(((x*34.0)+10.0)*x);
}
  `
 }
function taylorInvSqrt(){
return     `
vec4 taylorInvSqrt(vec4 r)
{
  return 1.79284291400159 - 0.85373472095314 * r;
}
    `
}
function snoise(){
return `
vec4 snoise(vec3 v,  vec3 gradient)
{
  const vec2  C = vec2(1.0/6.0, 1.0/3.0) ;
  const vec4  D = vec4(0.0, 0.5, 1.0, 2.0);

// First corner
  vec3 i  = floor(v + dot(v, C.yyy) );
  vec3 x0 =   v - i + dot(i, C.xxx) ;

// Other corners
  vec3 g = step(x0.yzx, x0.xyz);
  vec3 l = 1.0 - g;
  vec3 i1 = min( g.xyz, l.zxy );
  vec3 i2 = max( g.xyz, l.zxy );

  //   x0 = x0 - 0.0 + 0.0 * C.xxx;
  //   x1 = x0 - i1  + 1.0 * C.xxx;
  //   x2 = x0 - i2  + 2.0 * C.xxx;
  //   x3 = x0 - 1.0 + 3.0 * C.xxx;
  vec3 x1 = x0 - i1 + C.xxx;
  vec3 x2 = x0 - i2 + C.yyy; // 2.0*C.x = 1/3 = C.y
  vec3 x3 = x0 - D.yyy;      // -1.0+3.0*C.x = -0.5 = -D.y

// Permutations
  i = mod289(i); 
  vec4 p = permute( permute( permute( 
             i.z + vec4(0.0, i1.z, i2.z, 1.0 ))
           + i.y + vec4(0.0, i1.y, i2.y, 1.0 )) 
           + i.x + vec4(0.0, i1.x, i2.x, 1.0 ));

// Gradients: 7x7 points over a square, mapped onto an octahedron.
// The ring size 17*17 = 289 is close to a multiple of 49 (49*6 = 294)
  float n_ = 0.142857142857; // 1.0/7.0
  vec3  ns = n_ * D.wyz - D.xzx;

  vec4 j = p - 49.0 * floor(p * ns.z * ns.z);  //  mod(p,7*7)

  vec4 x_ = floor(j * ns.z);
  vec4 y_ = floor(j - 7.0 * x_ );    // mod(j,N)

  vec4 x = x_ *ns.x + ns.yyyy;
  vec4 y = y_ *ns.x + ns.yyyy;
  vec4 h = 1.0 - abs(x) - abs(y);

  vec4 b0 = vec4( x.xy, y.xy );
  vec4 b1 = vec4( x.zw, y.zw );

  //vec4 s0 = vec4(lessThan(b0,0.0))*2.0 - 1.0;
  //vec4 s1 = vec4(lessThan(b1,0.0))*2.0 - 1.0;
  vec4 s0 = floor(b0)*2.0 + 1.0;
  vec4 s1 = floor(b1)*2.0 + 1.0;
  vec4 sh = -step(h, vec4(0.0));

  vec4 a0 = b0.xzyw + s0.xzyw*sh.xxyy ;
  vec4 a1 = b1.xzyw + s1.xzyw*sh.zzww ;

  vec3 p0 = vec3(a0.xy,h.x);
  vec3 p1 = vec3(a0.zw,h.y);
  vec3 p2 = vec3(a1.xy,h.z);
  vec3 p3 = vec3(a1.zw,h.w);

//Normalise gradients
  vec4 norm = taylorInvSqrt(vec4(dot(p0,p0), dot(p1,p1), dot(p2, p2), dot(p3,p3)));
  p0 *= norm.x;
  p1 *= norm.y;
  p2 *= norm.z;
  p3 *= norm.w;

// Mix final noise value
  vec4 m = max(0.5 - vec4(dot(x0,x0), dot(x1,x1), dot(x2,x2), dot(x3,x3)), 0.0);
  vec4 m2 = m * m;
  vec4 m4 = m2 * m2;
  vec4 pdotx = vec4(dot(p0,x0), dot(p1,x1), dot(p2,x2), dot(p3,x3));

// Determine noise gradient
  vec4 temp = m2 * m * pdotx;
  gradient = -8.0 * (temp.x * x0 + temp.y * x1 + temp.z * x2 + temp.w * x3);
  gradient += m4.x * p0 + m4.y * p1 + m4.z * p2 + m4.w * p3;
  gradient *= 105.0;

  float n =  105.0 * dot(m4, pdotx);
  return vec4(n,gradient);
}
      `
}
//-------------



function animation( time ) {
    renderer.render( scene, camera );

}

<script src="https://cdn.jsdelivr.net/npm/[email protected]/build/three.min.js"></script> 
<script src="https://cdn.jsdelivr.net/npm/[email protected]/examples/js/controls/OrbitControls.js"></script>

UPDATE: For anyone curious, this code shows how to generate noise in tangent space without seams. The drawback is that you have to sample the noise 3 times. I didn't use this as an answer because it doesn't show how to convert from world space normals to tangent space. That problem still persists:

  vec3 noiseNormal(vec3 worldPosition,vec3 normal, vec4 tangent){
    float noise = snoise3D(worldPosition.xyz);

    vec3 displacedPosition = worldPosition.xyz + normal * noise;
    
    float offset   = 0.01;
    vec3 tangent_  = tangent.xyz;
    vec3 bitangent = normalize(cross(normal, tangent_));
    vec3 neighbour1 =  worldPosition.xyz + tangent_ * offset;
    vec3 neighbour2 =  worldPosition.xyz + bitangent * offset;
    vec3 displacedNeighbour1 = neighbour1 + normal * snoise3D(neighbour1);
    vec3 displacedNeighbour2 = neighbour2 + normal * snoise3D(neighbour2);
    
    vec3 displacedTangent = displacedNeighbour1 - displacedPosition;
    vec3 displacedBitangent = displacedNeighbour2 - displacedPosition;
    
    vec3 displacedNormal = normalize(cross(displacedTangent, displacedBitangent));
    return displacedNormal;
    
  }

Answer 1

Your coordinate conversions are incorrect.

The tangent and normal attributes that three.js provide to you are in object space. For your planes, that means the tangent is always (1, 0, 0) (+x), and the normal is (0, 0, 1) (+z).

You need to map this to world space by multiplying it by your model matrix. That's how it will account for each plane being oriented differently for the six faces of the cube.

Then the matrix you form from mat3(tangent, bitangent, normal) will convert from tangent space to world space.

To map the other way, encoding a world space normal into tangent space, you need its inverse. Fortunately, for an orthonormal matrix (which this is by construction), that's just the transpose.

Working this way, you'd have one consistent tangent frame for the entire cube face. Strictly speaking, this tangent frame is correct only in the center of the face. After puffing your cube out into a sphere, this is the only spot where the surface normal and tangent still point straight out along axis-aligned directions. The rest of the round surface naturally curves away from that axis-aligned frame. So if you use this to encode your normals into tangent space, you'll get that familiar pale blue only in the middle of each chart, and they'll tint to pink/green/purple toward the edges, as the normal map is forced to encode the curvature of the sphere that you didn't account for in your TBN matrix.

If you want to get more uniform behaviour over the surface, you can compute a unique tangent frame at each point, taking into account the local orientation of the sphere there. Here's a function that does this:

// Takes a point on the unit cube centered at the origin
// and a worldspace tangent for the cube face we're rendering,
// then computes a surface normal for the corresponding point
// on the unit sphere, and a tangent perpendicular to that normal.
void sphereBasis(in vec3 cubePosition, inout vec3 tangent, out vec3 normal) {
    float scale = dot(cubePosition, cubePosition);

    normal = cubePosition/sqrt(scale);

    tangent = normalize(tangent * scale - cubePosition * dot(tangent, cubePosition));    
} 

You can then use the cross trick to get a bitangent to complete the orthonormal TBN matrix. This is correct across the u direction, but doesn't match the v direction to the actual curvature of the mesh (look at the corners of the cube after it's puffed into a sphere: the lines along its edges point 120 degrees from each other, rather than the usual 90). But this inaccuracy may be worthwhile for still giving you the cheap and easy transpose-to-invert.

If you want the binormal to splay out to match the "vertical" grid lines along each cube face, you can use the same formula we used for tangent, just swapping the binormal instead. But then inverting the matrix is a little messier, since the tangent and bitangent directions are no longer orthogonal. There might be a fast formula for this, since two of the three pairs of axes are still orthonormal, I just don't know it off the top of my head

---

Here's a sample showing that this correctly maps the world space normal we get from our noise function into a reasonable tangent space at each point, and back, seamlessly. You can change the noiseOctaves variable to see how it looks with both high-frequency detail and smooth gradients.

• At the back, you can see the generated tangent space normal maps, laid out as the net of the cube. We create these by sampling a noise function to get a world space normal, constructing a tangent space using the code above, and using the transpose of the TBN matrix to map the world space normal into a tangent space version to write into the texture.

  Don't worry about the visible seams between charts in the net. Just like in the previous Q&A, this is due to the projection changing abruptly at those edges - they're tangent space seams. The projection back onto a sphere and mapping back into world space using the TBN matrix will undo this projection and bring adjacent sides back into agreement, as confirmed by the two spheres:

• On the left, there's a sphere showing world space normals computed directly from the noise function.

• On the right, there's a sphere showing normals read from the tangent space normal maps, then mapped back to world space using the same TBN matrix we used to encode them. If there were an error in the mapping, we'd expect visible seams in this cube, or dramatic differences from the cube on the left. (Minor differences are inevitable due to rasterization and quantization artifacts when saving the continuous noise output to an 8bpc texture, but I'd say this looks quite adequate for typical shading needs).

---

If you're curious about the derivation, it comes from this - taking the partial derivative of our "spherify" function \$s\$ (also called normalize()) to see how our on-sphere position changes as our input x coordinate increases:

$$ s(\begin{bmatrix}x\\y\\z\end{bmatrix}) =\frac 1 {\sqrt{x^2 + y^2 + z^2}} \begin{bmatrix}x\\y\\z\end{bmatrix}\\ \text{let} \, l = \sqrt{x^2 + y^2 + z^2}\\ \frac \delta {\delta x} l = \frac 1 2 l^{-1} \cdot 2 x = \frac x l\\ \begin{align} s(\begin{bmatrix}x\\y\\z\end{bmatrix}) &= \frac 1 l\begin{bmatrix}x\\y\\z\end{bmatrix}\\ \frac \delta {\delta x} s(\begin{bmatrix}x\\y\\z\end{bmatrix}) &= \left(l \begin{bmatrix}1\\0\\0\end{bmatrix} - \frac x l \begin{bmatrix}x\\y\\z\end{bmatrix}\right) \frac 1 {l^2}\\ l^3\frac \delta {\delta x} s(\begin{bmatrix}x\\y\\z\end{bmatrix}) &= l^2 \begin{bmatrix}1\\0\\0\end{bmatrix}\ - x \begin{bmatrix}x\\y\\z\end{bmatrix}\\ \end{align} $$

I pulled the \$l^3\$ term to the left side at the end because we're going to normalize the vector anyway, so we don't care about a scalar length factor, and this makes the expression on the right much simpler. The code version just generalizes this to the case where the tangent is an axis-aligned unit vector, not necessarily always (1, 0, 0).

As it turns out, conveniently, both the normal and tangent vectors change linearly over the face, so you could do this calculation in the vertex shader and interpolate the result, but you'd still have to normalize them and compute the bitangent per-fragment, so you might not gain much from taking up the extra interpolator slot (particularly on mobile, where tile-based rendering can make fatter vertices more costly).