1

I'm trying to interpolate coordinates in my fragment shader. Unfortunately if close to the upper edge the interpolated value of fVertexInteger seems to be rounded up instead of beeing floored. This happens above approximately fVertexInteger >= x.97.

Example:

  • floor(64.7) returns 64.0 --> correct

  • floor(64.98) returns 65.0 --> incorrect

The same happens on ceiling close above x.0, where ceil(65.02) returns 65.0 instead of 66.0.

Q: Any ideas how to solve this?

Note:

  • GL ES 2.0 with GLSL 1.0
  • highp floats are not supported in fragment shaders on my hardware
  • flat varying hasn't been a solution, because I'm drawing TRIANGLE_STRIP and can't redeclare the provoking vertex (only OpenGL 3.2+)

Fragment Shader:

varying float fVertexInteger;
varying float fVertexFraction;
void main() {

    // Fix vertex integer
    fixedVertexInteger = floor(fVertexInteger);

    // Fragment color
    gl_FragColor = vec4(
        fixedVertexInteger / 65025.0,
        fract(fixedVertexInteger / 255.0),
        fVertexFraction,
        1.0 );

}
0

1 Answer 1

1

I don't know completely what the precision of a mediump float actually is in OpenGL ES on Android, but the most likely explanation is, that the literal (or the varying value in the actual code) 64.98 is already rounded to 65 (using round to nearest) in the first place, before it is even put into the floor.

7
  • Also thought about this, but I can't explain to myself why this happens. After all the nodes to interpolate between are clearly defined (e.g. 64.0 and 65.0). Interpolation on fVertexFraction running between 0.0 and 255.0 works pretty fine. Also tried to pass a shifted fVertexInteger (factor 1000) from vertex to fragment shader, without improvement.
    – Peter K.
    Commented Nov 13, 2013 at 12:47
  • @PeterK. "After all the nodes to interpolate between are clearly defined (e.g. 64.0 and 65.0)" - That doesn't mean the intermediate values like 64.98 are exactly representable. "Interpolation on fVertexFraction running between 0.0 and 255.0 works pretty fine." - That may be because the interval compared to the overall values is much larger. When using a rather small interval on rather large values, the individual increments are much less significant... Commented Nov 13, 2013 at 13:24
  • @PeterK. ...For example when interpolating 100 steps from 0 to 100, the step size is 1, which is in the 3rd digit when compared to the overall values (or the largest of those). But when interpolating 100 steps from 10 to 11 (or 64 to 65 for that matter), the step size is 0.01 and thus in the 4th digit compared to the actual values. So the step size is much less significant to the resulting value and more likely to get rounded away. Commented Nov 13, 2013 at 13:26
  • 1
    @PeterK. "Also tried to pass a shifted fVertexInteger (factor 1000)" - What do you mean with shifted? +1000 or *1000. The latter shouldn't make a difference for floating point numbers and the former should make things even worse (since you're decreasing the relative size of the interval even further). Commented Nov 13, 2013 at 13:29
  • You're right, but shouldn't a shifting by e.g. factor *100 on fVertexInteger work this out? Btw, floor() seems to work fine, as if (floor(fVertexInteger) > fVertexInteger) never occurs.
    – Peter K.
    Commented Nov 13, 2013 at 13:32

Not the answer you're looking for? Browse other questions tagged or ask your own question.