In this question I have prepared this Geometry Nodes setup to represent the following function
$$f(x,y) = \frac{1}{2}\,\left( y - x + 1 - \sqrt{\left(y-x+1\right)^2 -4y}\,\right) $$
where $$0<=x<=1$$ $$0<=y<=1$$
But the Delete Geometry node will result in jagged edges. Is there another way to use Geometry Nodes to have smoother edges, perhaps without this Delete Geometry node or maybe by merging vertices?
(Using Blender 3.6.8)
The function $f(x,y)$ is defined in the set of real numbers if and only if $(y-x+1)^2-4y \ge 0$. By restricting $(x,y) \in [0,1]^2$, this condition is equivalent to $1-\sqrt{x} \ge \sqrt{y}$ (see the Mathematics section thereafter).
Because the derivative of $\sqrt{u}$ is $\frac{-1}{2\sqrt{u}}$, the limit of the gradient of $f$ along the curve of equation $\sqrt{y}=1-\sqrt{x}$ is $+\infty$. Consequently, the surface of equation $z=f(x,y)$ exhibits a vertical slope along this curve, but $z$ is a finite value equal to $\sqrt{y}$.
As a conclusion, the surface of equation $z=f(x,y)$ is smooth for $(x,y) \in [0,1]^2$ restricted to $\sqrt{x} + \sqrt{y} \le 1$. The observed jagged edges are a consequence of numerical round-off errors, not a mathematical behaviour.
(NB: this figure was achieved with a 2001x2001 grid)
The upper branch of the above graph is generating a mesh restricted to $\sqrt{x} + \sqrt{y} \le 1$ with $(x,y) \in [0,1]^2$ and $z=0$, as displayed in the top right subfigure.
1. To do so, a grid of size 1x1 is created to span the space $(X,Y)=(\sqrt{x},\sqrt{y})$. Because the Grid node is providing an object centred on the origin, $(X,Y) \in [-\frac{1}{2},\frac{1}{2}]^2$. A Transform Geometry node is used to shift it by $(\frac{1}{2},\frac{1}{2})$ such that $(X,Y) \in [0,1]^2$.
2. The condition $Y \le 1-X$ is defining the triangle of vertices with coordinates (0,0), (1,0) and (0,1). Conveniently, the Triangulate node is dividing each square along the diagonal of slope -1. Therefore a set of edges is created along the line of equation $Y=1-X$, between vertices with coordinates (1,0) and (0,1), dividing in two triangles the domain $[0,1]^2$ along its diagonal of slope -1.
3. Points such that $1-X \lt Y$ are removed by a Delete Geometry node to keep only those in the lower triangle, including the diagonal. To cope with numerical round-off errors, it is to notice that in the Subtract math node, a value of 1.0001 (instead of 1) is set.
4. The coordinates $(x,y)$ are computed from $(X,Y)$ with a component-wise Multiply vector math node as $(x,y)=(X^2,Y^2)$. Eventually the remaining grid points are moved using a Set Position node from coordinates $(X,Y,0)$ to $(x,y,0)$.
The lower branch of the above graph is generating the Offset $(0,0,f(x,y))$ used by a Set Position node to put points at coordinates $(x,y,f(x,y))$.
5. After recovery of $x$ and $y$ with a Separate XYZ node, $g(x,y)=y-x+1$ is computed.
6. Then $h(x,y)=-4y+g(x,y)^2$ is computed, knowing that $h(x,y) \ge 0$.
7. Eventually, $f(x,y)=\frac{1}{2} \left( g(x,y)-\sqrt{h(x,y)} \right)$ is assembled.
In what follows, it is assumed that $(x,y) \in [0,1]^2$. One consequence is that $(1+\sqrt{y})^2-x \ge 0$.
The purpose of this section is to demonstrate that:
$$
(y-x+1)^2 - 4y \ge 0 \Longleftrightarrow 1-\sqrt{x} \ge \sqrt{y}
$$
Indeed, $$ \begin{array}{rcl} (y-x+1)^2 - 4y \ge 0 & \Leftrightarrow & (y-x+1)^2 - (2\sqrt{y})^2 \ge 0 \\ \mbox{} & \Leftrightarrow & \left[y-x+1-2\sqrt{y}\right] \left[y-x+1+2\sqrt{y}\right] \ge 0 \\ \mbox{} & \Leftrightarrow & \left[(1-\sqrt{y})^2-x\right] \left[(1+\sqrt{y})^2-x\right] \ge 0 \\ \mbox{} & \Leftrightarrow & (1-\sqrt{y})^2-x \ge 0 \\ \mbox{} & \Leftrightarrow & 1-\sqrt{y} \ge \sqrt{x} \\ \mbox{} & \Leftrightarrow & 1-\sqrt{x} \ge \sqrt{y} \end{array} $$
Blender file: 
