It seems that you're trying to find a formula that gives a bijection between $\Bbb N$ and $\Bbb N\times\Bbb N$. Such a thing is possible, but not always pretty. Let me give you as nice a setup as I can.
Note: Hereinafter, I will use the definition $\Bbb N=\{0,1,2,3,...\}$, considered as a subset of the real numbers (so that absolute value and square roots make sense). Let me know if your definition of the natural numbers doesn't include $0$, and I'll adjust my answer accordingly.
For each $n\in\Bbb N$, let "gnomon $n$" of $\Bbb N\times\Bbb N$ be the set $$G(n):=\bigl\{\langle x,y\rangle\in\Bbb N\times\Bbb N:\max\{x,y\}=n\bigr\}.$$ Note that the $G(n)$ are pairwise disjoint, non-empty sets. Also, $$\bigcup_{k=0}^nG(k)=\bigl\{\langle x,y\rangle\in\Bbb N\times\Bbb N:x\leq n,y\leq n\bigr\}$$ for any $n\in\Bbb N$. From this, it follows that $\Bbb N\times\Bbb N=\bigcup_{n\in\Bbb N}G(n)$. Moreover, noting that $G(0)$ has a single element, and that $G(n+1)$ consists of the $(n+2)^2$ elements of $\bigcup_{k=0}^{n+1}G(k)$, less the $(n+1)^2$ elements of $\bigcup_{k=0}^nG(k)$, and so $G(n+1)$ has $(n+2)^2-(n+1)^2=2n+3$ elements. Thus, each $G(n)$ has $2n+1$ elements.
For each $n,$ we define a relation $\sqsubset_n$ on $G(n)$ by $\langle w,x\rangle\sqsubset_n\langle y,z\rangle$ if and only if (i) $w<y$ or (ii) $w=y=n$ and $x>z$. Each $\sqsubset_n$ is a strict total order relation (as you can check).
We now define a relation $\sqsubset$ on $\Bbb N\times N$ by $\langle w,x\rangle\sqsubset\langle y,z\rangle$ if and only if (i) $\langle w,x\rangle\in S(m)$ and $\langle y,z\rangle\in S(n)$ for some $m<n$ or (ii) $\langle w,x\rangle,\langle y,z\rangle\in S(n)$ with $\langle w,x\rangle\sqsubset_n\langle y,z\rangle$. $\sqsubset$ is a strict total order relation (as you can check).
The idea, then, is to match up the elements of $\Bbb N$ and those of $\Bbb N\times\Bbb N$ so that the $<$-order of the elements of $\Bbb N$ agrees with the $\sqsubset$-order of the corresponding elements of $\Bbb N\times\Bbb N$. It will be simpler to start with the $\Bbb N\times\Bbb N\to\Bbb N$ map (call it $f$). We'll go slice by slice.
Starting in $S(0)$, we need $f(0,0)=0$. Now, suppose for some $n>0$ that we have finished assigning $f$-values to the members of $G(k)$ for $0\leq k< n$, so we have mapped pairs to each of the first $n^2$ natural numbers--that is, $0$ through $n^2-1$. Next, take $f(0,n)=n^2,$ and in general $f(j,n)=n^2+j$ for each $0\leq j\leq n$. Then take $f(n,n-1)=n^2+n+1$, and in general take $f(j,n)=n^2+n+(n-j)$ for $0\leq j< n$. Thus, we define the map $f:\Bbb N\times\Bbb N\to\Bbb N$.
This gets us partway to a nice formula, but having the two different general formulas when we were working in gnomon $n>0$ is a problem. We can fix it, though, by noting that $$f(j,n)=n^2+j=n^2+n+(j-n)$$ for $0\leq j\leq n,$ so that in general, $$f(x,y)=n^2+n+(x-y)$$ for $x,y\in\Bbb N$ with $\max\{x,y\}=n$. This was how we defined $f(x,y)$ for $\langle x,y\rangle\neq\langle 0,0\rangle$, and it's easy to see that it works when $x=y=0$, too. Hence, given any $x,y\in\Bbb N$, we have $$f(x,y)=\bigl(\max\{x,y\}\bigr)^2+\max\{x,y\}+(x-y).\tag{#}$$ At this point, we can use the formula $(\#)$ to check that $f(w,x)<f(y,z)$ if and only if $\langle w,x\rangle\sqsubset\langle y,z\rangle$. Since $\sqsubset$ strictly orders $\Bbb N\times\Bbb N$, it follows that $f$ is a one-to-one function. I'll leave it to you to check that it's onto, and so is a formula of the kind that (it seems) you're trying to find.
If you don't want the $\max$ notation in your formula, then observe that $$\max\{a,b\}=\frac12\left(a+b+|a-b|\right)$$ for any real $a,b$. (Why?) Then $$\begin{align}\left(\max\{x,y\}\right)^2 &= \frac14\left(x+y+|x-y|\right)^2\\ &= \frac14\left((x+y)^2+2(x+y)|x-y|+|x-y|^2\right)\\ &= \frac14\left((x+y)^2+2(x+y)|x-y|+(x-y)^2\right)\\ &= \frac14\left(2x^2+2y^2+2(x+y)|x-y|\right)\\ &= \frac12\left(x^2+y^2+(x+y)|x-y|\right),\end{align}$$ and $$\begin{align}\max\{x,y\}+(x-y) &= \frac12\left(x+y+|x-y|\right)+\frac12(2x-2y)\\ &= \frac12\left(3x-y+|x-y|\right),\end{align}$$ so our formula $(\#)$ can instead be written as $$f(x,y)=\frac12\left(x^2+y^2+3x-y+(x+y+1)|x-y|\right).\tag{##}$$
We'll define $g:\Bbb N\to\Bbb N\times\Bbb N$ in a similar fashion. We need $g(0)\mapsto\langle 0,0\rangle$.
Given $m>0$, there is some greatest $n$ such that $n^2\leq m$. We'll need to have $g(m)\in G(n+1)$ to avoid skipping any pairs. If $m-n^2<n+1$, we'll put $g(m)=\langle m-n^2,n+1\rangle$, and otherwise, we'll put $g(m)=\langle n+1,(n+1)^2-m\rangle$. Since $n^2\leq m<(n+1)^2$ by our choice of $n$, then $g(m)\in G(n)$. Again, though, we'd like a consistent formula. If $m-n^2<n+1,$ then $m-n^2\leq n,$ so $$(n+1)^2-m=2n+1-(m-n^2)\geq 2n+1-n=n+1.$$ Hence, we have $$g(m)=\bigl\langle\min\{m-n^2,n+1\},\min\{(n+1)^2-m,n+1\}\bigr\rangle,$$ where $n$ is the greatest natural number such that $n^2\leq m$.
For any real $a$, we denote by $\lfloor a\rfloor$ the greatest integer that is $\leq a$. For $a\geq0$, we necessarily have that $\lfloor a\rfloor$ is a natural number. Note that for natural numbers $m,n$, we have $n^2\leq m$ if and only if $n\leq\sqrt{m}$. For $m>0$, we defined $$g(m)=\bigl\langle\min\{m-\lfloor\sqrt{m}\rfloor^2,\lfloor\sqrt{m}\rfloor+1\},\min\{(\lfloor\sqrt{m}\rfloor+1)^2-m,\lfloor\sqrt{m}\rfloor+1\}\bigr\rangle,\tag{$*$}$$ and the definition also works for $g(0)=\langle 0,0\rangle$.
Now, for all real $a,b$ we have $$\min\{a,b\}=\frac12\left(a+b-|a-b|\right).$$ We could use this to rewrite $(*)$ without the $\min$ notation, but honestly, that just makes it worse.
It turns out that $f$ and $g$ are inverses of each other, though that's easier to see from the construction than from our final formulas.
