Disclaimer: Karakfa's answer originally helped me solve this problem. Although he provided the correct solution, I believe it can be helpful for other users to have a more detailed, pedagogic solution to this question. What follows is my attempt on doing so.
The key to solving this question is to separate the cases when player A loses and when player A wins. These correspond to when $n_a<N$ and $n_a=N$. Let's start with the first one.
In this case, we note that since $n_a<N$, then player A lost the game and consequently player B won. This means that $n_B=N$ (where $n_B$ denotes the points accumulated by player B in a given game), and hence $k+N$ rounds were played. Additionally, we also know that the winner of the last round was player B, since in this game the winner necessarily has to win the last round (otherwise the game would either not end at that round, or the winner already had $N$ points before that round, which means that the game would have already ended).
Now, we note that the problem of counting in how many different ways player A wins $k$ times, and player B wins $N$ times, including the last one, is equivalent to counting in how many ways we can select $k$ games, among $N+k-1$ (-1 since the last round was necessarily won by player B) to be won by player A. Putting it this way, it's clear that this is trivially ${N+k-1}\choose{k}$. For example, in the case where $k=2$ and $N=3$, there are 6 such sequences: $(AABBB), (ABABB),(ABBAB),(BAABB),(BABAB),(BBAAB)$ (here, each letter in the sequence denotes who won that particular round).
Now, back to the general case, we can see that all of the ${N+k-1}\choose{k}$ possible sequences of wins have probability $p^k(1-p)^N$ of happening, since each round is independent of the others. Finally, the sequences are mutually exclusive for one realization of the game, so we can obtain the desired probability by summing these individual, equal probabilities. This leads to:
$$ P(n_a=k) = \binom{N+k-1}{k} p^k(1-p)^N \quad,\quad\text{if}~k<N.$$
Now, for the case where player A wins, i.e. when $k=N$, then we know that $n_b<N$. This time, we have to consider all the sequences where player A won $N$ times and player B won $0,1,2,...,N-1$ times.
For a given $n_b=j,j<N$, there are $\binom{N+j-1}{j}$ possible sequences where player A wins the last round (similar argument as before). All of them are mutually exclusive and have the same probability: $(1-p)^jp^N$. Therefore, $P(N_b=j)=\binom{N+j-1}{j}(1-p)^j p^N,j<N$. To find $P(n_a=N)$, we simply have to sum the probabilities of the mutually exclusive events $n_b=j$, for $j=0,1,2,...N-1$.
$$P(n_a=N)=\sum_{j=0}^NP(n_b=j)$$
$$ = \sum_{j=0}^N\binom{N+j-1}{j}(1-p)^jp^N$$
$$ = p^N\sum_{j=0}^N\binom{N+j-1}{j}(1-p)^j$$
Finally, putting both results together yields Karakfa's answer:
$$ P(n_a=k) = \left\{\begin{matrix}
\binom{N+k-1}{k}p^k(1-p)^N & \text{if }k<N\\
\\
p^N\sum_{j=0}^N\binom{N+j-1}{j}(1-p)^j & \text{if }k=N
\end{matrix}\right. $$
