I think that this post is full of answers with great suggestions and excellent discussion. However, you seem very interested in a direct approach, and so I'd like to offer a mathematical approach that answers your question more directly with a meta-analysis approach. For various reasons, you may not want to actually use this approach.
For simplicity, let's focus on a one-sided test
\begin{align*}
H_0: \mu_x &= \mu_y \\
H_1: \mu_x &> \mu_y \\
\end{align*}
Now let's get some notation out of the way. Suppose our first sample consists of data $X_{11}, \ldots, X_{1n_1}$ and $Y_{11}, \ldots, Y_{1m_1}$, and our second sample consists of $X_{21}, \ldots, X_{2n_2}$, and $Y_{21}, \ldots, Y_{2m_2}$. Let's define
\begin{align*}
D_1 &= \frac{1}{n_1}\sum_{i=1}^{n_1}X_{1i} - \frac{1}{m_1}\sum_{i=1}^{m_1}Y_{1i} \\
D_{12} &= \frac{1}{n_1+n_2}\left(\sum_{i=1}^{n_1}X_{1i}+ \sum_{i=1}^{n_2}X_{2i}\right) - \left(\frac{1}{m_1}\sum_{i=1}^{m_1}Y_{1i}+ \sum_{i=1}^{m_2}Y_{2i}\right).
\end{align*}
Since we already know that the first sample leads to a p-value of less than $0.05$, we can condition on the fact that $D_1 \in \mathcal R_1$, where $\mathcal R_1 = \{d : d > d_0\}$ is a rejection region of size $0.05$ for the appropriate choice of $d_0$. Conditioning on this fact, we can calculated an adjusted p-value using the following formula:
\begin{align*}
\text{p-val} &= P(D_{12} > \hat D_{12} | D_1 > d_0, H_0) \\[1.5ex]
&= \frac{P(D_{12} > \hat D_{12} \cap D_1 > d_0 | H_0)}{P(D_1 > d_0)} \\[1.2ex]
&= \frac{P(D_{12} > \hat D_{12} \cap D_1 > d_0 | H_0)}{0.05},
\end{align*}
where $\hat D_{12}$ is the value actually obtained in your sample. Thus, to compute the adjusted p-value, you can run a similar permutation test as before, but we need to (i) throw away all cases where $D_1$ is not significant and (ii) we multiply the obtained p-value by $20$.
I will give R code to perform this below, but first let me describe the results. I simulated data using $n_1=n_2=m_1=m_2 = 500$ using normal distributions with $\mu_x=10$, $\mu_y=9.9$ and standard deviations equal to $1$. I then wrote code to approximate the p-value for a permutation test with and without correction. Without correction (possible p-hacking) I get a p-value of $0.011$. With the correction, I get a p-value of $0.133$. Note that, for this particular simulation, you would have gotten a smaller p-value $(0.082)$ had you just used the second sample by itself. Results may vary.
R Code
Note: Finding $d_0$ can be a challenge. In the code below, it is estimated via permutation, based on the sample (effectively some form of Bootstrapping). In reality, it is probably better to find it via Monte Carlo simulation. This requires making assumptions about the distributions of $X$ and $Y$, such as the value that $\mu_x=\mu_y$ should take via simulation.
# Simulate data
set.seed(1209102)
n1 <- 500
n2 <- 500
m1 <- 500
m2 <- 500
X1 <- rnorm(n1, 10, 1)
X2 <- rnorm(n2, 10, 1)
Y1 <- rnorm(m1, 9.9, 1)
Y2 <- rnorm(m2, 9.9, 1)
# Compute statistics
Tx1 <- sum(X1)
Tx2 <- sum(X2)
Ty1 <- sum(Y1)
Ty2 <- sum(Y2)
D1 <- Tx1/n1 - Ty1/m1
D12 <- Tx1/(n1+n2) + Tx2/(n1+n2) - Ty1/(m1+m2) - Ty2/(m1+m2)
# Get d0 threshold
M <- 100000
D1_perm <- rep(NA, M)
XY <- c(X1, Y1)
for(i in 1:M){
ind <- sample(n1+n2, n1, replace=FALSE)
D1_perm[i] <- sum(XY[ind])/n1 - sum(XY[-ind])/m1
}
d0 <- quantile(D1_perm, 0.95)
D1 - d0
# Get joint p-value
M <- 100000
cnt1 <- cnt2 <- cnt12 <- 0
XY <- c(X1, X2, Y1, Y2)
for(i in 1:M){
# Assign new groups to data
indx1 <- sample(n1+n2+m1+m2, n1, FALSE)
indx2 <- sample((1:(n1+n2+m1+m2))[-indx1], n2, FALSE)
indy1 <- sample((1:(n1+n2+m1+m2))[-c(indx1, indx2)], m1, FALSE)
indy2 <- (1:(n1+n2+m1+m2))[-c(indx1, indx2, indy1)]
D1_perm <- sum(XY[indx1])/n1 - sum(XY[indy1])/m1
D2_perm <- sum(XY[indx2])/n2 - sum(XY[indy2])/m2
D12_perm <- (sum(XY[indx1]) + sum(XY[indx2]))/(n1+n2) - (sum(XY[indy1]) + sum(XY[indy2]))/(m1+m2)
if(D1_perm > d0 & D12_perm > D12){
cnt1 <- cnt1 + 1
}
if(D2_perm > D2){
cnt2 <- cnt2 + 1
}
if(D12_perm > D12){
cnt12 <- cnt12 + 1
}
}
cat("Uncorrected p-value: ", cnt12/M,
"\nCorrected p-value: ", 20*cnt1/M,
"\np-value for 2nd sample alone: ", cnt2/M)
