This seems to work:
1. Nf3 e5
2. Nxe5 Ne7
3. Nxd7 Nec6
4. Nxb8 Nxb8
And the position looks like this:
Apart from the symmetrical solution, this might very well be unique:
For starters, white's moves 2, 3 and 4 must be captures. This greatly limits the available options and brings the problem within the limits of solvability.
White cannot move a pawn: if he does, it must get captured. This means that all the captured pieces must be pawns. This means that the white pawn must be the piece doing all white's capturing: any other piece doesn't have the time to move back to its starting position, and as noted earlier, all the captured pieces must be pawns in this case.
From there, we can deduce that if white starts with a pawn, white's fourth move must be exactly "pawn captures pawn on the seventh rank", and of course there is no way for black to capture it back with a pawn.
Because white cannot move a pawn, the knights are the only option. Since white's fourth move must be "knight captures something", there is no way to for a black piece to take it and return to its starting position. Also, since black takes the white knight, white must also take a knight. Therefore, black's fourth move must be "black knight takes the white knight at the other black knight's starting square", and there seems to be exactly two ways to achieve that if white must take pawns on the intervening two moves. (The other solution starts with the other white knight and is symmetrical.)