He's counting on you to avoid confrontation. And you feel that saying something more forcefully will make things uncomfortable and be rude. But here's the thing: he's already being rude, and things are already uncomfortable, at least for you.

So, in addition to writing things down after they happen each time, start speaking up anyway. When he touches you, say "Please don't touch me." Get louder each time. It will become easier as you do it, and enforcing boundaries is a useful skill.

When he asks you out, say "I'm not interested in going out with you. Please quit asking." If he asks again, say "I said I wasn't interested, why are you still asking?" "I need you to quit asking me out, can you do that?" And you can get louder there, too.

Eventually, it will become more uncomfortable for him than it is for you, and he will stop. Or else the boss will start seeing and hearing the problem. And when the boss finally takes notice, you'll have your log, if he's interested.

If you have an HR, it's worth going to them and explaining that you've told him to stop, and he won't.

As BradC says in the comments "to be absolutely clear, there is zero chance he doesn't know exactly what he's doing. The reason to be clear and loud, as this answer recommends, is not to "make sure he understands", the reason is to remove his plausible deniability, and to make sure that everyone else in the office also sees what is going on." Yes, start making it more uncomfortable for him, start making it really visible to others what he is doing. He already knows what he is doing, and will continue as long as only you are uncomfortable.

He's counting on you to avoid confrontation. And you feel that saying something more forcefully will make things uncomfortable and be rude. But here's the thing: he's already being rude, and things are already uncomfortable, at least for you.

So, in addition to writing things down after they happen each time, start speaking up anyway. When he touches you, say "Please don't touch me." Get louder each time. It will become easier as you do it, and enforcing boundaries is a useful skill.

When he asks you out, say "I'm not interested in going out with you. Please quit asking." If he asks again, say "I said I wasn't interested, why are you still asking?" "I need you to quit asking me out, can you do that?" And you can get louder there, too.

Eventually, it will become more uncomfortable for him than it is for you, and he will stop. Or else the boss will start seeing and hearing the problem. And when the boss finally takes notice, you'll have your log, if he's interested.

If you have an HR, it's worth going to them and explaining that you've told him to stop, and he won't.

As BradC says in the comments "to be absolutely clear, there is zero chance he doesn't know exactly what he's doing. The reason to be clear and loud, as this answer recommends, is not to "make sure he understands", the reason is to remove his plausible deniability, and to make sure that everyone else in the office also sees what is going on." Yes, start making it more uncomfortable for him, start making it really visible to others what he is doing. He already knows what he is doing, and will continue as long as only you are uncomfortable.

Edit, based up updates to question If he's touching you and not backing down, then you need to go to HR and your boss now. (Which you have.) Don't arrive early when he might be the only one there. Don't work late. Make sure, as far as you can, that he never has a chance to be alone with you. Working from home is a good start. (If it hadn't escalated as much as it has, I'd suggest that when he's around, make your documentation a lot more obvious: pull out your phone, and tell him that you'll be taking a video of your immediate area, and if he moves into it, he will be in the video. But don't ever be alone in the building with him now, even with the camera phone - that's not enough to protect you, and it might make it worse. You don't want evidence after something happens, you want nothing to happen.)

If he isn't completely professional when you return to the work building, you should consider finding a new job and getting a restraining order, and ask to work from home until that happens. (You might mention to your boss that you're considering a restraining order, and ask if he'd like to handle it differently than you working from home.)

He's counting on you to avoid confrontation. And you feel that saying something more forcefully will make things uncomfortable and be rude. But here's the thing: he's already being rude, and things are already uncomfortable, at least for you.

So, in addition to writing things down after they happen each time, start speaking up anyway. When he touches you, say "Please don't touch me." Get louder each time. It will become easier as you do it, and enforcing boundaries is a useful skill.

When he asks you out, say "I'm not interested in going out with you. Please quit asking." If he asks again, say "I said I wasn't interested, why are you still asking?" "I need you to quit asking me out, can you do that?" And you can get louder there, too.

Eventually, it will become more uncomfortable for him than it is for you, and he will stop. Or else the boss will start seeing and hearing the problem. And when the boss finally takes notice, you'll have your log, if he's interested.

If you have an HR, it's worth going to them and explaining that you've told him to stop, and he won't.

He's counting on you to avoid confrontation. And you feel that saying something more forcefully will make things uncomfortable and be rude. But here's the thing: he's already being rude, and things are already uncomfortable, at least for you.

So, in addition to writing things down after they happen each time, start speaking up anyway. When he touches you, say "Please don't touch me." Get louder each time. It will become easier as you do it, and enforcing boundaries is a useful skill.

When he asks you out, say "I'm not interested in going out with you. Please quit asking." If he asks again, say "I said I wasn't interested, why are you still asking?" "I need you to quit asking me out, can you do that?" And you can get louder there, too.

Eventually, it will become more uncomfortable for him than it is for you, and he will stop. Or else the boss will start seeing and hearing the problem. And when the boss finally takes notice, you'll have your log, if he's interested.

If you have an HR, it's worth going to them and explaining that you've told him to stop, and he won't.

As BradC says in the comments "to be absolutely clear, there is zero chance he doesn't know exactly what he's doing. The reason to be clear and loud, as this answer recommends, is not to "make sure he understands", the reason is to remove his plausible deniability, and to make sure that everyone else in the office also sees what is going on." Yes, start making it more uncomfortable for him, start making it really visible to others what he is doing. He already knows what he is doing, and will continue as long as only you are uncomfortable.