Call them out when you notice it. Stop what you're doing - single the chit-chatters out, and ask them if they have anything they'd like to say. If they have questions they should ask them so that the entire class can hear - that way everyone can learn. Otherwise If they do have questions, remind them of whatever protocol you have for questions (raise a hand, hold questions for designated question breaks, etc).
Otherwise, make it clear that you won't tolerate chit-chat and, if they persist, simply kickremove them out offrom the class. Make it clear that they are welcome to return to the next lecture on the condition that they keep quiet during the lecture. Be sure to be polite and cordial, of course, but don't waver on being firm.
It is not acceptable for a few noisy students to compromise the learning environment for everyone else. Just get them out of there. You probably won't have to do this more than once or twice before the rest figure out that you're serious. As a lecturer, you're the captain of the ship. Don't be afraid to be the captain - keeping order is your job. The quiet students are relying on you.
Also keep in mind that it is not your job to make sure that they pay attention - that's their job. "Listen to me" should not be your responsibility, it should be theirs. Your responsibility is primarily "Don't disrupt my lecture! Others are trying to listen to me". If they would like to quietly not pay attention, that's fine. They're paying for a seat in the lecture hall - what they do with it is up to them, so long as it does not affect the learning environment for everyone else (who are also paying for their seat in that lecture hall!).