They didn't notice the invasion because Arrakis has no satellites for reasons that will become clear outside the scope of this film (or in the book). They also had no comms due to Yueh's sabotage. Also, the Spacing Guild transports move faster-than-light (space-folding), so the enemy ships effectively appeared directly in orbit.
This is a detailed response to a secondary question that OP has raised a few times in the comments: "Why would the Duke accept the move to Arrakis if he knew it was so dangerous?"
Just wanted to address OPs skepticism around the Atreides family accepting the move to Arrakis. The politics of the Dune universe are huge and complicated and it's all pretty skimmed over in the film due to time constraints, so here's a quick (simplified) run down. I'll try to keep this spoiler free as far as the first half of the book (in line with the first film), but it's all a pretty expansive and tangled web so sorry if I touch on anything that isn't mentioned
Basically there's 3 major players:
The Spacing Guild
A completely neutral force that has an absolute monopoly on interstellar travel. They use massive quantities of spice to artificially evolve humans into superhuman computers called Navigators, who are the only people capable of plotting safe travel through the stars. The spice exposure also give Navigators a small amount of prescience (no magic, just extreme statistical analysis). They basically provide the service of interstellar travel to the Imperium and the Landsraad at an exorbitant fee. No one messes with them for fear of being denied interstellar travel and effectively cut off from the rest of the known universe.
Key points:
- Technically neutral
- People pay them for interstellar transport
- Heavily reliant on spice to maintain their power
- Some level of prescience
- No one dares to mess with them
The Landsraad
All the Great Houses (including Atreides and Harkonnen). There are 30+ Great Houses and they all have their own alliances/enemies amongst themselves. Skirmishes between the houses over resources etc are common, but outright war isn't, due to the complexity of the politics, the massive cost of transporting armies through the Spacing Guild and the fact that the Imperium wouldn't allow it (if houses are busy having full scale war, they aren't busy paying taxes.). The Landsraad also act as a sort of "council" in that, together they are more powerful than the Imperium, so they are both ruled by and hold power over the Imperium. No one house or small group of houses would dare upset the Imperium. If the Emperor says jump, a House basically has to say "how high?".
In the past, houses have contracted with the Spacing Guild to take them somewhere secret, out of reach of the Imperium when they have felt like they had no other option but, of course, this means cutting yourself off from the rest of the universe. This is called "going renegade" and is about the worst possible option. It's considered a disgrace and is effectively admitting defeat.
Key points:
- 30+ Houses. Lots of internal politics
- Can't do interstellar travel without the Spacing Guild
- Individually much weaker than the Imperium
- Technically always have the option of "going renegade", but this is basically giving up everything you have in defeat and exile. Not exactly desirable.
The Imperium
Rules over the houses of the Landsraad and effectively takes taxes and tells them what to do. Y'know, emperor stuff. The Imperium benefits if all the great houses are more or less getting on with business and paying taxes, so has a vested interest in keeping them in line and not at each others' throats. Has a big scary army of very elite soldiers called Sardaukar that are occasionally used as a "peacekeeping" force and just generally as a threat to the Great Houses to not get above their station. Generally has the allegiance of the Great Houses (as much as any Emperor ever does), so an offence to the Imperium is also kind of an offence to the other Great Houses (even if only because they are scared to not take the Imperium's side.)
Although very powerful, the Emperor still needs to be be political. Can't afford to offend too many of the Great Houses at once for fear of a revolt. Can't afford to offend the Spacing guild for fear of being cut off.
Key points:
- Very powerful.
- Generally has the allegiance of the Great houses
- Still subject to the movements of politics
Some important things to know about the Emperor/Imperium/Atreides
The Imperium and the Great Houses are not actually really separate entities. Basically, one of the House Leaders sits on the throne and that house is called The Imperium. If any of the other house leaders want to be the Emperor, really they just have to shove the current Emperor off the throne, sit down and not get shoved off themselves. Obviously, politically, this is a very difficult trick to pull off. Not only would your house have to survive a war against the Imperium, their Sardaukar and all their allies, but you'd also have to be strong enough at the end of that war to be able to defend it. Basically this isn't something that happens often.
The current Emperor (at time of the events in Dune) is the head of House Corrino. As it happens, Duke Leto Atreides was the Emperor's cousin, so he was technically of noble birth. He was also very popular within the Landsraad and widely recognised as a strong and fair leader. He functioned as the de-facto spokesperson for the Landsraad and is growing in power and influence over the Landsraad.
So why the House Atreides agree to go to Arrakis?
The Emperor mandated that House Harkonnen had controlled Arrakis for long enough and it was now someone else's turn. He chose House Atreides. Obviously, from the Duke's point of view, this was an extremely dangerous political position to be in. Atreides and Harkonnen were old enemies and he knew they wouldn't take kindly to the change of hands of the most valuable resource in the universe. He fully expected hostile actions from House Harkonnen but, as far as he was concerned, House Harkonnen were also being told what to do by the Emperor and wouldn't dare commit a full assault against the Emperor's command. So being given the fief of Arrakis by the Emperor was a massive opportunity for House Atreides to become incredibly rich and the only risk was a few skirmishes with the Harkonnens that he was confident he could deal with.
So the Duke has 3 options:
- Do what the Emperor says, take control of the most important planet in the known universe, become incredibly rich and deal with a little more aggression from a house you've already been warring with for ages.
- Deny the Emperor (because reasons?) and face his (and all his allies, including the Harkonnen's) wrath and definitely be completely destroyed.
- Give up everything you have to pay the space guild to move what's left of your house the the end of nowhere to live in squalor forever.
Which would you choose?
Ok, so if it was such a good deal, why did it go so wrong?
Right so this is where a lot of intrigue comes in that was necessarily cut from the films. As mentioned before, Duke Leto was both of noble birth and very popular with the Landsraad, so not only could he potentially be a danger to overthrow the Emperor with the support of much of the Landsraad, he could even do it with a legitimate claim to the bloodline. The Emperor is paranoid that his house is weakening, that Atreides will eventually make a move for the throne. He wants Atreides out of the picture, but needs to find a way to do it without upsetting the Landsraad.
The Harkonnens have been at war with the Atreides for a long time. The reasons are a little beyond the scope of this (already very long) post. Basically the Harkonnens would jump at the chance to wipe the Atreides off the map for good.
They conspire together to hand Arrakis over to House Atreides (which in itself isn't particularly unusual), then the Emperor bolsters the Harkonnen forces with Sardaukar and basically turns a blind eye to what happens next. Now that the Harkonnens have an enormous army and the green light from the Emperor, they throw an unheard of amount of money at the Guild to transport the force and destroy House Atreides. Even the Guild are happy to play their part. Because they have some level of prescience they have foreseen the threat that Paul Atreides having control over Arrakis could present to them (not to give anything away...) They want him out of the picture, so they are happy to turn a blind eye to the happenings on Arrakis.
The Harkonnens are happy because House Atreides is gone and they get Arrakis back. The Emperor is happy because House Atreides is gone and he can write it off as a transition scuffle that got out of hand on a remote planet with no witnesses. The Guild are happy because Paul will be killed as part of the assault. Everyone is happy. Except House Atreides...
Duke Leto Atreides had no way of knowing that the Imperium and the Harkonnens were conspiring. As far as he was concerned, the Imperium would afford him some political protection during the transition, if anything.
Edit: I actually vaguely remember Duke Leto mentioning that he thinks it's possible the Emperor is conspiring with the Harkonnens and that the move to Arrakis could be part of an assassination plot, but he trusts Hawat to be able to handle it. He never considers anything so blatant as a full on assault with Sardaukar.
There's also a bunch more about how the Bene Gesserit and the Spacing Guild play into all of this, but, yeah... Basically this seemed like a pretty normal exchange of fiefdom that happens every so often as standard practice. Duke Leto had no reason to think anything out of the ordinary was happening and was possibly slightly blinded to any conceivable risks by the massive opportunity Arrakis presented. He had no reason to say no, but even if he had, saying no would mean certain destruction anyway.