Grammar Exists to make Complex Speech Possible.
We use grammar as a way of creating relationships between the words in a sentence, and for expressing additional information without needing to use additional words. So lets pretend you tried to say "He has a daughter with a yellow hat. She went to school with it.".
Let's see what happens if you take away the gender rules:
It has a child with a yellow hat. It went to school with it.
Okay, so we are now less precise, but still readable. But pronouns are also a grammar rule; so, let's get ride of those too.
Johnathan has a child with a yellow hat. Elizabeth went to school with the yellow hat.
This actually made the sentence more clear, but longer; so, we can see grammar sometimes intentionally trades off clarity in favor of brevity, but we are still just getting started. Now let's take away those annoying prepositions and articles we use for establishing relationships between other words.
Johnathan has child yellow hat. Elizabeth went school yellow hat.
Congregation is also annoying, let's get ride of that too.
Johnathan has child yellow hat. Elizabeth go school yellow hat.
So, we've actually stripped away a lot of grammar rules, and still have a pretty readable sentence here. Sure, there is room for confusion, but we can work with this as long as we have a little context. The big reason we can still work with it is that if follows a familiar word order telling us a lot about how these words relate to one another. But when you remove familiar word order you get something more like:
Child Johnathan yellow has hat. Hat Elizabeth go yellow school.
Oh, and let's not forget, the very idea of a sentence is a grammar rule, so why not just say:
school has child yellow has hat hat Elizabeth go yellow Johnathan
So, as you can see, without the structure of grammar, any random arrangement of words would be equally valid meaning there is no way to compose words to be more complex than individual ideas.
Basically, the only way to really get away from grammar is to restrict a language to each word expressing a complete, uncomposed thought. Toddlers (and many animals) communicate this way. A toddler can say "milk" when they mean that they want milk. They can say "up" when they mean that they want you to pick them up. But you can not build a civilization on this kind of language. Part of using a composed language system means that we can explain words that other people do not already know. So, if we had a grammarless language with billions of words to express each idea we could possibly have, teaching it would be impossible because the sentence structure "A is like B but C" can not exist without grammar. You also can't short-hand learn anything with conjugation where you know "picked" is like "pick", but in the past because they have the same root with a known suffix.