I think the fundamentally wrong notion implicit in most coarse-graining accounts of irreversibility is that phase space should be overlaid with a uniform lattice. That's how you end up smuggling in the irreversibility from the dynamics of the system, through Lyapunov coefficients as you alluded to. Which is neither necessary nor, as you pointed out, sufficient for the argument.
The actual partitioning of the phase space of a large thermodynamic system is defined by its macrostates, that is by macroscopically defined and measurable sets of properties such as temperature and pressure. That partitioning is far from uniform, and in fact some macrostatates occupy exponentially larger volumes than others. That means that whatever the underlying fundamental dynamics of your physical system are, as long as they are Liouvillian, transitioning to a large volume macrostate from a smaller one means you lose the capability of tracing back its trajectory. Hence, irreversibility.
You might say that begs the question, because the burden is kicked upwards to the choice of macrovariables defining the macrostates that partition phase space the way they do. And as far as I know the answer is philosophically similar to the ones for the "measurement problem" in quantum mechanics: we choose the macrovariables to describe macroscopic objects just like we choose in which basis to measure an electron's spin.
This is by the way not a novel take. It was essentially laid out more than half a century ago by E. T. Jaynes (http://bayes.wustl.edu/etj/articles/gibbs.vs.boltzmann.pdf) and also more recently by J. Bricmont, and I'm sure by many others too. But somehow this kind of reasoning ends up being resisted in at least a big chunk of the statistical mechanics community, probably for philosophical intuitions against "subjectivity" or "anthropomorphizing" of physical concepts such as entropy.
