7
$\begingroup$

I'm studying bioengineering but in conversations and reading I've found that a great background in information theory as it applies to probability, statistics, random process, causation and inference is really useful. Can someone recommend some textbooks that cover information theory not so much from a communications standpoint?

Thanks!

$\endgroup$
2
  • $\begingroup$ I'm sorry but this is too general for me to answer. There is a huge literature about the mathematical foundations of probability (a topic on its own), statistics (another whole topic on its own), stochastic processes (another whole topic on its own), and I'm not sure what you mean when you say "inference" and "causation". $\endgroup$
    – Tim van Beek
    Commented Aug 23, 2011 at 15:04
  • $\begingroup$ I had a feeling it would be too general. Honestly I just overhear these terms in conversation and wanted to get a better understanding for them as mathematical concepts. I'll try to narrow down my question in a future edit very soon. $\endgroup$
    – Diego
    Commented Aug 23, 2011 at 17:13

3 Answers 3

Reset to default
7
$\begingroup$

There really isn't much on information theory that isn't done from a communications perspective because that's historically how the subject has grown up.

At the intersection of information theory and biology you might try some of the following for a (gentle) broader general perspective on their interaction:

Avery, John. Information Theory and Evolution.

Yockey, H. P. Information theory, evolution and the origin of life.

Loewenstein, W. R. (1999). The Touchstone of Life: Molecular Information, Cell Communication and the Foundations of Life. Oxford University Press.

Moving more toward the technical side of information theory and likely a more sophisticated place for you to start:

Schneider, Thomas D. (2008). Information Theory Primer. (Available for download at Schneider's NIH website)

Adami, C. (2004). Information theory in molecular biology. Physics of Life Reviews, 1(1), 3-22. doi:10.1016/j.plrev.2004.01.002

And then the paper that spawned it all (you should be able to easily find it online), and which is highly readable: Shannon, C. E. (1948). The mathematical theory of communication. Bell Labs It's certainly worthwhile to purchase the UofI published version of the book with Warren Weaver's (lengthy) introduction.

Then to cap it all off, and assuming you've got some reasonable background in probability theory, random variables, and stochastic processes (a semester course will typically be sufficient) you might finally try one of the standard engineering texts:

Cover, Thomas M. and Thomas, Joy A. Elements of Information Theory, 2nd Edition (Wiley Series in Telecommunications and Signal Processing)

Keep in mind that the majority of the field is highly mathematical and there's no way of getting around it, so embrace the mathematics and delve right in! Many of the most advanced books get into analysis and measure theory, but generally with a reasonable amount of basic probability theory and a dash of combinatorics, you'll be fine.

$\endgroup$
2
$\begingroup$

"Information Theory, Inference, and Learning Algorithms" by David MacKay is very nice, & free pdf: http://www.inference.phy.cam.ac.uk/mackay/itila/book.html

$\endgroup$
2
$\begingroup$

A belated, but perhaps interesting addition. Claude Shannon, did think and write about genetics. The following link describes some of it:

Shannon's Brief Foray into Genetics

$\endgroup$

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .