Reconstruction of NMR spectrum by entropy maximization (MaxEnt)

Question

Currently NMR spectra are generated by Fourier transformation of free induction decay curves from time domain to frequency domain. In the late 60 s-80s some researchers proposed reconstruction of NMR spectra using a mathematical technique called "entropy maximization" from the FIDs. Fourier transformation was involved but only during intermediate steps. Just a note that this entropy maximization has nothing to with classical thermodynamics. It is a constraint optimization technique for reconstructing images with least amount of information. Consider entropy as just a mathematical function which shares the same name (thanks to Shannon) as the thermodynamic entropy.

Papers from the 1980s (e.g. Nature volume 311, pages446–447 (1984)) show that the appearance wise signal-to-noise ratio is higher in the MaxEnt reconstructed NMR spectrum as compared to standard FT method. However, in the Journal of Magnetic Resonance, this idea has been contested, as one cannot get something out of nothing. Some authors say, especially Hoch et al., that this is just a cosmetic exercise, because MaxEnt reduces the noise in the baseline but not in the peak. Therefore, the signal-to-noise ratio improves, but the sensitivity is no better. However, others say that MaxEnt is still very useful, and they contested that Hoch’s calculations were not realistic.

Very crudely, graphically the standard approach is:

but MaxEnt reconstructed NMR does the following magic as shown below. Since we have a couple of expert NMR spectroscopists here, my question is why MaxEnt technique never caught up in mainstream NMR? If it were so good at enhancing SNR, conventional FT should have become obsolete. I understand that it is a very involved calculation, but it looks too promising to an outsider.

Answer 1

The entropy in "maximum entropy" refers to the information entropy concept as introduced by Shannon. The MaxEnt technique referred to by Hoch may be regarded as cosmetic because what it does is rescale amplitudes in a nonlinear way. That is, big signals become bigger, small signals become smaller. Along the way the method seeks to avoid introducing spurious information (for instance peaks that might be interpreted as true signals) into a spectrum by assuming that noise is uniformly distributed across a spectrum and by penalizing excessive scaling of any potential signal.

The MaxEnt algorithm applies a "regularization" technique, also used in numerical least squares optimization to avoid issues with non-uniqueness of solutions (the "ill-defined" problem) during inversion of data. Non-uniqueness is not a problem with traditional FT-NMR because the Fourier Transform is lossless, it is a linear technique, all information originally in the time domain data is retained in the spectrum, which is one reason it is so nice, another reason including the existence of fast algorithms to compute the FT, namely the FFT, which introduced the era of modern NMR and (among other reasons) won Richard Ernst a Nobel prize.

There are a number of reasons there has been and continues to be a lot of interest in MaxEnt and related methods such as compressed sensing. These include noise reduction, but perhaps the most important reason - and importantly related to noise reduction by the $S/N \propto \sqrt{\textrm{time}}$ relation in FT-NMR - is potential savings in time. Multidimensional NMR experiments can be very time consuming. The issue of S/N that can hamper NMR can in fact be circumvented by performing long experiments, and since MD-NMR experiments are long, then low concentrations and S/N are not necessarily an issue- time is! (Sample stability of course also plays a role here). Optimization methods that combine FT with other non-linear methods can save enormous amounts of time by allowing data acquisition in the indirect dimension of MD-NMR experiments to be truncated or, even more importantly, sparsely (non-regularly or even non-linearly) sampled.

If an experiment takes on the order of days to complete using traditional NMR, but in practice S/N considerations say that you only should need a few hours, then there is a huge advantage in implementing non-linear methods that allow that. So what is the problem? The main one is probably the "black-box" approach to data processing that is implemented in many cases to reduce the complexity of applying NMR, ideally to save time and to allow researchers to focus on the question they want to answer, not on technical details such as how to implement numerical algorithms. Applying non-linear sampling methods requires additional steps to set up the experiment in advance, and to process the data and interpret the results. Those details matter if you are e.g. organic chemist looking to obtain a spectrum as quickly as possible, ideally via automation.

Answer 2

The question here seems to be:

why MaxEnt technique never caught up in mainstream NMR?

This is not really true.

MaxEnt and compressed sensing (CS, as mentioned in Nicolau Saker Neto's comment) are just different ways of reconstructing undersampled data - the method is different but the application (in the context of NMR) is very much the same. More broadly speaking, all of these are lumped under the umbrella of non-uniform sampling (NUS).

If we're talking about NUS as a whole, I would definitely refute the assertion that it hasn't caught on in mainstream NMR. NUS is very, very popular, and it's really a mainstream technique nowadays — virtually every practising spectroscopist will know of it.

Sure, it's not covered in the basic spectroscopy textbooks, but in practice many people use NUS, sometimes even without knowing (the data acquisition and processing is so streamlined nowadays that you wouldn't be able to tell unless you knew where to look). Part of the reason why the processing is so easy is because it's so popular: this creates demand for common software packages to implement it.

However, the main reason why it has caught on is not sensitivity.

One should bear in mind that NUS spectra are typically acquired in much shorter time compared to 'normal', uniformly sampled (US) spectra, as explained in Buck Thorn's answer. Even if the sensitivity (= SNR per unit time) of the NUS spectrum is larger, the fact that the NUS spectrum is acquired for such a short time means that its overall SNR will inevitably be lower than a US spectrum acquired for a longer time.

The real reason why NUS is hugely popular is time savings.

(The SNR per unit time of NUS vs 'normal' uniformly sampled spectra is quite an involved topic. I'm not an expert on it, so I will defer to the literature on this, see e.g Palmer, M. R.; Suiter, C. L.; Henry, G. E.; Rovnyak, J.; Hoch, J. C.; Polenova, T.; Rovnyak, D. Sensitivity of Nonuniform Sampling NMR. J. Phys. Chem. B 2015, 119 (22), 6502–6515. DOI: 10.1021/jp5126415.)

Answer 3

Around 15 years ago I was an intern in the NMR department at a company. Someone higher up had heard about Maximum Entropy and suggested that we try a specific program to test if it is useful for us. I ended up trying this, and in the end simply could not get that software to work. It was buggy and almost entirely undocumented. These methods also tend to have a few parameters that you are supposed to set to reasonable values, which can be rather intimidating especially when they are unclear or even contradictory on what "reasonable values" are and you have no idea what the consequences of setting these values wrong are.

Applying FFT to NMR spectra is very easy, you get a reasonable result out of the box without changing any parameters. And the parameters that you can and should change are pretty straightforward to understand if you read up a bit on them.

You have to keep in mind that many, if not most people measuring NMR spectra are not NMR experts. They're synthetic chemists or other scientists that use NMR as a tool. So to be useful to these users the process of measuring spectra has to work mostly in an automated fashion with reasonably user-friendly software to process and handle the data.

As mentioned in @orthocresol's answer NUS is a somewhat common method today that doesn't use FFT to process NMR. The difference today compared to a decade or two ago is that in newer spectrometers and software this is already integrated and is easy to setup. You could already use this method earlier, but you had to set it up with separate tools and process as well with different software. That is a very large barrier, especially if you consider that any mistake you make is likely to waste hours to days of expensive measurement time.