Efficiently Visualising Very Large Data Sets (without running out of memory)

Question

I have put a few really hard problems in combinatorics up against Mathematica 8. I'd have to say that it works really well, until you want to view the data. If you look at my question Advanced Tupling there is an elegant solution. One line function. Data of about 12 million sets given the sample data presented.

Let's use that question to illustrate. Here are the data:

a = {1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 17, 18, 21, 22, 23, 25, 26, 28}; 
b = {2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 25, 26, 27, 28, 30, 31, 33, 37, 41}; 
c = {6, 10, 13, 14, 15, 16, 18, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 32, 35, 37, 39};  
d = {17, 19, 25, 30, 31, 33, 34, 35, 36, 38, 44}; 
e = {31, 41, 45, 47}; 
f = {23, 26, 31, 32, 33, 34, 35, 36, 38, 39, 40, 41, 42, 43, 45, 46, 47, 48, 49, 50, 51, 52, 53}; 

And here is one of the solutions posted in that question:

newTuples[t_, x_] := 
  Flatten[ ParallelTable[Append[s, #] & /@ Complement[x, s], {s, t}], 1]; 

One would however like to verify the results. From everything seen, it looks good. But one wants to see it all. Click 'Show All' output and the program grabs all available memory resources, including consuming all available disk space for page file and still takes literally days to complete. Is there an efficient way to render data?

I have had the same problem with graphical plots, it just chokes up after processing. It is as if the entire problem has to be reprocessed for it to render. Surely this cannot be how Mathematica was intended to work. I must be missing something.

What are some memory-efficient ways to visualize a data set as large as the output of the above function?

Answer 1

Showing humongous data by screen-fulls can be done using Manipulateas follows:

(* generate some data to show *)
res = Tuples[{1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12}, 7];

(* size of the data*)
ByteCount[res]

(* ==> 1003290792 *)

screenNumbers = 100;
Manipulate[
 Take[res, {i screenNumbers + 1, (i + 1) screenNumbers}], {i, 0, 
  Length[res]/screenNumbers - 1, 1}
]

Since the slider slides over 100,000's of values in a short distance you can slow down its movement using the modifier keys when you drag the slider.

The resulting slider can be finely manipulated by holding down the Alt key (or Option on Macintosh) while dragging the mouse. This causes the slider to move at 1/20 the rate of the mouse. The slider can be even more finely manipulated by also holding the Shift and/or Ctrl keys. [Last bullet of the More Information part of the Slider doc page]

Scrolling through the 1GB of data is almost instantaneous.

Answer 2

The difficulty lies in the rendering process used by the Front-end. If you have a large data-set, then it is entirely possible to lock up your machine when you try to show it all. While a visual inspection of such data sets would be nice, it is likely that it would be useless as you could unintentionally skip over data that is incorrect. I would suggest an automated approach instead.

The simplest test is to determine if each element of the tuple is a member of the correct distribution. To do that simply,

Select[data, 
   Function[{f}, 
    ! And @@ MapIndexed[
       With[{dist = #, idx = #2}, MemberQ[dist, f[[ idx[[1]] ]]] ] &, 
       {a, b, c, d, e, f}]]
]

which will return any data that does not conform. If you just wish to strip out any such data, instead, just remove the ! in the above function.

Answer 3

I deal with lots of text as well, and this is going to be a serious problem. In short, the comments are correct that you probably won't be able to just throw 2GB of information up there on your screen. One function that a person mentoring me in Mathematica provided was this:

viewData[x_] :=
 Framed[Pane[x, {Automatic, 200}, Scrollbars -> True]]

You could then ViewData[x] where x is the information you're trying to visualize. You still won't be able to view 2GB of data all at once, but this might give you an ability to view a bit more of it for whatever reason.

There have been some discussions over at Stackoverflow on the related problem of viewing large files, if you were to export it textually, including some text editor suggestions (here and here).

Answer 4

The solution to this is after processing, write a text stream to a text file, open it with a text editor capable of handling large files. I used UltraEdit. This example created a ~319MB text file.

fname = ["C:/YourDir/Yourfile.txt"];
stream = Openwrite[fname];
Write[stream, ts];
Close [stream];

This took about the same time to write out as the original tupling solution took to process.

I think the problem is kind of like when Windows 95 was released. Sure you could play DOOM 2 in windowed mode, but the lag was atrocious and the frame rate was unacceptable. Pop back into full screen mode and voila you were good!

So, it appears to be a similar problem with mma that it performs some kind of a formatted rendering that could be dolled up with proper adjectives on the commands. Would be nice just to have raw unformatted, good old fashioned text output as an option, but I'm not aware of any such facility in mma.