I'll add some 'progress so far', ie what was posted in September 2000 by Jim Gillogly.
Doing an index of coincidence at various periods shows that it has some kind of periodic behavior at offsets of 7 and 8 bytes -- whatever breaking-up of characters is going on appears to come back into focus then. Here are the periods with highest IC's up to about period 120:
56 0.0226 112 0.0216 28 0.0134 84 0.0118 98 0.0098 14 0.0083 70 0.0079 42 0.0073 16 0.0070 120 0.0069 49 0.0069 8 0.0069 104 0.0068 80 0.0067 40 0.0066
All values with factors of 7 or 8 or both.
This effect is confirmed by a Kasiski analysis. Here are the longest repeated strings:
4 at 52, offset 672: 4ddb0f63 4 at 197, offset 784: 6153b403 6 at 219, offset 392: f7d0aea35d74 3 at 235, offset 784: e55e73 3 at 389, offset 616: 707367 3 at 407, offset 672: d50120
And the factorization of their offsets: 392: 2 2 2 7 7 672: 2 2 2 2 2 3 7 784: 2 2 2 2 7 7 616: 2 2 2 7 11 672: 2 2 2 2 2 3 7
All 7's and 8's. Perhaps the basic operation is sending ASCII with the high-bit trimmed off, but doing something additional when the "special" bytes are encountered on output. This might show the kind of periodicity I'm seeing.
It might be instructive to re-do the IC and Kasiski analysis on a bit level rather than a byte level. I suspect we'd see longer repeated sequences (by which I mean longer than 6*8 bits).
Jim Gillogly Hevensday, 21 Halimath S.R. 2000, 16:35 12.19.7.9.15, 4 Men 18 Mol, Sixth Lord of Night