20
$\begingroup$

I'm currently looking to hone a system using market depth however I am looking for a good source of historical level 2 data.

As of right now the best source of historical I have found is automated trader, however at 6 grand for 1 year I need to make some more money before that happens. The other vendor I'm in talks with is dxfeed, however I'm not sure if they offer level 2 historical data.

For my platform I use DTN IQ feeding into matlab which sends trades to IB. DTN IQ does not provide historical level 2 data.

If I can't find a good source I'm instead going to test by paper trading the same system on the current market to hone in a strategy.

Any other thoughts or opinions on how to get access to historical level 2 would be much appreciated.

Improve this question
$\endgroup$
6
  • 3
    $\begingroup$ Just buy the historical data directly from the exchange. Probably not the answer you want, but it's the only one that's realistic. $\endgroup$
    – chrisaycock
    Commented Mar 15, 2012 at 15:22
  • $\begingroup$ For which asset class? If futures, take a look at CQG Data Factory . $\endgroup$
    – michaelv2
    Commented Mar 15, 2012 at 17:05
  • 1
    $\begingroup$ I wonder how automatedtrader.net is able to sell native exchange data. Each exchange sells its own historical data and usually they are very restrictive about the distribution rights. You might want to compare those 3rd party prices against the exchange offer. $\endgroup$
    – Robert Kubrick
    Commented Mar 15, 2012 at 17:35
  • $\begingroup$ Exchanges sell through third parties. I have used trading physics which had more competitive prices. $\endgroup$
    – Steve Severance
    Commented Mar 17, 2012 at 0:44
  • $\begingroup$ Dylan or anyione else How did he manage to connect IQFeeds with Matlab. I've ben looking all over the place for a Matlab toobox. As far as I'm aware Matlab's DataFeed toolbox only allows for Reuters, BLoomber, Yahoo and eSingals but not IQFeeds and they say will only be avaiable in Q4 2012 in version 2012b. Anyone know anything about this? thxs Jake $\endgroup$
    – user2643
    Commented Jul 3, 2012 at 22:30

6 Answers 6

Reset to default
5
$\begingroup$

Collect the data yourself and self-timestamp it or buy it from a vendor that timestamps on receipt.

If you do buy data from the venues you should consider very strongly the fact that venue timestamps are never in agreement. Therefore, an event that INET reports at time A does not coincide with an event that BZX reports at time A. This makes buying this sort of data somewhat hazardous. You are better of collecting it yourself or buying it from a vendor who has collected it and self-timestamped the data on receipt.

Improve this answer
$\endgroup$
3
  • 1
    $\begingroup$ Collecting the data yourself also means decoding the native exchange formats. Depending how many exchanges you need, this goes a long way from buying normalized data from vendors. $\endgroup$
    – Robert Kubrick
    Commented Mar 19, 2012 at 21:44
  • 2
    $\begingroup$ Of course it means writing feed handlers for each, that was implied. However, you can have accurate data or you can have crap from a vendor. You pick. Relying on venue specific time stamps will result in crap data. $\endgroup$
    – Louis Marascio
    Commented Mar 19, 2012 at 23:09
  • $\begingroup$ @LouisMarascio very true regarding crap data $\endgroup$
    – user997112
    Commented Jun 30, 2019 at 18:22
3
$\begingroup$

I use interactive brokers as well. What I do is I collect the data throughout the day with their API (reqMktDepth functionality) and I'm building my personal data warehouse. It may seem silly but I've been doing it for two years now and I have a good dataset to play with. Just get started and you'll see that it makes sense, the nice thing about it is that you only download whatever you really need and dont pay for the rest (actually you don't pay anything, period)

Improve this answer
$\endgroup$
2
  • $\begingroup$ What if you want to investigate a new strategy and the data you've downloaded isn't sufficient? $\endgroup$
    – chrisaycock
    Commented Mar 15, 2012 at 16:27
  • $\begingroup$ The data you download from interactive brokers is standardized, you have a set of requests and you can only ask those requests, so in that sense the data I download is always sufficient (cause it's all there is, nothing more nothing less). What can be insufficient is which underlyings (i.e. contracts) you need for your strategy, they might not be in the set of contracts you have been downloading so far, that's true. $\endgroup$
    – mepuzza
    Commented Mar 16, 2012 at 8:54
3
$\begingroup$

dxFeed does offer level 2 historical data on demand. You can try it out for free. For example, to see how all market makers pulled out their quotes on ACN (Accenture) during May 6, 2010 Flash Crash go to the following URL: http://demo.dxfeed.com:7070/onDemand/data?categories=E-M&symbols=ACN&start=20100506-1447&end=20100506-1449 and use demo/demo when prompted to login. Add "&format=csv" at the end to download the data in Excel-friendly comma-separated values format. Demo access it limited to this particular day and to 1 second conflation interval.

Unless you host your own servers close to exchange and engage in nanosecond-level HFT, you don't have to build your own data warehouse. You can request the data you need from a vendor with adequate precision even for strategies on millisecond scale.

Improve this answer
$\endgroup$
1
$\begingroup$

You are not clear if you mean Level 2 data as in top-of-book for each exchange showing the Bid/Ask from exchange or each exchange's direct feed with full depth.

The exchange full depth direct feeds have become very expensive in last few years as exchanges increase revenue from them, for example NYSE OpenBook is now approx 6,000 per month and Nasdaq Itch is $3,500 per month.

Here are some vendors (in alphabetical order) who can help:

AlgoSeek.com : Has years of consolidated feed (aka the TAQ, Consolidated feed) showing Bid/Ask from each exchange. Also has direct feed full depth from ARCA, BATS, EDGEX/A, Nasdaq. Reasonably priced as these things go.

dxFeed.com : See posting above.

Interactive Brokers. As mentioned above, you can collect data through IB but be aware that it is snapshot data taken every 100 milliseconds (every 1/10th of a second) so you do not get a real accurate dataset.

Nanex.net : Has been collecting and analyzing data for many years now and they will sell it. Published pricing is about $600 to buy one month of data, but will negioate.

QuantGo.com : A clould platform where clients can rent access to years of historical tickd data from multiple data vendors. Most affordable solution (eg. $275 per month to access all 6 years of Equities trades and quotes) BUT you have to use your own comptuer instance in the Amazon AWS cloud. You get your own computer in the cloud...it is how they ensure the data can be rented.

TickData.com : Have all the consolidated data with Bid/Ask from each exchange. Great data quality. You can buy just the symbols you need or the whole universe. Do not have full depth data. More expensive than other vendors but great quality.

Improve this answer
$\endgroup$
0
$\begingroup$

I agree with @Louis Marascio and @chrisaycock, data is expensive! Going directly to the exchanges is the safest way to obtain huge blocks of historical data, and real-time sources.

If $6000 for a year is too much, than your not going to be able to afford the exchange's data sources, than your not going to be able to afford a big chunk of historical data.

Louis's point about generating your own database by either downloading it from your broker is a good one. Or consider subscribing to a more cost-effective service ($60-$100 a month) and using that to build your database.

Improve this answer
$\endgroup$
0
$\begingroup$

Agreed, it's usually best to build your own database. Data is really expensive, especially level 2 (due to all the licensing/fees).

Worth noting https://polygon.io as an end user/trader provider (couple hundred dollars/m).

If you're wanting direct exchange data feeds check out http://www.ctaplan.com or http://utpplan.com. Those will usually run a minimum of a couple thousand $ per month.

Improve this answer
$\endgroup$

Not the answer you're looking for? Browse other questions tagged or ask your own question.