I think that the sequence beginning 一番・二番・三番
corresponds roughly to number one, number two, number three. Each word in this sequence is formed from the combination of a numeral (一・二・三
) with a counter meaning number (番
).
You can take this list and add the ordinal suffix 目
to each word, producing the list 一番目・二番目・三番目
, corresponding roughly to first, second, third.
What may be confusing here is that an adverb has been derived from 一番
, but not from the other words formed with 番
. As a result, 一番
can represent either number one or first/most, but 二番
and 三番
mean number two and number three. You can see this if you compare dictionary entries; the entries for 一番 in 大辞林 and 大辞泉 list it as both a noun and an adverb, while the corresponding entries for 二番 don't list a derived adverbial sense.
So, if you're talking about the tallest, second tallest, and third tallest buildings, I think the list naturally looks like this:
And your example 二番速い人
is, I think, ungrammatical.
As for whether 〜目
is a counter, I don't think so. It doesn't combine directly with numbers like counters do. Instead, it's a suffix that attaches to words that consist of a number and a counter. If we check 大辞林, we find the following:
二 ( 接尾 )
① 数を表す語に付いて,順序を表す。 「一つ目」 「三番目」
You'll notice that it says this is a 接尾語 (suffix). I've bolded the actual definition, which I'll translate here as "attaches to words that express number, and expresses ordinality."
You'll note the definition includes an example where 目
attaches to a word with つ
, as well as another example where it attaches to a word with 番
. With 目
attached, both 一つ目・二つ目・三つ目
and 一番目・二番目・三番目
express first, second, third, but of course the つ
list can only count to nine, while the 番
list can be used to express arbitrary positions in a list.