I wouldn't shoot this down so quickly - consider, e.g. Mulliken electronegativity, defined as the average of the electron affinity and ionisation energy. If you think about it, electronegativity depends not only on how strongly an atom pulls other electrons to itself, but also how willing it is to give up its own electrons. The former can be somewhat represented by the EA, the latter by the IE.
One point that hasn't been mentioned is that electronegativity is not a physical parameter which can be measured. It is a human construct, which is often helpful in explaining chemical reactivity, but at the end of the day it is not something which can be derived from physics. Therefore, there is no real reason to expect that there should be any one-to-one correlation between the EN and the EA (unless the EN scale is explicitly defined to follow the EA). For example, if you use the more common Pauling scale (defined by bond dissociation energies), there's no real reason why the trend in EN should match the trend in EA exactly.
Broadly speaking, the trends match (because of what you've said) - e.g. both EN and EA generally increase going across a period. However, the small details like chlorine vs. fluorine won't necessarily match.