This somewhat resembles Mahou Sensou, specifically around Episode 9 or 10. The series aired in 2014 and revolves around two competing magical factions - Wizard Brace, the protagonists, and the Ghost Trailers, the antagonists.
There were 2 magician brothers who were fighting each other in a subway. One of them had something like a big bastard sword which had a gun at the bottom and one of them could do purple-colored magic? (The things that appeared in the air before the magic happens were purple magical writings.) One of those brothers had brown hair and their family was messed up. At the end of the fight, one of the brothers leaves.
One episode features a fight between the protagonist, Takeshi, and his brother Gekkou, who hates him due to a childhood incident. (Said incident also clouded their relationship with their mother, which fits with the "messed-up family" you mentioned.) Takeshi fights while wielding a sentient sword named Twilight; Gekkou fights with pure magic. The fight ends with Gekkou defeating Takeshi and walking away. What differs from your recollection is that both brothers have black hair, and the fight takes place in the corridors of the magic school Takeshi attends, not a subway.
I do not remember anything else about the episode except the end of it: There was a broken wooden church and there were a lot of people in black clothes. All of them were killed by a blonde woman
A significant plot point in the series is a magical barrier, sustained by a group of monks, that separates the magical and non-magical worlds, and curses anyone who uses magic against a non-magic user. One episode ends with the monks having all been massacred by a female Ghost Trailer, removing the barrier and allowing the Trailers to invade the non-magical world.
As a footnote: regardless of whether Mahou Sensou is the correct answer or not, I must advise against watching it because it is extremely badly-written.
