A colony was the term for a group of insects, such as ants, that lived together, whether naturally or in captivity. Insect colonies were similar in concept to hives, in that both contained queens. (VOY: "Muse")
In 2269, Spock surmised that the Lactrans were as advanced a civilization compared to Humans as Humans were to a colony of ants. (TAS: "The Eye of the Beholder")
In 2376, Mezoti's science project for the Annual Voyager Science Fair was a Teirenian ant colony. She had infused the soil with a blue ion dye to make the ants more visible. (VOY: "Child's Play")