Adding to group layers in loop is slow in ArcGIS Engine?

Question

I am using ArcGIS Engine 10.5.1.

I wanted to ask and see if anyone else has encountered this and or found a work around. I have a loop in c# where I am taking a large set of layer files and adding them to the map. I have tried this two ways but both ways seem to be equally slow (~10 minutes for ~180 layers). The contents of the layers files are just shape objects. Not a lot of data. The first way I tried to achieve this is

IGroupLayer groupLayer = new GroupLayer();
foreach(SortedListLayers layer in MySortedListLayers)
{
    ILayerFile layerfile = new LayerFile();
    layerfile.Open(layer.location);
    groupLayer.Add(layerfile.Layer);
    layerfile.Close();
}

On the groupLayer.Add line there is more and more overhead each time the call is made to complete adding the layerfile.Layer to the group even though the group is not yet on the map. My alternative was to use IMapLayers and implement IEnumLayers on my SortedListLayers object (which only contains a sorted list of 'layers' and the location that it belongs to) and use the maplayers.InsertLayersInGroup call to try to diminish looping but that turned out even slower.