I believe that the classic argument that conceptual regress goes back forever may be wrong.
For instance, if I try to infinitely regress on concepts, I actually end up at a point where I can't go on much further, and I find what is irreducible or basic (or perhaps even indubitable?). I find that there is a start, or something which IS really basic or foundational.
I end up with things like the distinction between is and not, the distinction between infinite and finite, between concept and not a concept. Even the schema of infinite regress must be accepted first, as well as the inferences it draws from and leads to, and the fact that it negates its opposing viewpoint.
The infinite regress, being of no beginning or end, cannot actually be situated as anything. It is formless, and yet in doing so, we are still talking about it somehow.
Does this not prove that the infinite regress problem itself has foundational beliefs, and that the members of the infinite regress are somehow existing in what has no form at all? And, if it does not have a form, how can we even say it is formless? Surely, it cannot oppose or be like anything, even itself.
Even the most radical skeptic must see that they accept similarity and distinction as concepts. They clearly aren't saying their position is not different from a non-skeptic’s, for instance.
So they actually DO have certain things which are foundational to even their own formulation. One such primitive notion is that of NEGATION or NOT or whatever you want to call it. This is not reducible to anything else. IT has no other parts, and it just is what it is. How do we know what it means? This is a whole other question, which I fear may never be answered. Perhaps the next species after us might be able to.
I am reminded of Wittgenstein’s hinges, before anyone answers with this, and even this has some criticisms. For instance, he says hinges have certain properties in other writings, and that they are not true or false, propositions or not, conceptual or not, in others. In certain notes, he says hinges are actually necessary to even explicate the concept of hinges, and yet he NEVER addresses the simple fact that hinges are opposed to not-hinges. There is an opposing viewpoint, and one in which someone (a radical skeptic) could envision as being the correct explanation of a part of reality, as opposed to the one he propounded. He cannot escape saying something is something else, and yet he is trying to explain this in the first place.
The concept of a concept, the concept of 'itself', the concept of listing things out the way I am listing them out right now, these are all concepts. I have heard some saying that concepts are wholes with parts, and primitive concepts are unities without any parts, and which are indivisible. But even this description itself is a bunch of concepts, and even saying 'bunch', 'of', etc, are all concepts, and even saying that all of these are concepts one-by-one, like so, is itself a bigger framework, and is a concept nonetheless.
This begs the question: how do we know what 'concept' means? It doesn't seem to be divisible or indivisible, and yet, as a concept, it encompasses all of them, none of them, and some of them. Everything I have written right now, including even the question and answer framework, is a concept.
This so-called infinite regress cannot be, without members which get situated in this absolute nothingness that is supposed to be the infinite. And in the infinite regress it is odd that we should prescribe a direction to it, as well as others, about the way things should be justified, like a ladder or chain - a chain which would not exist without its individual links nonetheless, and which provides the reference for us to even know that it is before or after; that it is in fact a regress from previous belief periods.