Just the question in the title, I'm trying to understand how something like analysis could be developed without formal constructions of the real numbers.
I'm also very interested, if the answer is "yes", to know what was that people thought was a "real number" at that time.
See e.g.:
ARTICLE I
Whatever is capable of increase or diminution, is called magnitude, or quantity.
[...] §4. the determination, or the measure of magnitude of all kinds, is reduced to this: fix at pleasure upon any one known magnitude of the same species with that which is to be determined, and consider it as the measure or unit; then, determine the proportion of the proposed magnitude to this known measure. This proportion is always expressed by numbers; so that a number is nothing but the proportion of one magnitude to another arbitrarily assumed as tne unit.
§5. From this it appears, that all magnitudes may be expressed by numbers; and that the foundation of all the Mathematical Sciences must be laid in a complete treatise on the science of Numbers, and in an accurate examination of the different possible methods of calculation. This fundamental part of mathematics is called Analysis, or Algebra.
And page 39 :
§128. There is therefore a sort of numbers, which cannot be assigned by fractions, but which are nevertheless determinate quantities; as, for instance, the square root of $12$: and we call this new species of numbers, irrational numbers. They occur whenever we endeavour to find the square root of a number which is not a square; thus, $2$ not being a perfect square, the square root of $2$, or the number which, multiplied by itself, would produce $2$, is an irrational quantity. These numbers are also called surd quantities, or incommensurable.
Yes, mathematicians used the concept of real number long before rigorous definitions arose, just as mathematicians used the concept of a complex number before the argand plane was described, the dirac delta function was used before it became rigorous, etc. Intuition almost always arises before rigour.
In modern times, the style has become to model every mathematical discipline as a field of the theory of sets. Before the late 19th century, sets were virtually nonexistent. Newton could not have possibly come up with dedekind cuts, cauchy sequences, etc, for one requires some intuition about set theory to interpret these results. If one were rigorous, one would instead proceed by an axiom system as in the style of Euclid (though not a system as rigorous as those in the mathematical logic of today).
All constructions of the real numbers above are meant to construct a ``completion'' of the rational line $\mathbf{Q}$. Most of the time, we prove things about $\mathbf{R}$ from abstract field axioms, with a completeness axioms included (least upper bound principle, etc). So Newton could have proceeded from this route, but if we look at Newton's work we do not find this style of axiomatics. Most of Newton's proofs are not analytical -- they do not involve numbers. If you read the Principia, you will find that most proofs proceed by geometrical diagrams, like the greeks. Newton's main method is treating diagrams as infinitisimal, using intuition to obtain the results about the limits of diagrams (for instance, if we take a line between two points on a circle, then when they are taken to be infinitisimally close the line is perpendicular to the circle. In geometry, one uses much more intuition than rigour. In particular, I imagine at some point in the principia Newton applies (via intuition) a disguised geometric form of the intermediate value theorem -- a theorem which, if taken axiomatically, implies the completeness of the real numbers, and thus implies that Newton really is using the reals. This theorem is `obvious' to the unititated, but in introductory analysis courses one finds the idea is much more subtle. Remember that in order for these types of principles to be scrutinized, one needs paradoxes which challenge thought, which space filling curves and nowhere differentiable functions provided in ample amount.
Of course, there was still controversy back then. The philosopher George Berkeley in particular criticized the method:
It must, indeed, be acknowledged, that [Newton] used Fluxions, like the Scaffold of a building, as things to be laid aside or got rid of, as soon as finite Lines were found proportional to them. But then these finite Exponents are found by the help of Fluxions. Whatever therefore is got by such Exponents and Proportions is to be ascribed to Fluxions: which must therefore be previously understood. And what are these Fluxions? The Velocities of evanescent Increments? And what are these same evanescent Increments? They are neither finite Quantities nor Quantities infinitely small, nor yet nothing. May we not call them the Ghosts of departed Quantities?
Nonetheless, the results Newton obtained were correct, so there wasn't too much of this backlash.
The history of mathematics is a huge topic with a large literature. In ancient Greek mathematics, what we would call numbers were called magnitudes and represented continuous quantities like lengths, areas and magnitudes. The theory of proportions in Book 5 of Euclid's elements is particularly relevant, e.g., see How to best understand Euclid's definition of equal ratios? How does it relate to Dedekind cuts? A favourite of mine is Proposition 2 of Book 10, which gives one of the methods the Greeks used to show that certain numbers are irrational: roughly speaking you apply Euclid's algorithm to try to find a greatest common divisor of given magnitudes $x$ and $y$ and if it doesn't terminate $x/y$ is irrational. (This leads to very nice geometric proofs of the irrationality of quadratic surds like $\sqrt{2}$.)
It is perhaps also worth pointing out that Euclid was writing the equivalent of an undergraduate textbook several centuries after Pythagoras. Results like some of those obtained by Archimedes using Eudoxus's method of exhaustion (a method of calculating areas and volumes)are not in the Elements.
Other answers to your question relate to what happened over the subsequent two millennia. Virtually all western mathematicians up to the mid-twentieth century will have had Euclid's Elements as their first mathematical textbook.
Yes, they were using them, even though mathematicians never really knew back then "what they were". They were generally understood to be those numbers that could be values of physical quantities (such as time, length, energy, speed etc.).
Note, though, that in the really distant times (contemporary to Plato, say), the Greeks were still perplexed by the existence of irrational numbers (they knew that the ratio of the diagonal of a square to its edge was not rational - what we call today $\sqrt 2$). It took some time for humanity to "swallow" irrational numbers...
As a side-note, to understand how confused even the greatest mathematical minds of that time were (with respect to foundational issues), note that Gauss once called the imaginary number $\rm i$ "vera umbrae umbra" ("a true shadow of shadows" - i.e. something without a proper existence), and Euler was puzzled by the fact that if he replace $x$ by $2$ in the (formal) equality $\dfrac 1 {1-x} = \sum \limits _{n = 0} ^\infty x^n$ he obtained a negative number equal to a positive infinite one...
Simon Stevin used unending decimals to represent all numbers (whether rational or not) already at the end of the 16th century. In the 17th century, Descartes seems to have been the first to use the term real to describe ordinary numbers.
Since unending decimals provide a satisfactory account of the real numbers, the latter should be attributed to Stevin rather than either Cantor or Dedekind. In particular, many authors since Stevin have used these "ordinary" numbers to prove theorems without awaiting the more abstract developments of the last third of the 19th century.
In particular, Cauchy provided a satisfactory proof of the intermediate value theorem way before Cantor. The existence of the root follows simply by constructing an unending decimal.
