It was stated at the very beginning of the novel that the ground of Sir Walter Elliot’s character is his personal vanity.
Vanity was the beginning and the end of Sir Walter Elliot’s character; vanity of person and of situation. He had been remarkably handsome in his youth; and, at fifty-four, was still a very fine man. Few women could think more of their personal appearance than he did, nor could the valet of any new made lord be more delighted with the place he held in society. He considered the blessing of beauty as inferior only to the blessing of a baronetcy; and the Sir Walter Elliot, who united these gifts, was the constant object of his warmest respect and devotion.
Jane Austen (1817). Persuasion, chapter 10. Project Gutenberg.
This vanity overflows into the judgment of others based on their superfical appearance. For example, Sir Walter objects to the Navy due to his personal disgust at the faces of sailors:
“Yes; it [the Royal Navy] is in two points offensive to me; I have two strong grounds of objection to it. First, as being the means of bringing persons of obscure birth into undue distinction, and raising men to honours which their fathers and grandfathers never dreamt of; and secondly, as it cuts up a man’s youth and vigour most horribly; a sailor grows old sooner than any other man. I have observed it all my life. A man is in greater danger in the navy of being insulted by the rise of one whose father, his father might have disdained to speak to, and of becoming prematurely an object of disgust himself, than in any other line. One day last spring, in town, I was in company with two men, striking instances of what I am talking of; Lord St Ives, whose father we all know to have been a country curate, without bread to eat; I was to give place to Lord St Ives, and a certain Admiral Baldwin, the most deplorable-looking personage you can imagine; his face the colour of mahogany, rough and rugged to the last degree; all lines and wrinkles, nine grey hairs of a side, and nothing but a dab of powder at top. ‘In the name of heaven, who is that old fellow?’ said I to a friend of mine who was standing near, (Sir Basil Morley). ‘Old fellow!’ cried Sir Basil, ‘it is Admiral Baldwin. What do you take his age to be?’ ‘Sixty,’ said I, ‘or perhaps sixty-two.’ ‘Forty,’ replied Sir Basil, ‘forty, and no more.’ Picture to yourselves my amazement; I shall not easily forget Admiral Baldwin. I never saw quite so wretched an example of what a sea-faring life can do; but to a degree, I know it is the same with them all: they are all knocked about, and exposed to every climate, and every weather, till they are not fit to be seen. It is a pity they are not knocked on the head at once, before they reach Admiral Baldwin’s age.”
Chapter 3.
His lawyer has to reassure Sir Walter that he will be able to stand the sight of his prospective tenant:
Mr Shepherd hastened to assure him, that Admiral Croft was a very hale, hearty, well-looking man, a little weather-beaten, to be sure, but not much, and quite the gentleman in all his notions and behaviour
Chapter 3.
So Sir Walter’s comment on Colonel Wallis’s appearance emphasizes the superficiality of his judgment. Wallis’s personal appearance is irrelevant to the question of whether his information about William Elliot’s marriage is reliable, but in Sir Walter’s view it is the most important consideration. The comment is a hint to the reader that maybe Wallis has not told the whole story.