As a fiction writer and an ex-perfectionist, I have a really intimate understanding of your friend's situation. I just dug myself out of it and trust me, it's a terrible combination which will lead to mediocrity. In fact, a fear of failure is quite inter-disciplinary, but I find it is especially disastrous in the arts.
Writing is hard. It is an emotional, arduous process filled with hopes and dreams. Typically, a writer will pour their entire self-worth into their work, and if it fails to achieve the status or effect that was intended, then the "logical" explanation is that they are failures. Writers are paradoxically the proudest and most self-conscious people. However, this mindset is destructive and counter-creative.
Writers must be able to objectively separate their worth from their reader's perception of their work. Your friend seems to be seriously struggling with this problem. Abandoning an entire project because of a few problems! How many opportunities has he missed because of that fatalistic perspective? Writing is not a discrete event. It's a continuous iterative process. It relies on months of re-writing drafts, tweaking characters, revising until your eyes bleed. Nobody ever writes a perfect novel in one draft. Not Rowling, not Tolkien, not Martin. Nor Fitzgerald, nor Orwell, nor Faulkner. The greatest writers sometimes re-write their stories 30+ times. That is how the industry works, like it or not.
If your friend can't understand that reality, then there is nothing more you can do on your end, except continue to offer constructive criticism and reinforcing the positives of his work. As long as you give criticism with the intent of helping him improve his work, then you are being a perfect reader--please don't be gentle. It's horrible advice. A real publisher and editor will not be gentle. They will be blunt and heartless. You will get your printed manuscript back with entire pages circled, labelled "Yawn... useless, remove it". It took me a while to finally figure out that I am not perfect, that I make mistakes, but I can also fix them to produce a piece which is even better. A writer must be prepared for difficulty and sharp criticism. Then after digesting it, they have to buckle down and revise. This is the blight of the writer.
Now, what to do. First, if he does not already have a workshop, I would advise him to look for writers' workshops near him. These are a necessary tool and incredibly helpful for helping a writer understand the process of writing and receiving feedback. Then, I would start with the positives. Try to touch upon big literary terms such as: plot pacing, character development, tone, diction and style (does he write flowery, intricate sentences, or terse, matter-of-a-fact sentences? Compliment his style, writers like to hear that), etc. Whenever you give feedback (to anyone in any situation), the positives always offset any potential negativity from your feedback. Let's say the novel had strong pacing, solid characters, with a rough and tumble tone to it. I would start like this (I am bolding words and phrases which are both common in writing feedback, and good ways to sound objective but still descriptive):
This was an exciting read, you yet again prove you have a strong grasp on narrative and realistic world-building. Larry, the protagonist, was really well developed. His arch was also satisfying, I could identify with him well. The dialogue also felt natural, especially Larry's femme fatale, who you did a good job making into a mysterious erotic character. In fact, reading about all the characters was a pleasure.
Find the good things first, so you can validate his skill. Then, ease into your hesitations. Luckily there is only one major issue, the ending. One tip when giving feedback on stories is to rarely prescribe things and do not say an entire idea or character is "stupid, useless, unoriginal", leave those nouns to the busy, impersonal publishers. Be specific! Do not say:
The ending scene when Larry and the evil mastermind Harry fought in the train station and Larry accidentally shot his love was lame and such a cop-out. Really what you should've done is have...
Two no-no's! No lame descriptions, and no prescriptions!
Instead, focus on the real story in the novel, and describe why something was not the best it could be. Always talk in potentials. After touching upon the positives:
Okay, I did have one slight reservation, and with your creative brain, easily fixable. But, this is your story, so take everything I say with a grain of salt: Remember friend, this is your world, and your creativity, I only want to help you achieve the best version of your novel. That being said, I think your ending has room for improvement because I thought it was a tad too predictable. I like how everything culminated to that point, but maybe you should reconsider how everything goes down. For instance, it felt rushed...For instance, it wasn't as exhilarating as the rest of those exciting scenes you wrote...etc.
Those 'For instances' are where you describe what you didn't like, such as the pacing, the resolution, the character interactions, the contrived nature of the scene, etc. Then, a transition to the final remarks. Here you want to re-affirm how everything else was great, and how the ending is just a tiny fraction of the greater work.
Everything else was great, just revisit the ending, and I would LOVE to work with you bouncing off ideas and whatnot because frankly, I'm attached to the characters. Also, don't forget that endings are always super hard! So don't worry!
So, now you listen to his response, he may ask many more questions--at least he should. This could go two ways. If he decides to throw his hands up and scrap the whole thing, I would suggest explaining how destructive that mindset is to his writing career. Direct him to research "revision in writing" (Robert Boswell's After the Workshop: Transitional Drafts is a great start, cannot recommend enough). Always be a token of positivity and constructive feedback, but ultimately overcoming the fear of failure is a personal problem he must deal with. If a writer can't deal with failure, then they are simply not cut out for writing. Good luck!
