Where is the border line between a novel, creative work and work that is of scientific value? I have received reviewer comments like:
the ideas are creative and interesting but work adds little scientific value.
What do reviewers look for when making such comments/judgements? They often do not clearly state what was missing in the paper. My field is computer science.
Let me describe two extremes, though they come from very different fields.
In mathematics, if you re-prove an old well-known theorem with a new technique that might be applied elsewhere it will be very interesting, whereas if you prove a new theorem with only old standard techniques it may have much less interest.
Similarly, in Computer Science, a boring and straightforward program might answer an interesting question. If the question isn't about CS itself, this might not be considered "scientifically interesting", whereas if it were a longstanding CS question it would be. On the other hand an interesting and creative program might answer a question of no significance. This might be judged either way.
In many of the sciences (chemistry, psychology, ...), you can, and many do, use very standard statistical techniques to answer questions. But to be interesting, the questions themselves have to be significant since the technique isn't. However, what is significant to you might seem trivial to others and vice versa. Even if you use a "creative" technique to answer an insignificant question it might not have much scientific merit unless someone can conceive of using that technique to answer other, more significant questions.
So, the variables are, at least, (a) the question attacked (b) the techniques used. I'm guessing (only) that the comments you got imply that you are strong on (b) but not so strong on (a) and the reader didn't extrapolate. But it depends on the field.
In my experience this is a sign that you submitted to the wrong journal/conference. For every journal/conference there is a community of researchers that like to do research in a certain way. So if a reviewer likes your work but it does not match what he thinks valuable research should look like, that is the response you get.
For example, in my area of computer networks there are researchers who do more formal analysis and others who prefer a more practical approach. So some researchers think that if you can not provide a formal proof that your system works it has low value while for others the demonstration in a real-world scenario is essential.
Once I submitted a practical paper to a more theoretically oriented conference and it got rejected with very similar wording than what you wrote. It later got accepted at a reputable application-oriented journal. The same happened the other way around.
So try to find the community that is the best fit to your work. That can include - unfortunately - a lot of trial and error. The other option is to try to suit everybody, but that includes a lot of work and you will have a hard time sticking to page limits (been there...).
this paper seems to be an industry report for developing a tool, and I feel that the scientific contribution of this paper is low. The question is how do I avoid such comments in the future. I feel there is a general expectation for ideas and solution to be characterized or formalized.
This question is rather hard to answer, as "little scientific value" is extremely subjective.
There are ways to figure this out, however. Have you discussed this with your scientific adviser? If you don't have one, you can seek feedback from your colleagues, as they will be most familiar with your field.
I think it will be of value for you to present work at a conference, or put it on pre-print server (Bio/Arxiv) and then ask community for feedback.
Also, keep in mind that reviewers are humans (even though it might seem counter-intuitive) so they might be wrong.
