I got the following question in my e-mail a little while back and thought it and the subsequent answer might be of interest:
Q: "I was curious about something...when you look at response times on various sites (such as Duotrope), one of the potential responses is "Rewrite request." I've never gotten one of those, even when the rejection included "We liked this, but..." and followed with a thing or two that would be relatively simple to fix. I'm not saying that I should have gotten such a request. There may have been more wrong with the story than I thought or they said. But I am curious -- what makes a story worth asking for changes in vs. just letting it go? How do you (as a writer) know if it's really a simple fix or the story has fatal flaws?"
My answer (now slightly edited and expanded) was:
There are two kinds of rewrites I deal with, one more serious than the other. The more serious of the two amounts to - Hey, I like this story, and if you will fix A, B and C to these specifications, then I'll buy the story. The other kind of rewrite request is more along the lines of - Hey, this has potential, and if you'd consider reworking these areas or developing this, that, or the other a bit more, I'd be willing to look at it again.
I try to make a point, also, of telling authors that any decision to rewrite is ultimately in their hands, and if they disagree with what I'm suggesting they should feel free to say "No thanks, I like my story the way it is." And there have been one or two stories that I probably would have bought anyway. In fact, I can think of one story where the author said "No thanks" and I did buy it anyway.
On the other hand, I can also think of one particular writer who kicked and screamed through the whole revision process, but he did it anyway. And when all was said and done he publicly acknowledged that the story was better for the work we did on it. My job is a delicate balancing act of trying to make every story I publish be the best that it can be, yet to do so without taking over or being heavy-handed about it. These aren't my stories; they belong to the people who created them.
Of course, I just realized that I haven't answered your original question about what makes a story worth asking for changes vs. just letting it go. And to be blunt, the answer probably isn't any more complicated than this (usually): the closer a story is to being publishable, the more likely I am to ask for a rewrite.
There are, of course, always exceptions. Every once and a while I'll see a story and happen to particularly like the voice that author has captured (which is one of the harder things to do well), and I'll work through several rounds of revision with them to bring the story up to par. That's a LOT of work though, so I'm pretty selective about when I do that.
On the other hand, there are some authors I have enough experience with that I know what they are capable of and trust them to do a good job. Eric James Stone and Brad Beaulieu are two who come to mind. I don't have to say more than: Can you give me a little bit more of X, or; Please refine Y a bit, and they've come back to me with levels of work far beyond what I thought I was asking for, so that makes it a lot easier for me to ask for revisions again in the future.
The bottom line is that a rewrite request is serious business. I won't waste an author's time or my own with one unless I believe it will result in publishable work that I will want to buy and IGMS readers will want to read.
But that's how I, as an editor, decide whether or not to ask for a rewrite. And the original question was how does a writer know if the story has fatal flaws or not. I'd say there you simply have to trust your own instincts. Robert Heinlein is famous for saying that you should never rewrite except to editorial order - from an editor who is prepared to buy the work. (All five of Heinlein's rules are available in a variety of places, including Rob Sawyer's web-site: http://www.sfwriter.com/ow05.htm).
Personally speaking, several years ago I got the most scathing rejection I've ever received (from an editor I will refrain from naming) on a story I wrote titled, "Jeannie In A Bottle." The editor in question actually took the time to handwrite a note telling me the story was, among other things, "boring and predictable." I disagreed with him, so I stuck it in a new envelope and sent it to Gary Fry, editor of the British magazine Fusing Horizons. Same exact story. Didn't change a word. And Gary not only bought the story, he called it (forgive me if this sounds like bragging, but these are his exact words) "dazzlingly original, just the kind of thing I'm looking to publish." The point is, I believed it was a good story and trusted my instincts, and I'm (obviously) glad I did. It's vitally important to remember that all editors do not have the same tastes, interests, or needs.
Conversely, I've also had a few stories that were rejected several times and after I had enough distance from them, I was able to look at them objectively and say that they would never be published without major rewrites and so I abandoned them. So I guess the real answer is that if you do this long enough, you'll just know.
How's that for helpful?