Look, fantasy is the mother genre -- e.g. Gilgamesh, the Illad, Odyssey and most religions. Science Fiction is the brash offshoot. All literature has deep roots in fantasy, which in turn emerges from the font of our dreams.
Having said that, what is my definition of the separation? I think it is very basic, revolving around the notion of human improvability:
"Do you believe it is possible for children to learn from the mistakes of their parents?"
For all the courage and heroism shown by fantasy characters across 4000 years of great, compelling dramas -- NOTHING EVER CHANGES!
== Nothing Ever Changes ==
No, the root element is right there in that word "change." Science fiction borrows many elements from the mother genre - fantasy -- elements of boldness and the fantastic that date back to Homer and Gilgamesh. But sci fi then rebels against all literary foundations by embracing change. Even when it warns against BAD change it is relishing, exulting, expanding upon what Einstein called the "gedankenexperiment" or thought experiment: What if?
When SciFi goes "whatif" it takes the sacred word seriously.
Fantasy has its attractions. Something about feudalism resonates, deep inside us. We fantacize about being the king or wizard. It's in our genes. We are all descended from the harems of the guys who succeeded at that goal. The core thing about fantasy tales is that, after the adventure is done and the bad guys are defeated... the social order stays the same.
It may be the natural genre... but should we be proud of that?
== The Possibility of Change ==
Science fiction, in sharp contrast, considers the possibility of learning and change.
Not that children always choose to learn from their parent's mistakes! When they don't, when they are obstinately stupid and miss opportunities, you can get a sci fi tragedy... far more horrible than anything "tragic" in Aristotle's Poetics. Aristotle says tragedy is Oedipus writhing futilely against fate. A sci fi tragedy portrays people suffering, same as in older tragedies... but with this crucial difference -- things did not have to be this way. It wasn't "fate." We - or the characters - could've done better. There was, at some point, a chance to change our own destiny.
One type of tragedy makes you weep - hey, Oedipus is powerful stuff. But for millennia the deep moral lesson - the thing taught in all "campbellian myths" - is that resistance is futile. The overall situation, the rule of fate, remains the same.
The other type of tragedy - the new kind - is a cautionary tale that may change your decisions. It may alter destiny.
You can see why the absurd old farts who inhabit most lit departments hate science fiction. SF considers it possible that the eternal "verities" and relentless stupidities praised by Henry James might someday be obsolete! If we make kids who are better than us (our goal, duh?) then their Startrekkian heirs will still have problems.
Why insist that our descendants have to fret over the same ones? Can't they assume the solutions we find, take them for granted, and move on to new, interesting issues of their own?
Isn't that what we did?
==The Rulers of Destiny==
The implicit assumption in most fantasy is that the form of governance that ruled most human societies since the discovery of grain must always govern us. Oh, kingly rulers my topple and shift, but the abiding assumptions and social castes generally do not.
Aragorn may be a better king than Sauron would have been. Hurray. Fine. But he's still a freaking king. And the palantir on his desk that lets him see faraway places and converse with viceroys across the realm is still reserved for the super elite. No way are we going to see mass-produced palantirs appearing on every peasant's tabletop from Rohan to the Shire. (The way our civilization plopped such a miracle on YOUR tabletop.) It never even occurs to Aragorn or Gandalf to give the poor the godlike powers they themselves get to wield... let alone provide them with libraries, running water, printing presses or the germ theory of disease. Only little Peregrin Took seems to get a glimmer of an idea in that direction. The only character who briefly ponders possibilities, and he's soon bullied out of it.
Of course, magical thinking solely the province of non-technological minds. In many ways, the nerdy-techno "singularity" is - at root - just a modern manifestation of magical-transcendentalism. Indeed, our 21st Century America is awash in mystics! The technological illiterates among them either wallow in the Book of Revelations or lefty-Gaian nostalgism or else solipsistic AynRandianism, Those who are tech-empowered shift their transcendentalism to what's been called the "rapture of the nerds." Same stuff though, when you dig deep to the level of personality, and thousands of years old.
The trend toward feudal-romantic fantasy may seem harmless. But dreaming wistfully about kings and lords and secretive, domineering wizards is simply betrayal. Pure and simple. Those bastards were the enemy for 6,000 years. Some kings and wizards were less bad than others. But they were all "dark lords." We are the heirs of the greatest heroes who ever lived. Pericles, Franklin, Faraday, Lincoln, Einstein. Any one of whom was worth every elf and dragon and fairy ever imagined.
And when an author like Tim Powers resists these assumptions, in books such as The Drawing of the Dark, he is writing science fiction, whether or not there are pirates, or wizards or demons.
This is why, despite her dragons and bards and medieval craft in her Dragons of Pern universe, Anne McCaffrey proclaimed loudly that "I am a science fiction writer!"
Because her characters know that change is coming. Some resist, many are eager to bring it on as fast as they can. And the future on Pern will have both dragons and flush toilets. Songs and tapestries and universities and hyperdrive ships.
Terry Pratchett writes science fiction because his Discworld Universe (borne through space by a mythical turtle) has something called progress. People are waking, rising up.
On the other hand, George Martin's depressing Game of Thrones saga has very little magic in it, but it consigns the peasants to endless, endless, endless misery and feudal oppression, with absolutely no hope of progress. It is part of the longer/older tradition stretching back to Homer. It is fantasy.
Science fiction contemplates the possibility of successfully defying fate, creating a better future through our own efforts.
The best science fiction does that, head on.
My essay: Science Fiction and its Heresies
and Speculations on Science Fiction