Literary fiction is also known as "serious fiction," though personally I dislike both of those terms, implying as they do that all other fiction - genre fiction, in particular - is somehow less literate and less serious.
Still, literary fiction is the term that is universally used to describe these types of books, and so we are stuck with it.
If you go into a bookstore, you can usually tell the genre novels from the literary ones instantly. Here's how...
1. The covers will be different
Whereas the genre novels have eye-catching covers - handsome men on the romances, dripping blood on the horror novels - literary novels are more subtle, more "arty."
Literary books sometimes have stickers on the cover, too, saying that the novel was short listed for the Booker Prize or won the Orange Prize or something similar.
2. The two types of novel might be sold in a different format
Genre fiction is usually sold in the "mass-market paperback" format (unless you happen to be one of the big, household names and have a bestseller on your hands every time you write a novel).
Literary books, on the other hand, usually appear in hardback form first (or else as a "trade paperback," which is the same size as a hardback but has a soft cover)... and then in standard paperback a year later.
3. The titles will be different
The titles of commercial fiction tend to be more direct and encapsulate perfectly what the novel is about...
Literary titles are more offbeat, more "arty" again, but just as eye-catching in their way...
4. You'll find the two types of fiction in different sections of the bookshop
Genre fiction will have areas of shelving all to itself (one area for crime novels, one for romances, and so on).
Literary fiction will appear in the "General A-Z" section, along with mainstream fiction.
One last point...
"Serious" novels generally sell in smaller quantities than genre novels or mainstream novels, meaning publishers are less likely to take a gamble on them - though you shouldn't let that put you off writing them.
You must always write the type of book that you want to write.
And remember that if a literary novel wins a prestigious award, or is on the receiving end of some positive word-of-mouth buzz, sales can be huge.
Apart from looking different to genre fiction, and being shelved in a different location in the bookstore, what else sets a "serious" book apart? In a nutshell, it is this: Literary fiction is more character-driven and less concerned with a fast-paced plot than genre fiction.
Depending on your point of view, this either makes a great work of literature moving and profound or else as dull as reading the phone book (because nothing much happens).
But here is the thing...
Just as the best genre novels are peopled by well-crafted fictional characters, so the best literary novels have a page-turning plot. (Admittedly, this plot is not very likely to consist of car chases and explosions, but things still happen nonetheless.)
It is really just a difference of emphasis. If writing a gripping plot is paramount in genre fiction, in a literary work the plot can be less momentous, more subtle, less frenetically-paced, more beneath the surface - but it still needs to be there.
Fans of quality literature might consider genre novels to have less artistic merit, to be formulaic, melodramatic, and so on. And fans of genre fiction might consider literary novels to be boring books in which nothing much happens.
Both views would be wrong.
"Serious" fiction isn't better than genre fiction in the same way that a table isn't better than a wheelbarrow - they are simply different products serving different needs.
Think of it like this: each of the fictional genres is aimed at a specific group of readers who take pleasure from reading those types of books...
In that respect, literary fiction can be seen as just another genre - it is simply fiction aimed at a specific group of readers who like what literary novels have to offer.
But what do they have to offer, exactly?
I've talked generally about literary novels having a concern for the exploration of character, and less concern for a page-turning plot. But here is a more detailed look at the Defining Characteristics of Literary Fiction.