Literary fiction is also known as "serious fiction," though personally I dislike both of those terms, implying as they do that all other fiction - genre fiction, in particular - is somehow less literate and less serious.
Still, literary fiction is the term that is universally used to describe these types of books, and so we are stuck with it.
If you go into a bookstore, you can usually tell the genre novels from the literary novels instantly...
Whereas the genre novels have eye-catching covers - handsome men on the romances, dripping blood on the horror novels - literary novels are more subtle, more "arty." They sometimes have stickers on the cover, too, saying that the novel was short listed for the Booker Prize or won the Orange Prize or something similar.
The titles will also be different: genre titles tend to be more direct and encapsulate perfectly what the novel is about; literary titles are more offbeat, more "arty" again, but just as eye-catching in their way. So if genre novels have titles like Beastchild (Dean Koontz) and The Lake of Darkness (Ruth Rendell), literary titles are Instances of the Number 3 (Salley Vickers) or The Alchemy of Desire (Tarun J Tejpal).
The two types of novel might be sold in a different format, too. As I mentioned, genre fiction is usually sold in the "mass-market paperback" format (unless you happen to be one of the big names), whereas literary books appear in hardback form first (or else as a "trade paperback," which is the same size as a hardback but has a soft cover) and then in standard paperback a year later.
Finally, of course, you will find them in different sections of the bookshop. Genre novels will have areas of shelving all to themselves (one area for crime novels, one for romances, and so on), while literary ones will appear in the "General A-Z" section, along with mainstream fiction.
Literary novels generally sell in smaller quantities than genre or mainstream novels, meaning publishers are less likely to take a gamble on them - though you shouldn't let that put you off.
You must always write the type of book that you want to write.
And remember that if a literary novel wins a prestigious award, or is on the receiving end of some positive word-of-mouth buzz, sales can be huge.
Apart from looking different to genre fiction, and being shelved in a different location in the bookstore, what else sets a literary novel apart? In a nutshell, it is this: Literary fiction is more character-driven and less concerned with a fast-paced plot than genre fiction.
Though just as the best genre novels are peopled by well-crafted fictional characters, so the best literary novels have a page-turning plot. It is really just a difference of emphasis.
If writing a gripping plot is paramount in genre fiction, in a literary work the plot can be less momentous, more subtle, less frenetically-paced, more beneath the surface - but it still needs to be there. Here is literary agent Nathan Bransford on the subject...
"Sooooooooo much literary fiction I get in the old query inbox is plotless. It's just a character musing about the vagaries and eccentricities of everyday existence. The prose is lush, the character detailed, but one problem - absolutely nothing is happening and thus it's (forgive me) extremely boring. Good literary fiction has a plot."
Fans of quality literature might consider genre novels to have less artistic merit, to be formulaic, melodramatic, and so on. And fans of genre fiction might consider literary novels to be boring books in which nothing much happens.
Both views would be wrong.
"Serious" fiction isn't better than genre fiction in the same way that a table isn't better than a wheelbarrow - they are simply different products serving different needs.
Think of it like this: each of the fictional genres is aimed at a specific group of readers who take pleasure from reading those types of books...
In that respect, literary fiction can be seen as just another genre - it is simply fiction aimed at a specific group of readers who like what literary novels have to offer.
What do literary novels have to offer, exactly? I've talked generally about them having a concern for the exploration of character (and less concern for a page-turning plot), but I need to be a lot more specific. Here are three defining characteristics of literary fiction...
You can read more in-depth explanations of these three points (together with a fourth characteristic of literary fiction I haven't mentioned here) in the detailed version of this article. This is available in my downloadable Ultimate Novel Writing Guide.
Next Step: Now it is time to take a look at the final type of book: Mainstream Fiction...