Literary fiction is also known as "serious fiction," though personally I dislike both of those terms. They imply, at least to my ear, that all other types of fiction (genre fiction, in particular) is somehow less literate and less serious.
Still, literary fiction is the term that the book-selling business uses, so I guess we're stuck with it!
If you go into a bookstore, you can usually tell the genre novels from the literary ones instantly. Here's how...
1. The covers will be different
Whereas the genre novels have eye-catching covers – handsome men on the romances, dripping blood on the horror novels – literary novels are more subtle, more "arty."
Literary novel sometimes have stickers on the cover, too, saying that the book was short listed for the Booker Prize or won the Orange Prize (or something similar).
2. The two types of novel might be sold in a different format
Genre fiction is usually sold in the "mass-market paperback" format (unless you happen to be one of the big, household names and have a bestseller on your hands every time you write a novel).
Literary fiction, on the other hand, usually appears in hardback form first (or else as a "trade paperback," which is the same size as a hardback but has a soft cover)... and then in standard paperback a year later.
Of course, these old rules are rapidly flying out of the window as e-books (Kindle books, in particular) continue to gain in popularity.
Even for physical books, the old ways of doing things no longer apply... particularly as more and more authors publish their novels independently and are free to choose whatever format they like.
3. The titles will be different
The titles of commercial fiction tend to be more direct and encapsulate perfectly what the novel is about...
Literary titles are more offbeat, more "arty" again, but just as eye-catching in their way...
4. You'll find them in different sections of the bookstore
Genre fiction will have areas of shelving all to itself (one area for crime novels, one for romances, and so on). Literary fiction will appear in the "General A-Z" section, along with mainstream fiction.
One last point...
"Serious" novels generally sell in smaller quantities than genre novels or mainstream novels, meaning publishers are less likely to take a gamble on them - though you shouldn't let that put you off writing them...
You must always write the type of book that you want to write.
And as publishing your novel independently has become not just viable but (arguably) preferable, what the publishing houses are "likely" or "not likely" to do becomes increasingly irrelevant.
True, the book-buying public still buys fewer literary novels, whether they're shopping at Amazon or in a bricks-and-mortar bookstore. But if a literary novel receives some positive word-of-mouth buzz on social networks, or even wins a prestigious award, sales can still be huge.
Apart from looking different to genre fiction, and being shelved in a different location in the bookstore, what else sets a "serious" book apart? In a nutshell, it is this: Literary fiction is more character-driven and less concerned with a fast-paced plot than genre fiction.
Depending on your point of view, this either makes a great work of literature moving and profound... or as dull as reading the dictionary (because "nothing exciting happens").
But here's the thing...
Just as the best genre novels are populated by well-crafted fictional characters, so the best literary novels have page-turning plots. (Admittedly, this plot is not very likely to consist of car chases and explosions... but things still "happen" nonetheless.)
So it's really just a difference of emphasis. If writing a gripping plot is important in genre fiction, in a literary work the plot can be less momentous, more subtle, less frenetically-paced, more beneath the surface... but it still needs to be there.
Fans of literature might consider genre novels to have less artistic merit, to be formulaic, melodramatic, and so on. And fans of genre fiction might consider literary novels to be boring books in which nothing much happens. But...
Both views would be wrong.
"Serious" fiction isn't better than genre fiction in the same way that a table isn't better than a wheelbarrow – they are simply different products serving different needs.
Think of it like this: each of the fictional genres is aimed at a specific group of readers who take pleasure from reading those types of books...
In that respect, literary fiction can be seen as just another genre – it is simply fiction aimed at a specific group of readers who like what literary novels have to offer.
But what do they have to offer, exactly?
I've talked generally about literary novels having a concern for the exploration of character, and less concern for a page-turning plot. But there are other important factors which set "serious novels" apart. Let's take a more in-depth look at...
This isn't to say that genre novelists aren't concerned with deep characterization. I'm sure that Stephen King, for example, cares deeply about his characters and tries to make the most fully-rounded characters that he can.
It is just that he has a lot less space to do it in than his literary counterparts. Why?...
Fans of genre fiction are essentially after a "good read," and that means a concrete plot with a lively pace and plenty of twists and turns along the way.
Disrupting the flow for a few pages to describe in detail a character's mental anguish, or an incident from their past, is therefore risky in a genre novel. But it's exactly what the reader of a literary novel expects.
The danger for the literary novelist is to go to the other extreme....
All good novels, whatever the genre, should have a theme. A theme runs beneath the surface and is essentially what the novel is "about."
As readers of fiction, we like to be entertained by the surface plot. But we also like a deeper experience, one in which the novel's events "say" something about what it means to be a human and what it takes to get by in this world.
(We might not notice this happening explicitly, but we'll sense that we're having a richer and more rewarding reading experience, even if we can't quite put our finger on why we are.)
Just as a literary novelist has more time at his or her disposal to characterize in depth, so they have more time to explore the "philosophical" issues and ideas and insights running through their novel.
This philosophical exploration won't take place literally (literary novels are still novels, not academic papers). Instead, it will run "beneath the surface" of conversations, of a character's thoughts, or of the events themselves.
The theme will form a kind of subtext to the entire novel, if you like, but one which the novelist still needs time and space to bring to the surface occasionally. (And genre novelists, remember, don't have the luxury of much time and space, because that slows down the plot.)
Don't worry if you don't understand all that right now. It will make more sense once you have read the article defining theme.
All you need to understand here is that literary novels have more space in which to explore theme than genre novels do.
Fans of literary fiction are perfectly willing to read a novel in which deep characterization and the exploration of ideas (i.e. theme) is placed above the forward momentum of the plot. As a matter of fact, they expect it.
And what they (and a publisher) also expect is that the writing itself be of excellent quality...
Instead, literary writers simply need to possess a masterful way with words. Their style can be simple or more ornate, but the prose must always be rich and finely-crafted and a pleasure to read.
Does that mean the genre novelist can afford to be sloppy in their writing? No, of course not. But because fans of genre fiction are mainly after a page-turning plot, the genre writer can get away with more "workmanlike" prose.
The genre novelist is confined by the conventions (or "rules") of their chosen category of fiction. If readers of crime novels, for example, expect a body to appear within the first three chapters, your own crime novel had better not disappoint them.
Of course, the rules can be bent and boundaries can be pushed. But bend or push them too far and your genre novel may no longer be recognizable as belonging to its category - which means that a fan of that category of fiction is unlikely to find it satifying.
In literary novels, there are no such things as rules.
You are free to tackle any subject matter and any theme you choose, and to structure the story however you wish.
Literary fiction is more likely to break the wider fictional rules, too – the ones about how to create characters, how to plot fiction, and so on.
All novelists should be willing to break some of the rules some of the time. But writers of literary fiction have more freedom than most.
In literary fiction, anything goes – just so long as it works.