If you are at all confused about the different types of novels - genre, literary, and mainstream - you are not alone: publishers and booksellers aren't always sure, either (and I know because I've asked them).
For every novel that is easy to categorize, there is another one that isn't nearly such a neat fit.
As a matter of fact, there are actually as many fiction genres as there are fiction writers, in the sense that the fictional world created by each and every novelist is essentially unique - and, yes, that means your fiction will also be unique.
The best that the book industry can do is group similar writers into categories (they need to do this in order to better market their products). But these categories can never be wholly satisfactory, due to the fact that every novelist's output is essentially unique.
Where does that leave you?
It leaves you having to face the reality that, assuming you are interested in having your fiction reach as wide an audience as possible, you will have to position your book in the market - which means aiming at one of the existing categories.
Because categorizing fiction is such an inexact science (with very blurry edges), you shouldn't get too hung up on what to call your novel. (If you can't decide if it's a "psychological thriller" or "psychological suspense," just pick one or the other and move on.)
So long as you are satisfied that published novels exist that are similar enough to the one you are writing (but not so similar that there is nothing unique about your own), you will be fine.
If the novel you are writing has no counterparts whatsoever in the bookstore, you are on more dangerous ground - but still in with a shot at creating a brand new genre all of your own.
And that is pretty much all there is to say - oh, except for the fact that you've probably forgotten half of what I wrote in the articles looking at the three types of novels and now need a neat summary of all the key points to help you make your mind up once and for all. Here it is...
The first decision you need to make is whether any of the fiction genres appeal to you, and the best way to know that is to ask yourself if you regularly read genre novels (crime, science fiction, horror, and so on).
It is crucial that you are a fan of your intended genre of fiction.
If you love romantic novels, for example, you will have the necessary knowledge and passion to write romance yourself. (Always write the types of books that you are passionate about.)
If you are not a fan but think there might be more money in it, you are really not setting off on the strongest of footings.
Assuming that genre fiction is for you, the next step is to decide which genre in particular (and which sub-genre) you intend to target. Again, your taste in reading should provide the answer here.
Having decided on your precise niche, the final step is to research it thoroughly, and you do that by studying as many novels from the niche as you can, especially recently-published ones.
The idea is to familiarize yourself with the conventions, or rules, of the genre, and you do that by analyzing what all of the novels on your reading list have in common. Then, as you write your novel...
If writing a genre novel - a horror novel, say - broadly appeals to you, but you feel that the conventions of horror fiction are too restrictive, you have the option of writing, for example, a mainstream novel about zombies, or even a literary one. Speaking of which...
As we have seen, literary fiction (and mainstream fiction, too, for that matter) can stem from genre fiction.
You simply take your book genre of choice (private eye novels, say) and then break so many of the conventions - not least the one about putting plot above all else - that it ends up being not even marketable as a genre novel.
But literary novels can cover any subject matter...
Whatever subject matter you choose for your literary novel, the prose must be of the highest quality, as must the depth of characterization and the layers of meaning beneath the story's surface (i.e. the theme).
This is the category for you if you are simply not interested in writing either genre or literary novels. In other words...
Does this sound like you?
Then the book that you write will be classified as mainstream fiction, and it will appeal to a wide audience due to its universal subject matter and its concern with good old-fashioned storytelling (as opposed to the sometimes less user-friendly storytelling of literary novels).
Like I said at the top, despite the efforts of publishers and booksellers to categorize fiction, there are actually as many types of novels as there are types of novelists.
Every novel is unique, in other words, and although it is possible to sort some books into the literary category, some into the mainstream category, and so on, there will always be novels which fall between the gaps.
If you believe your own novel might be one of the "gap" novels, I can offer this advice...
If you are unsure if your genre novel will break too many of the conventions of that genre and become a mainstream or literary novel instead, say that it is a genre novel when the time comes to sell it to a publisher.
Why? Because genre fiction is generally less of a gamble for publishers. If they think you have broken too many rules, they will either suggest some rewriting to bring it back on target, or else decide to go for the mainstream market. But in the first instance, state that it's a genre novel.
On the subject of approaching publishers - or, more likely, literary agents - be crystal clear when you state which category your novel falls into. Say it is a "seafaring historical thriller" or a "mainstream suburban drama" - something pithy like that.
Do not say, "Well, it's kind of a science fiction novel with a bit of horror thrown in. Oh, and there's plenty of romance in there, too, and a couple of murders."
All of that might well be true, but picking the most important element - the one at your novel's core - and going with that is far more likely to make publishers want to investigate the merits of your novel further.
And that is it - well, almost...
I've got just one more thing to say about the different types of novels, and here it is: Despite everything I have said above, there are actually only two types of novels - good ones and bad ones. I explore this in more depth in What Is a Novel's Purpose?.