The first thing to say is that an unreliable narrator is used in a first person point of view novel. So if you already have your heart set on writing your next novel in the third person, they won't be for you!
As readers of fiction, we expect a character telling a story in the first person to be straight and honest with us.
They can lie all they want to other characters, but not to the audience listening to them tell their tale. When a first person character breaks this trust, they are no longer reliable storytellers.
What do I mean by a narrator being "honest"? Well, the obvious thing is that they can't lie to us. But over and above that, they cannot deliberately withhold information that has a bearing on the story being told.
In short, they mustn't just tell the truth, they must tell the whole truth, too. If they don't, they will be unreliable and you will be writing a totally different novel.
It is simply one of those unwritten agreements between writers and readers of first person novels; break the agreement and your readers will feel cheated.
Now, there are two things you must be clear about here...
First, although a standard first person narrator is expected to share with us all of the relevant information in their possession, it only applies to information that they are in possession of at this particular point in the story.
When a first person narrator sits down to write a story, the story itself is over, in the past, done with. So when they write Chapter 1, for example, they already know how the novel will end.
The viewpoint character doesn't, though - for them, the story's end is still in the future.
And so, in Chapter 1, the narrator cannot withhold information that the viewpoint character is in possession of at this point in the story. But they can withhold information that the viewpoint character won't discover until the next chapter.
The second point to be clear about is that the information that the narrator - or more precisely, the viewpoint character - provides us with does not have to be correct information, but they have to believe it to be correct at the time.
So if the viewpoint character is convinced in the first chapter that his wife is cheating on him, he can state this as a fact ("my wife is having an affair"), even if it later turns out that he was mistaken.
The above, then, is a description of the conventions you need to stick to if you write a "standard" first person novel. But if you write a first person novel using an unreliable central character, all that changes.
A first person novel using an unreliable narrator actually makes use of the unwritten convention between writers and readers that the narrator should be honest.
When you write an "unreliable" novel, there will be no indication to start out with that the narrator of this particular novel is different to any other first person narrator - and so the reader will naturally trust them.
It won't be until the story begins to unfold that the reader will start to suspect that everything is not as it seems.
Unreliable characters can be either deceitful or deluded. The tricky part is letting your readers know this - not an easy task when you consider that the narrator is the only one telling the story, and that they are not going to admit to being deceitful or deluded.
(As a matter of fact, if they are deluded they probably won't even know it themselves).
In short, using a narrator who isn't reliable is a difficult trick to pull off. More to the point, you have to ask yourself, why bother using one at all?
The answer is that most, if not all, of you reading this won't want to use one. You will want to stick to using standard narrative techniques.
But if you wanted to write a novel that was an exploration of insanity, say, or the way we can delude ourselves, or how we can all start to believe our own lies if we lie too much, using an unreliable narrator could be the perfect viewpoint choice for you.