From The Art of Fiction by John Gardner:
"When one begins to be persuaded that certain things must never be done in fiction and certain other things must always be done, one has entered the first stage of aesthetic arthritis, the disease that ends up in pedantic rigidity and the atrophy of intuition. Every true work of art—and thus every attempt at art--must be judged primarily, though not exclusively, by its own laws. If it has no laws, or if its laws are incoherent, it fails—usually—on that basis."
And this paragraph encapsulates, I think, the dilemma that keeps many writers, especially writers who are trying to get their work read or published, tossing and turning at night. In our most fabulous imagining, someone will take our novel at face value. They’ll understand all the thematic elements, love the pace, totally “get” everything we were trying to do. It’s like the perfect relationship: they’ll love everything about it, unconditionally, and they won’t try to change it.
I like thinking about that other type of writer, the one who is merely a product of writing courses, a pious student of things that “must always be done.” It’s fun to think of him wasting away with the disease of “pedantic rigidity and the atrophy of intuition.” But then again, he’s probably cashing his checks, too, because he has stayed true to his brand, has given his faithful readers a recognizable plot and familiar characters in each best-selling book.
It is a fine and valiant thing to have a vision for your writing and to remain faithful. But if you’d like your book to be read (and most of us do), then the difficulty lies in finding a middle road. Acknowledging an audience and their expectations (even those driven by ideas of what "must always be done") while remembering the cues, the ideas, the inspiration that began the project. Staying true to your own laws but making them livable for others. A book is like a party guest. Make it too aloof, too unlikeable, too tedious, too conceited, etc., etc…and everyone will leave the room. Of course, a party is a subjective thing. Some people don't like them at all.