Well, here's my take on this.
Just a warning, I had planned this before opikenopi posted the backstory guidelines, so if this doesn't fit that's why.
So, feys... it always comes down to babies, doesn't it?
A woman a few generations ago (2, 3?) made a pact with a fey lord. She loved a noble man, a strong and powerful knight, but he did not return the feeling. She made a pact with the fey: he would give him a potion that would cast a spell on the knight, so that for one night she would be his lover (a lot like Lancelot was deceived by Elaine and on that single night engendered Galahad). In the morning, the knight would still not know what had happened and would not know her. But she would be pregnant from the spell, and in exchange, the fey only asked for the child that would be born. After the birth, he would be able to cast the second part of the spell that would cast so strong a love on the knight that would bind her to the woman for his whole life.
The woman accepted to the terms. By some means (I think it's easy to set this up if she moves in the right society circles) she manages to poison the drink of the knight and indeed take advantage of the spell to spend a night in his bed. She left before dawn, quite certain she had a seed in her belly.
But after nine months, when the time came to give back the baby to the fey, she refused it. She stole a baby from one of the peasants in her property, therefore of much lowerblood than the promised baby, and she gave that instead to the fey.
What the fey had neglected to tell her was that the baby was an important material component of the second part of the spell, more exactly, the baby's blood. Because it would be of noble origin, it would somehow carry a stronger essence within it than the peasant baby's (sort of like the higher blood in the Birthright setting). It would therefore cast a reinforcement, a strenghtening of some feelings. Because it was meant also to be strongly related to the blood of the two targets (the knight and the noble woman), their binding would be effective.
Unaware of the exchange, the fey performed the spell and it backfired horribly. Instead of casting a spell of love and potency on the couple, it cast a spell of deceit and failure. Instead of strenghening their selves, it cast a waning on them, and because their child was still alive, on the child too.
The elf understood the error then and explained the results. Curse or spell, here they are: everything owned by them would lose its vital strength over time. Its essence would not be what it appeared.
It began with the sheep in the property. Their milk seemed white, but was in effect water. Over time, it expanded to the human household: the womans would not give milk, only water would come out of their breasts. Eventually, the cereals in the fields would crumble to sand instead of flour, and in time, even the house began to crumble down, as the rock lost its strength and the pillars and columns stopped supporting the buildings.
The couple grew older sooner than expected, their bones became brittle and their muscles lost their power. The man became impotent.
The knight and the woman died eventually of their own wasting. The child, less affected, survived to adulthood, to getting married and having children of their own. They still carry the taint.
Her descendents take the curse with them. After they settle in some place, the curse begins to manifest in their property around them, taking first the weakest things then the strongest ones. The house is always the last to be affected, but without crops of flocks wealth doesn't last much. It always takes time to manifest, this is a very slow effect, but it's there and they will notice in the end.
Your paladin, of course, is a descendent of this line. The stories of their past wealth are true, though why they lost it is a shameful secret.
Possible Variants:
* There are some changes I considered in this story. The spell could affect only the true fathers of the sacrificed child, that is, the servant. In this scenario, for added drama, the servant might not even belong to the household of the culprit woman who then did not need to be wealthy, but to that of your target family. It would then be affected by a wrong acted upon servants of their own, a tragedy they would not be responsible for.
* Another change in this direction that takes the curse to someone else than the originator is that someone in your target family actually marries someone of the tainted line. If I DMed this, I'd actually pick one of these two, only because I enjoy the twist that the curse is really unfair and it would freak the paladin out to discover and try to correct that.
* Another variant: the fey actually requests that the woman be pregnant from him in exchange for the spell he would cast later, and that she give him his son. When she gives him a false one, playing the classic trick of feys on them, the elf lays the curse on her and her true (and his too) son, thus sacrificing his offspring for a retributive punishment. The reason that this woman must be wealthy is that this is the ancestor of your paladin's family and so the curse would easily lead her from wealth to poverty and to extend down the line as described above.
Whatever suits you best

I hope this is not too rated for these forums :-/
Alex