There is much fear involved for the spouse of someone faltering in the faith.
First I would make your husband aware that you love your life together and remain 100% committed to him - tell him and show him. Your feelings about the temple are not your feelings about him OR how much you might like to spend eternity with him.
Next, I would make a point that your values are not dependent on the church. You live your life because it is a good way to live - not because somebody told you that you had to.
Church relationships can be easy because many things can be assumed. Church relationships can also be somewhat fragile because so much was assumed to be built upon belief in the church - if the church belief crumbles where does that leave the relationship? In some ways you have to do the heavy lifting of re-establishing the relationship. How does your husband feel about various non-church topics? Are you and your husband able to respect each other on topics on which you disagree? It might take some time to work up to church topics and other sensitive issues that make us feel vulnerable.
Similar to this thought, I think it would be much harder for me to cope with the church if life was "all LDS all the time".
For me it became important to analyze, simplify, reduce, and prioritize my LDS involvement/LDS footprint. This might mean giving yourself permission to take a church break once in a while or simply saying no to callings that you would loathe (and that would consume all your free time!). The other side of this coin is to diversify! Find a social group, club, or organization that you can join. Maybe it is based around a hobby or a common interest. I remember a great hiking group that would plan once a month hikes. Fresh air, exercise, and socialization! Church is much less scary if it doesn't consume your whole life.
I hate that too. Someone once said that the true groups that are most successful at church are the saints and the liars. I find that statement true enough to be sad. I believe that is an outgrowth of our doctrine of perfectionism.