The timing of such things are neither person???s fault, but blame is often given, and often not forgiven.You???re mostly afraid of losing the right to see your grandchildren, but that is not likely if you handle it well, by not attacking him or his beliefs, just sharing your own feelings about your relationship.