I really admire who is a foster, it's a really honorable thing to do. I know it can't be easy, but try to think that he's with a good family and he'll be treated the way he deserves. When I kept a baby kitten for some weeks, his owners then let me visit him some times and that made it easier for me.
