It sounds like he is actually a little bit afraid of strangers, as exemplified by his furious barking when they get there. Often fear turns into a bit of defensive aggression, so as the people walk away, he can sort of "chase them away." Your presence sort of emboldens his bravery, and he is also finding out that the visitors always leave when he does this, so it works to get rid of the people who are making him uncomfortable!
So what I would say the way to go after this problem is to increase his comfort level with strangers. You could do that by having visitors get on the floor with him and toss him treats. If pup is willing to take treats from visitors hands, that's even better.
Then, when visitors are ready to leave, put pup on a leash to prevent him from "chasing them away" as that is a sort of self-reinforcing behavior and its best to prevent it.
So overall the theme would be to help him to enjoy visitors, increase his comfort with them, associate strangers with good things (treats) and prevent the unwanted behavior.
Please be skeptical of any advice you might get that involves punishment or corrections for the undesired behaviors. Anything negative will just reinforce for pup that strangers are dangerous and bad things happen when visitors come over, and will therefore increase his fear and increase his defensive-aggressive behaviors.
Good luck and do let us know how things go as you try to improve this situation!