This certainly is a valid point. Doing what you are proposing if you are could be seen to be doing something that wasn't the best interests of the Company or the shareholders. Depends who these people are but, the partner, could try and establish an argument that you weren't acting in the best interests of the Company. This could have ramifications for you.
The best option surely would be to get the Company valued and then try and buy his shares in the business from him legally and above board?
If you have questions in this respect from a legal point of view would suggest dropping me a PM