Sharing can be done only on the record level. So, there isn’t a way to hide fields based on ownership. You could hide the data in visualforce pages, but the data could still be accessible through reports or the API.
There can be one possible solution. You can hide the field visibility from the profile and create a permission set having field level security based on the requirement and assign this permission set to the desired user.