JomSocial 2.8.x should not be installed on Joomla 3.x in the first place, but if you have installed it, you'll need to remove everything before you can upgrade to JomSocial 3.x, to make it work properly. For more information, please visit our documentation page