Modded Server Coming Up In Vanilla Server Instead

hey, I'm having trouble setting up a server for me and a friend to play in. the server itself works fine but it says that it is a vanilla server when it is supposed to be modded. there is nothing wrong with my pc. i do not want any site to be hosting it because you have to pay to renew each time.
please help on this. thank you