Do I have any chance at an Ivy League school?

You do know that Ivy's only accept the cream of the crop? Which means you should be getting A's on everything, especially if it's not an honors class. And yes gym does matter because a C brings down your GPA which is crucial if you want to get into an Ivy. Atleast some of the B's you got were in honors classes which is good because they won't judge you so harshly, but the next few years you really should be getting all A's so that it increases your overall GPA (cumulative). Good Luck!