Well I started to think about how I felt when I was *with* him and the more I remembered my feelings then the more I thought maybe I did have feelings for him. So, before I told him this I asked him about what he meant by the "I wouldn't be happy with just being friends" and asked him to clarify. He said that he'd be really upset but not abandon me completely.
I told him I thought I might have feelings for him but left it at that. Ever since then I've been missing him more than usual, thinking about him more, thinking about how things would be if we lived closer.