What do you make of this guy I like?

I asked him out and he said he likes me too but he got out of a relationship. Anyway, I work with him. He always asks me what time I finish now he knows I like him and in the canteen. I was eating my lunch, there were three seats that were empty next to me but didn't want to sit next to me, instead sat on the other side of the canteen. When he sat down, I left the canteen as I finished and two minutes after I left, he had finished as well and found that he went upstairs and followed me. Usually he never leaves the canteen straight after he finishes his food. But he avoids my eye contact and never speaks to me. I know he's not the shy type because he liked a girl about four years ago and started chatting her up straight and flirting with her. Why not me?