As hard as it is to realize the falsity of those memories, doesn't it speak well of you that you became part of his legend? You were suited for it, and the world deemed you worthy of him. If anything, it might be just as painful to have been there, but to have been forgotten and left behind.