I made an introductory talk on word embeddings in the past and this write-up is an extended version of the part about philosophical ideas behind word vectors.
C. Xu, Y. Bai, J. Bian, B. Gao, G. Wang, X. Liu, and T. Liu. Proceedings of the 23rd ACM International Conference on Conference on Information and Knowledge Management, page 1219--1228. ACM, (2014)