Instant Messaging (IM) is in addition to Web and Email the most popular service on the Internet. With xOperator we present a strategy and implementation which deeply integrates Instant Messaging networks with the Semantic Web. The xOperator concept is based on the idea of creating an overlay network of collaborative information agents on top of social IM networks. It can be queried using a controlled and easily extensible language based on AIML templates. Such a deep integration of semantic technologies and Instant Messaging bears a number of advantages and benefits for users when compared to the separated use of Semantic Web technologies and IM, the most important ones being context awareness as well as provenance and trust. We showcase how the xOperator approach naturally facilitates contacts and calendar management as well as access to large scale heterogeneous information sources.