Technological advancements in the virtual assistants' domain pave the way to implement complex autonomous agents like diagnostic chatbots. Drawing on the assumption that chatbots are perceived as both technological tools and social actors, we aim to create a deep understanding of trust-building processes towards diagnostic chatbots compared to trust in medical professionals. We conducted a laboratory experiment in which participants interacted either with a diagnostic chatbot only or with an additional telemedicine professional before we interviewed them primarily on trust-building factors. We identified numerous software-related, user-related, and environment-related factors and derived a model of the initial trust-building process. The results support our assumption that it is equally essential to consider dimensions of physician and technology trust. One significant finding is that trust in a chatbot arises cognitively, while trusting a human agent is affect-based. We argue that the lack of affect-based trust inhibits the willingness to rely on diagnostic chatbots and facilitates the user's desire to keep control. Considering dimensions from doctor-patient trust, we found evidence that a chatbot's communication competencies are more important than empathic reactions as the latter may evoke incredibility feelings. To verify our findings, we applied the derived code system in a larger online survey.