We study mixing by chaotic advection in open flow systems, where the corresponding small‐scale structures are created by means of the stretching and folding property of chaotic flows. The systems we consider contain an inlet and an outlet flow region as well as a mixing region and are characterized by constant in‐ and outflow of fluid particles. The evolution of a mass distribution in the open system is described via a transfer operator. The spatially discretized approximation of the transfer operator defines the transition matrix of an absorbing Markov chain restricted to finite transient states. We study the underlying mixing processes via this substochastic transition matrix. We conduct parameter studies for example systems with two differently colored fluids. We quantify the mixing of the resulting patterns by several mixing measures. In case of chaotic advection the transport processes in the open system are organized by the chaotic saddle and its stable and unstable manifolds. We extract these structures directly from leading eigenvectors of the transition matrix.