The idea that simple visual elements such as colors and lines have specific, universal associations—for example red being warm—appears rather intuitive. Such associations have formed a basis for the description of artworks since the 18^th century and are still fundamental to discourses on art today. Art historians might describe a painting where red is dominant as “warm,” “aggressive,” or “lively,” with the tacit assumption that beholders would universally associate the works’ certain key forms with specific qualities, or “aesthetic effects”. However, is this actually the case? Do we actually share similar responses to the same line or color? In this paper, we tested whether and to what extent this assumption of universality (sharing of perceived qualities) is justified. We employed—for the first time—abstract artworks as well as single elements (lines and colors) extracted from these artworks in an experiment in which participants rated the stimuli on 14 “aesthetic effect” scales derived from art literature and empirical aesthetics. To test the validity of the assumption of universality, we examined on which of the dimensions there was agreement, and investigated the influence of art expertise, comparing art historians with lay people. In one study and its replication, we found significantly lower agreement than expected. For the whole artworks, participants agreed on the effects of warm-cold, heavy-light, and happy-sad, but not on 11 other dimensions. Further, we found that the image type (artwork or its constituting elements) was a major factor influencing agreement; people agreed more on the whole artwork than on single elements. Art expertise did not play a significant role and agreement was especially low on dimensions usually of interest in empirical aesthetics (e.g., like-dislike). Our results challenge the practice of interpreting artworks based on their aesthetic effects, as these effects may not be as universal as previously thought.