Initiatives such as the 2020 IEEE Global Road Damage Detection Challenge prompted extensive research in camera-based road damage detection with Deep Learning, primarily focused on improving the efficiency of road management. However, road damage detection is also relevant for automated driving to optimize passenger comfort and safety. We use the state-of-the-art object detection framework Scaled-YOLOv4 and develop two small-sized models that cope with the limited computational resources in the vehicle. With average F1 scores of 0.54 and 0.586, respectively, the models keep pace with the state-of-the-art solutions of the challenge. Since the data consists only of smartphone images, we also train expert models for autonomous driving utilizing vehicle camera data. In addition to detection, severity assessment is critical. We propose a semi-supervised learning approach based on the encodings learned by combining a class-conditional Variational Autoencoder and a Wasserstein Generative Adversarial Network to classify detected damage into different severity levels.