The extensive deployment of commercial AITs is increasingly shaping urban environments and what was previously the realm of urban planning and governance. We highlight the importance of cities as testing grounds for proprietary and commercial AITs and the socio-political implications of such processes amidst tensions between the flattening gaze of the software and the multiple, irreducible realities of the city. To describe how AITs become part of urban planning and policies, we show how the AI promise of delivering better governance quickly turned into controversial socio-political effects. These include lack of transparency and democratic oversight, technocratic solutionism, and encoding of racial, gender, and class discrimination in algorithmic systems. We then draw on Amazon as a case study to illustrate how powerful commercial AITs are ever more frequently controlling operations crucial to urban life, such as logistics and security, and how they are thus affecting how cities function and are governed.