Ever since modern-day financial markets existed, people have been trying to forecast movements in stock prices, as accurate predictions would entail economic benefits and the reduction of risks. This paper examines whether social media sentiment can be used to predict short-term stock movements. Using more than two years of data from Twitter, we assess the effect the extracted sentiment holds for 10 companies listed in the S&P500. Applying different sentiment analysis approaches and forecasting models, we find that for three out of the ten companies, sentiment does significantly improve the forecasting performance. A custom-built sentiment model outperforms an off-the-shelf VADER model, and tree-based models deliver better performance than linear ones. On the theoretical front, this provides evidence against the Efficient Market Hypothesis and warrants future research regarding the circumstances under which stock returns might be predictable.