Because the main file for the dummy-executable and the actual compile test were both called main.cpp, they were overwriting each other during in-source builds.