Add "yellow box" mode to pre-existing quiz?

I've created a quiz, but upon revisiting it, it seems that having the yellow-box mode enabled would be a much more reasonable level of difficulty than just letting people spam answers.

I go into the quiz editor and change Yellow Box to Yes, but that hasn't actually changed anything. This was yesterday, and it still hasn't updated. Any ideas what I might have missed?