You're right about the Ministerpräsident, but the CSU is exclusively active in Bavaria. In every other federal state the CDU (kind of) takes that place, because they formed a union. Reasons for that are complicated, but during the time of the Weimar Republic 1919-1933 Bavaria had 2/3 major parties, that were exclusively active in Bavaria (BVP, BBP). After the war this trend at first somewhat seemed to go on (see the Bayernpartei), but to combat this, conservatives agreed on this split, that Bavaria has his own party, that is in union with the federal CDU.
but there is a lot more...