You are right. Most of Panama was part of Colombia back then. That part is the province of Bocas del Toro which was part of Costa Rica. It was invaded by Colombia while the federal republic of central america still existed, but it was in a civil war at the time so it couldn't defend such a remote territory.