Should I leave the choir, my CCA?

If you have any options for other choirs you could join, I would say join them, and drop this one. If there isn't, then see if there is any way that your director could be more regular with timing. I can't imagine that you would need all of a persons free time, just for choir. Although I understand what you mean about the sadness and joy, I kinda feel the same with my choir, but for different reasons. But for some reason, I'm staying with it. I feel like it will be worth it in the end. Maybe it will for you too. :)