I have a 47 tall in Olivia's room that I've been slacking on. I did a no-no...so let's not get into the butt-woopin I know I deserve

. I haven't tested anything in FOREVER because everything's been thriving, getting larger and looking awesome...until a couple of days ago. My xenias started
melting. I just got rid of the tang and the coral beauty a month ago which
substantially reduced my bioload. I ended up having to take out all the rock to catch the little boogers and afterwards I did a 10 gal water change because I know how stressed they got. I decided to do a 5 gal water change biweekly since all I have in there (fish-wise) are 2 clowns. So...I tested the water today and everything was great...
phosphates: 0
nitrates: 0
ammonia: 0
nitrites: 0
until I got to pH, calc & alk...
pH: 7.2!!!!
calc: 275ppm!!!!
alk: 15.4 dkh!!!!
So...I mixed up some anhydrous calcium chloride in about 2.5 gal of ro/di water (~14.5 tsp according to reef chemistry calculators to raise the calc from 275 ppm to 425ppm) so that should help my calcium. I mixed it in my top-off water and dosed about 3/4 gal tonight and will dose the rest over the course of the next 2-3 days so I don't bring up the calcium too quickly. I'm anticipating some calcium perciptation because my alk is so high but I think I'll fix that with some kalk.
So...here's my question:
Once I bring the calcium up to an acceptable level, how do I deal with the pH and the alk? I'm assuming that if I start dosing kalk instead of using Kent's CB parts A & B that will keep my calcium up and help bring my pH back up. If I leave the alk alone, will it just go down since I won't be dosing it anymore? Is there anything else I need to do?