After a drab first half, the game picked up after the restart and the first flashpoint came in the 65th minute when Ballard was adjudged to have fouled Luka Lochoshvili when challenging for a cross after a VAR review.
Pierce saved the penalty from Paris St-Germain winger Kvaratskhelia, and Northern Ireland had an appeal of their own waved away when Charles appeared to have been caught in the area in the 88th minute.
The NI manager was also angry at the lack of a foul on substitute O'Neill, who is now tied to Northern Ireland for three years after he won his third cap, and Ballard was injured when chasing a loose ball as Georgia came forward on the counter-attack in the resulting play.
O'Neill said his players were "magnificent" and his team "showed every side of our character tonight".
"At times it didn't go our way. I've not seen the incident back but to me it seemed incredibly harsh to give a penalty for that. The guy was nowhere near the ball to head it.
"We feel aggrieved that we didn't get one at the other end. The foul on Ceadach when he came on the pitch causes Dan to get injured.
"I was just baffled by the quality of the refereeing."
O'Neill confirmed that Sunderland defender Ballard had injured his hamstring and would miss Northern Ireland's next three games in the window, against Hungary, Ukraine and Georgia.
"You saw the nature of how he pulled up. It's a hamstring," he said.
"We'll scan him tomorrow but he's certainly not going to be available for the next three games."
Extracted and lightly reformatted for readability. · Source: pt