Facing mounting backlash, Ishiba apologized for causing 'worries' but ruled out stepping down over the matter.