Republican presidential nominee Donald Trump says Democrat rival Hillary Clinton's performance at a forum on US security proves she is unfit to be president.