The City of Tokyo, and the President of the Japanese Olympic Committee have signed the Host City Contract, section 33 (c) of which states:
The final dates for the holding of the Games, including the number of
days of competition and the scheduling of the Opening and Closing
Ceremonies, shall be decided by the IOC in consultation with the OCOG
(the Organising Committee of the Olympic Games)
Furthermore, section 66: Termination of Contract only entitles the IOC to terminate the contract and withdraw the Games from the Host City, not the Host City itself. Section 71: Unforseen or Undue Hardship allows the Japanese OCOG to request that the IOC consider reasonable changes, provided that they do not adversely affect the Games or the IOC, but it makes clear that the IOC is not obligated to either consider or agree to any such changes.
The IOC is also the final arbiter of any dispute between the OCOG and the Olympic Family according to section 74.
If we look to Addendum 4 to the Host City Contract, which postponed the Games to 2021, the preambulatory statements note that:
the Prime Minister of Japan Mr. Abe Shinzo has declared and guaranteed
to the IOC the full support and commitment of the Japanese Government
towards the successful staging of the Games in the year 2021
This addendum was only implemented with the agreement of the IOC, and it makes clear that any further alterations to the schedule are subject to the agreement of that same body.
The contract is governed by Swiss law, and Tokyo, the Japanese Olympic Committee, and the OCOG have all waived their ability to claim immunity against any legal action initiated by the IOC in section 87.
This all being said, the Japanese government is of course sovereign - this power being delegated to them by the Japanese people, and it's impossible for the IOC to force the Games to take place. The IOC would of course take a dim view of a Japanese attempt to unilaterally postpone the games, especially given the assurances from then Prime Minister Abe, and legal proceedings would probably be brought in the Court of Arbitration for Sport.