manifest.json contains list of your dependencies(the packages you have used for your project). so if You clone your repository on a different machine, unity uses this list to download the required packages.
short answer: manifestYou should not put Packages in your .json is importantgitignore. It should be committed along with the rest of your project files.