Process for archiving binaries from different repos using Git?

Question

Working in the embedded systems space, our products usually require multiple binaries for each of the microcontrollers within a single product. Using SVN previously, we would bundle validation and production releases into folders that contain all the necessary binaries to be loaded, e.g., v01_00_00, v01_00_01, and so on. This was primarily to give the validation and production teams a simple way to get all the files they need to assemble a specific build of the product.

Our initial thinking is to simply keep the same process in Git. We would maintain a repository that contains nothing but tags that contain these binaries, but it seems sloppy to create orphan branches and tag them. Is there an intended method of applying Git for this type of use-case?

Answer 1

One important difference between SVN and Git is that in SVN, all files reside on a central server and only some revisions are checked out. In contrast, Git downloads the complete repository. While this allows Git to function as a decentralized version control system, it also means that it is preferable to keep the repository small.

Because size matters, checking in (large) binaries into Git is not a great idea. They will add up over multiple versions to significant storage. Whether or not this is going to matter depends of course on the total number of binaries and their size – run the numbers first to see whether this matters.

It is therefore correct to store the binaries externally. E.g. Git-LFS (large file storage) is a widely supported git extension to do exactly this. However, the server might need additional configuration. In particular, hosting services like GitHub charge extra for LFS storage. Git-Annex is another implementation of this idea.

Your idea to use a separate repository doesn't quite solve your problem but just moves it to a different repository. Sometimes that is good enough – I've done it as well. As you are here using Git as a storage mechanism and not for source control management, it is completely legitimate to ignore normal best practices. Lots of orphan branches? No problem.

But because you are not making use of Git, there's no strong reason to use a Git repository to out-source your blobs at all. If you don't want to use LFS, another possibility would be to upload the binaries to some server (but never modify these files!) and check in the names/URLs into your repository. Your build process would then download the binaries, possibly caching them locally. This is quite similar to how LFS works, except that LFS is part of Git and not part of your build process, and that LFS uses content-addressable storage: instead of specifying names/URls, the hash of the object is used.