4

Currently my company is storing the installers (and in some cases the installed directory copy) for some open source third party tools that our build uses. Were storing these files in our github repo which makes it very large and slow. I want to clean this up since the files themselves are not needed directly for the build.

I believe that the reason why these files were originally checked into github was because the developers must have feared that the third party tools might become unavailable in the future. For example a user just decides to delete their github account, where the releases are currently posted.

Now I realize the risk of this happening is infinitesimal, but at the same time some of the third party tools are pretty obscure. What would be a best practice to reasonably address this fear?

Improve this question
6
  • 6
    The risk is not as small as you think.
    – Greg Burghardt
    Commented Jun 15, 2022 at 20:40
  • What tech stack are you working in?
    – Greg Burghardt
    Commented Jun 15, 2022 at 20:41
  • C/C++/C# and some Python
    – Slack Groverglow
    Commented Jun 16, 2022 at 15:30
  • 2
    If you need a specific old version with which you created a historical product (for repair), then Git seems fine and clear, same methodology and risk factor. Even though there is no compression. You could make two repos.
    – Joop Eggen
    Commented Jun 17, 2022 at 12:23
  • shallow clones may be helpful.
    – Thorbjørn Ravn Andersen
    Commented Jun 21, 2022 at 22:24

2 Answers 2

Reset to default
7

It makes fully sense to store and keep installers of any third party tools you are using in your development process (not just your build process) in a location under your control. It does not matter whether those tools are open source or closed source - when your software system is going to be used for a few years, expect some of your tools to vanish from the web during that time. Without keeping a copy of the installers, you will likely be screwed when you try to setup a new developers machine.

However, in case these tools are large binary files, Git alone is not well suited for this task. Instead, you could either use some shared network folder or Git LFS. You may find more information looking through questions on this site (or stackoverflow) which contain the keywords "git binary files".

Improve this answer
1
  • 1
    +1 We used AWS S3 for this on my last project, worked well.
    – mmathis
    Commented Jun 17, 2022 at 2:06
3

Consider storing the third party tool's source code and binary etc in its own separate Git repo, but still under your control. Then just download the built binary during builds. If both projects live in Github (or similar), the download should be fast.

Another option, if you're using Docker or Kubernetes images, is to add the built binary to an image. The build, storage, and dependency management systems for Docker/Kubernetes are optimized for binaries. I recall Amazon having its own storage system for versioned Docker image layers that played nicely with other build and deployment tools.

Improve this answer

Not the answer you're looking for? Browse other questions tagged or ask your own question.