Organize photos for many overlapping projects

Question

I am working with a larger collection of photographs (~10,000) for my PhD thesis (in biology) and I am currently struggling with the organization of these. Sorry, if this question is in the wrong stackexchange community, I just thought that if someone people here should know about organization of photos.

My collection:

To elaborate a bit more on what I would like to achieve: I am currently doing my PhD working and need to take a lot of pictures to identify or protocol different plants. Since my project is divided in 4-5 different sub-projects (each with further sub-divisions) it is hard to keep an overview. Also, some pictures are of general interest for my project and so should be linked not only to the corresponding sub-project but as well to the whole. Additionally, I am awaiting more photos to pile up as this was only the first out of three field seasons...

My question:

• How would you organize such as a photographic library?

• What kind of folder structure would you suggest?

If I am searching in this forum (eg. here or here) or in any search engine, mostly ideas for daily life or professional pictures come up. But these use a very different kind of organization from what I need. For example, it is of little use to me to organize my pictures into people, places, events and dates as often suggested (eg. here).

Regarding folder structure, I thought about sort pictures corresponding to days (eg. 2019-06-23). But I am afraid this would get to messy!? Or maybe I could sort pictures into the sub-projects and use tags in the metadata to link them to other sub-projects?

Side notes:

• I am running a Linux machine (Manjaro, specifically) and so far I tried out digiKam for organization of my library. But I guess the OS is only of secondary relevance here (ie. each OS should have at least some software to deal with this kind of problem).

• For now I do not need to sync my files between devices.

• Usually, I do not need to edit my photos. Only maybe 1-2 for the cover later...

Answer 1

First step should be to conduct a detailed review of the data to establish solid project expectations so you can make these sort of decisions with confidence.

Key thing we're looking to answer is:

• How much overlap can an image have between sub-projects?
• How much 'meta data' do we want to embed in the file structure vs an external storage/library file?
• How much work would we want to do by ourselves, vs how much would we rather push off onto the computer to do for us...

---

Personally I would use Windows with Lightroom as an image library management solution, but this is less than ideal if you intend to remain within a Linux ecosystem.

However tools like Lightroom are kind of a bloated option with a lot of extra features that we probably don't need for this kind of project.

In a linux environment we may be better off scripting much of the handling ourselves over relying on ready-made tools. [It is also an excellent skill building task that gives highly useful experience in data-management.]

---

Manually sorting images into folders is 'less than ideal', prone to error, and awkward to reliably correct. This is especially true if there are large overlaps in sub-projects that any given image is likely to be involved in, or if you decide a major change is required later on.

• When dealing with data, we can waste our time, or we can waste a computers time. Choose wisely.

Keywording and Database interaction is a far more robust option than getting overly complicated with folders. Unless there is effectively zero overlap in sub-projects that an image is included in, then it is far better to allow the computer to "do the sorting" for us.

Keep the core of the archive simple with a standard timestamp based file structure.

Project/Year/Month/Day/[timestamped_filename]

or even just

Project/Year_Month_Day/[timestamped_filename]

From here we are going to want either existing software that can act like Lightroom or another Image Cataloging software, or create scripts to deal with things for us.

General workflow would be:

• Import images [or their file names] into a database. [flagged as 'new']
• Keyword and add meta-data as required by project. [Once finished, remove 'new' flag. Even in something like Lightroom we want to maintain a clear indicator of whether the entry for a given image is 'finished' or if it needs more work before it is ready to move forward within the project.]
• Define 'Views' of the data based on the above keywording and metadata to select the specific images needed for a given state of the project.

Project file structure then looks like:

\Project\
-\Core Image folder\[Subfolders]
-\Library, database, or Metadata\
-\Temporary Exports or 'views' folders\ {Flexible data generated on the fly as needed}

If you are comfortable with scripting and simple databases, then it is fairly easy to build out a basic toolchain yourself to generate view folders that contain symlinks back to the original source image if you don't require robust image review/editing tools from a more complex piece of software along the lines of Lightroom.

The specifics on how to implement something like this yourself easily vary, but the heart of it would be to define your target for a specific grouping based on a database query, which you then pass through a file management script.

Select all images that had keyword Alpha to create a list, and use that list to pull copies/links into a Sub-Project View Folder without changing data stored in the Core Image Folder.

If requirements of a sub-project change, then the old folder can be deleted and a replacement generated on the fly without having to manually copy and paste files to or from folders.

---

Remember to backup your core data and metadata! Exports/views can be regenerated on the fly as needed, but you want to make sure the original data is hard to lose or corrupt.

Answer 2

Consider using hard links in multiple folders with different methods of organization, as xenoid comments. That way, you can use whatever method is most convenient to find the images you want.

1. Organized by Date/location.

   YYYYMMDD (City, ST) Location - Description/YYYYMMDD-HHMMSS Description.jpg
   

2. Organized by Taxonomy, as Stan ponders.

3. Organized by Project.

   Projects/Title/Images/YYYYMMDD-HHMMSS Description.jpg
   

4. Any other organizational method you like.

To create the links, you can use cp -l src/ dst/. (ln is less convenient because it works on a per file basis.) Then use your favorite file manager to move files from dst/ to whereever you want them. To preserve links when backing up to external drives, you can use rsync. If you make a mistake and copy some files instead of linking them, existing tools, such as hardlink, can find and relink them.

Since they are hard links, advantages include:

• There is no need to keep track of special locations for images that are shared among multiple projects. Just create new links within each project folder using any existing link that is convenient.

• You can name and organize files independently of each other as needed.

• Changes to one file will affect all the others. File contents will be kept in sync. (Covers and specially edited images should be copied first to unlink them from the original.)

• The amount of hard drive space will not significantly increase as a result of having multiple "copies" in multiple locations.

• You don't have to worry about broken links, as you would with soft links.

• When you copy or archive folders (eg, to send to colleagues), the files will be copied in their entirety. (No need to track down "originals".)

• Any software that can access the filesystem can be used to work with the files.

• Tags embedded in the file as XMP metadata will be synced across all copies and move with the file, as xenoid notes. However, some software may not respect such tags.
• No external database to keep in sync when you move files.

Disadvantages:

• If you want to delete an image, you'll have to track down and delete each of the hard links. There may be tools to assist the process. At least ls tells you how many links are left.

Answer 3

I'd use adobe lightroom personally. You can use it to make keywords in a hierarchy to tag images for the various phases. It also let's you bulk tag images w/ keywords which I think would save you some time. It also gives you smart collection to autogenerate collections based on those keywords aka smart collections. so you could have a smart collection for each phase / subphase. I'm sure you could do the same thing w/ free solutions but it might turn into a kludge and it might be worth paying for the lightroom package thats like 10$/month in the US. You might be able to find a used version of it like version 4 or 5.