running a multi-threaded script on a macPro to a smb-mount folder throws an error

Question

We're trying to run a multi-threaded script from our macPro but we keep getting an error massage.

The xml script contains parameters to run a specific tool, the paths to the raw data and other running parameters.

We tested the exact same command with the script and the raw data files saved locally on the computer and the script runs without a problem, so we know both the script and the tool should be working.

When we put the raw data on a smb-mounted folder (we would like to test it to save storage on the local computer as well as save copying time), we get an error message that the process can't access the file, as it being used by a different process

% dotnet net7/MaxQuant/bin/MaxQuantCmd.dll orbitrapDDA/mqpar.mac.xml  
Configuring 
Assemble run info 
Unhandled exception. System.Exception: Exception during execution of external process: 729 Unhandled exception. System.IO.IOException: The process cannot access the file '/Volumes/pool-cox-testing/MaxQuant_testing_Mac/orbitrapDDA/mqpar.mac.xml' because it is being used by another process.
   at Microsoft.Win32.SafeHandles.SafeFileHandle.Init(String path, FileMode mode, FileAccess access, FileShare share, FileOptions options, Int64 preallocationSize, Int64& fileLength, UnixFileMode& filePermissions)
   at Microsoft.Win32.SafeHandles.SafeFileHandle.Open(String fullPath, FileMode mode, FileAccess access, FileShare share, FileOptions options, Int64 preallocationSize, UnixFileMode openPermissions, Int64& fileLength, UnixFileMode& filePermissions, Func`4 createOpenException)
   at System.IO.Strategies.OSFileStreamStrategy..ctor(String path, FileMode mode, FileAccess access, FileShare share, FileOptions options, Int64 preallocationSize, Nullable`1 unixCreateMode)
   at System.IO.StreamReader.ValidateArgsAndOpenPath(String path, Encoding encoding, Int32 bufferSize)
   at System.IO.StreamReader..ctor(String path)
   at MaxQuantLibS.Domains.Peptides.Basic.MaxQuantParams.ReadVersion(String filePath) in C:\Repositories\net7\net\MaxQuantLibS\Domains\Peptides\Basic\MaxQuantParams.cs:line 661
   at MaxQuantLibS.Domains.Peptides.Basic.MaxQuantParams..ctor(String filePath) in C:\Repositories\net7\net\MaxQuantLibS\Domains\Peptides\Basic\MaxQuantParams.cs:line 655
   at MaxQuantLibS.Domains.Peptides.Basic.MaxQuantParams.Read(String filename) in C:\Repositories\net7\net\MaxQuantLibS\Domains\Peptides\Basic\MaxQuantParams.cs:line 575
   at MaxQuantLibS.Domains.Peptides.Features.RunInfo.AssembleRunInfo(String mqparFile, Int32 fileIndex) in C:\Repositories\net7\net\MaxQuantLibS\Domains\Peptides\Features\RunInfo.cs:line 30
   at MaxQuantLibS.Domains.Peptides.Work.AssembleRunInfo.Calculation(String[] args, Responder responder) in C:\Repositories\net7\net\MaxQuantLibS\Domains\Peptides\Work\AssembleRunInfo.cs:line 17
   at MaxQuantLibS.Domains.Peptides.Work.MaxQuantWorkDispatcherUtil.PerformTask(Int32 taskType, String[] args, Responder responder) in C:\Repositories\net7\net\MaxQuantLibS\Domains\Peptides\Work\MaxQuantWorkDispatcherUtil.cs:line 9
   at MaxQuantLibS.Base.MaxQuantUtils.Run(Int32 softwareId, Int32 taskType, String[] args, Responder responder) in C:\Repositories\net7\net\MaxQuantLibS\Base\MaxQuantUtils.cs:line 377
   at MaxQuantTask.Program.Function(String[] args, Responder responder) in C:\Repositories\net7\net\MaxQuantTask\Program.cs:line 18
   at Utils.Util.ExternalProcess.Run(String[] args, Boolean debug) in C:\Repositories\net7\net\Utils\Util\ExternalProcess.cs:line 11
   at MaxQuantTask.Program.Main(String[] args) in C:\Repositories\net7\net\MaxQuantTask\Program.cs:line 12

   at QueueingSystem.WorkDispatcher.ProcessSingleRunExternalProcess(Int32 taskIndex, Int32 threadIndex)
   at QueueingSystem.WorkDispatcher.DoWork(Int32 taskIndex, Int32 threadIndex)
   at QueueingSystem.WorkDispatcher.Work(Object threadIndex)
   at System.Threading.Thread.StartCallback()
zsh: abort      dotnet net7/MaxQuant/bin/MaxQuantCmd.dll orbitrapDDA/mqpar.mac.xml

we assume, it has probably something to do with the smb mount, but we can't pin-point the problem or how to solve it.

Answer 1

ok, so Multi-threading is "hard", and that is primarily due to issues that fall under the category "Concurrency".

Files are subject to concurrency issues, and as such every OS implements some form of locking, to ensure that threads that wish to write to the file can do so without risk that their changes will be undone by another thread that already had the file open, and that every thread that reads the data can be assured that they have the most current version of the file at all times.

as such, you cannot have multiple threads open a file at the same time.