Questions tagged [parallel-processing]
Use this tag when asking questions about multi or parallel processing - a programming paradigm used to speed up computationally intensive geoprocessing tasks by executing them over multiple CPU cores or threads.
129
questions
0
votes
0
answers
69
views
ArcPy function produced None values in multiprocessing
I'm trying to get my script in multiprocessing, but the arcpy.management.CalculateGeometryAttributes produced None values. I tried it in for loop and it worked perfectly fine.
import os
import ...
0
votes
0
answers
19
views
QGIS Server GetPrint multiprocessing documentation?
Where can we find better documention for QGIS Server GetPrint multiprocessing. Especially regarding the multiprocessing for generate multiple pdf at the same time.
I am currently running a QGIS Server ...
- 15
0
votes
0
answers
27
views
ArcGIS 106385 "Workspace or data source is read only" when using multiprocess to intersect a featureclass on a network drive
I use arcpy.analysis.Intersect to bring data (specific to my study area) housed on a network drive to my local machine for analysis. Sometimes the study areas are quite large, and so I break them up ...
- 760
2
votes
1
answer
32
views
How to deal with concurrency when inserting polygons in a SQL table in a way that they don't intersect with each other
Problem:
I have a table X, in which I am inserting polygons (ex: geometry data for a city). Before inserting them, I check if the input polygon intersects with any existing polygon in the table and ...
- 21
0
votes
1
answer
84
views
GeoDataFrame.overlay parallelization with multiprocessing
I'm trying to speed up the calculation of the intersection between two geodataframes with multiprocessing. I'm using GeoPandas with JupyterLab from Anaconda on Linux.
My multiprocessing code currently ...
0
votes
0
answers
34
views
Optimizing batch image exports in Google Earth Engine
I'm creating a pipeline to train a machine learning model to create flooding probability maps at national scales using data in GEE via the Python API. For a given known flood event, I assemble data ...
0
votes
1
answer
361
views
AttributeError: 'Dataset' object has no attribute 'rio' when using a LocalCluster from dask.distributed
I am trying to parallelize a function that uses a zarr datacube read from an s3 bucket to perform some calculations. However, when using a LocalCluster from dask.distributed, I lose the .rio accessor ...
0
votes
1
answer
29
views
Missing Outputs with ArcPy RasterToASCII and ThreadPoolExecutor
When I use ThreadPoolExecutor, I am missing a random number of expected outputs. I tried making an MRE to show what I am trying to do. The first MRE works but when I run a second MRE (which is closer ...
- 906
1
vote
1
answer
30
views
How to iterate through raster conversion concurrently while using map function in Python
I am trying to troubleshoot some issues I'm having with the Python map function that is part of a larger problem I'm having with trying to run my code concurrently. Why does map() here create an empty ...
- 906
0
votes
0
answers
240
views
Optimize the resampling and projection of multiple NetCDF files in R using terra
I have a set of 35 NetCDF files (continental scale), each containing 365 daily layers of climate data with a resolution of 5 km, s. I need to do two main tasks:
Subset, s_sub, these files based on a ...
- 45
0
votes
0
answers
104
views
Error while creating Worker Pools in GeoTIFF
I have to extract rasters from a large TIFF file (about 8MB). For this purpose I use GeoTIFF v.2.0.7 readRasters, but performance is unsatisfying:
reading rasters took: 395143.48280000035 ms
As a ...
3
votes
2
answers
202
views
Accessing layers from QgsContext transferred by completed QgsTask using PyQGIS
We have a headless QGIS standalone script that needs to run some I/O intensive layer retrieval tasks, each as parallel background threads before doing further GIS processing.
We are invoking our ...
- 275
0
votes
0
answers
71
views
Multiprocessing passing a PySwig Object
I am trying to do some multiprocessing in Python using the PySAGA module. In order to avoid having to load the input raster for each worker I am trying to pickle the loaded raster to the ...
- 11
1
vote
0
answers
152
views
Using ThreadPoolExectuor doesn't work with ArcPy
I have a Python function that downloads a geopackage from somewhere then does some stuff with it and at some point converts it to an ESRI file geodatabase.
Now inside a python script I want to use ...
- 11
3
votes
2
answers
591
views
Easy examples for multiprocessing in PyQGIS
I've gone through a large number of forum threads and cannot make sense of how possible it is to easily run two tools at the same time in PyQGIS.
Basically what I'm wanting to do is:
def function1():
...
- 109