19
votes
Accepted
Explaining long computing time of Python code
Here are some issues with your script:
I've already learned that using GeoJSON for analyzing is a too bad idea. The script below that takes 20 seconds when I use GeoPackage takes about 8-10 hours ...
- 77.3k
18
votes
Finding nearest neighbor for each point in the same table
This is essentially a duplicate question of multiple others, with the sole difference being a table self-join.
However, all queries currently present in this post have delicate CRS misunderstandings, ...
- 30.4k
16
votes
QGIS slow performance with PostgreSQL/PostGIS
This is a broad question... here are a few things I have gathered through the years:
Make sure the geom/geog column is using a single type (ex avoid geometry but instead use geometry(point,4326)). ...
- 42.3k
15
votes
Faster calculation of cumulative sum in attribute table in QGIS
You can use the script below in QGIS Python Editor. It takes about 45 seconds (including saving to the source file) for 1.5 million features in SSD or HDD. Select the layer first.
import time
from ...
- 77.3k
14
votes
Accepted
QGIS form performance
It seems to be a bug in QGIS that has existed for some time. I've tested versions 2.18, 3.4 and 3.10.
When using a bigint as a primary key, QGIS will cast the primary key to a text field. This causes ...
- 1,357
12
votes
Fastest way to convert big raster to polyline using R or Python?
Edit: In the latest version of sf (v0.9.1) and stars (v0.4.1), you need to specify the merge = TRUE argument in the call to sf::st_as_stars() function.
For posterity, I've been having success with ...
- 315
12
votes
Accepted
What can be a reason of QGIS slow work?
First, I would try with a fresh profile. Go to Settings -> User Profiles - New Profile. This is basically like a fresh QGIS install - no Plugins are loaded, no (maybe wrong) settings you changed ...
- 3,022
11
votes
Accepted
How to improve performance of nested search cursors?
I have to keep posting that I will never use embedded search cursors ever for relating tables to each other. Only dictionaries should be used to do what you are doing. Performance will increase 100 ...
- 8,565
11
votes
Accepted
Alias for postgis function is really slow
You should use an SQL wrapper instead. plpgsql wrappers tend to be slower since they are not inlined and you are forcing an intermediary step too which can get slow the bigger the geometry.
So write ...
- 10.6k
11
votes
Accepted
Only union/dissolve intersecting or adjacent features to speed up query
I would look into using ST_ClusterDBSCAN. I have had tremendous success using this function to solve many cluster like geometric problems.
WITH clusters
AS(select st_clusterdbscan(geom, 0, 2) over()...
- 4,536
10
votes
QGIS rendering performance bottleneck?
There are two issues you have to consider:
Whether your data has spatial index or not. If not, then try to create a spatial index to your data. The spatial index can improve the speed of rendering ...
- 41.1k
10
votes
Disjoint is very slow in Virtual Layer but fast on symbology
The issue is the cross join on non-intersection. Your query reads:
for every row in g, return every row in s that does not intersect with the current g
which seems in fact what you want, but about ...
- 30.4k
9
votes
QGIS rendering performance bottleneck?
Make sure that under Settings -> Options -> Rendering you have "Render layers in parallel" checked.
- 27.8k
9
votes
ArcPy UpdateCursor auto pausing after every 1000 records
By default the auto commit interval is set to 1000. You can change the interval using the following:
arcpy.env.autoCommit = 5000
- 116
9
votes
Accepted
Measuring loading time of each layer when opening a QGIS project
Since QGIS 3.16 you can know the time it takes to load each layer from a QGIS project.
Go to View --> Panels --> Debugging/Development Tools and click on the Profiler in the left menu.
As soon ...
- 36.6k
8
votes
Increasing speed of crop, mask, & extract raster by many polygons in R?
This question is very old so my answer is just for posterities. Since 2020-11-20 there is the new exactextractr package which implement the exact_extract() function, which is the new benchmark for ...
- 298
8
votes
Speeding up QGIS spatial join?
Add a spatial index to each of your shapefile layers:
Layer > Properties... > General then Create spatial index.
Another way is to create them for all the layers in one go:
Vector > Data ...
- 885
8
votes
Accepted
Importing CSV (via VRT) to PostGIS is slow with -skipfailures?
You don't have to use a .vrt any more. ogr2ogr supports reading csv files with geometry directly since version 2.1.
The ogr2ogr command:
ogr2ogr -f "PostgreSQL" PG:"host=000.000.000.000 port=0000 ...
- 16.6k
8
votes
Accepted
Fastest way to extract a raster in R (improve the time of my reproducible code)
I get much faster results with velox if I crop the raster before running extract, e.g.:
r <- velox("testras_so.tif")
r$crop(poly)
r$extract(poly)
I've also been working on a package with ...
- 13.2k
8
votes
Accepted
Accelerating GeoPandas for selecting points inside polygon
With the help of @Taras I could solve my efficiency problem.
As the first try, I thought maybe the tqdm library is causing this delay. So I modified my code to
within_points = [points.geometry[i]....
- 251
8
votes
Multi-threaded compression in rasterio?
Yes, although this seems not to be mentioned in the documentation.
If you include num_threads=8 or num_threads='all_cpus' as an argument to rasterio.open then multithreading will be enabled (for ...
- 476
7
votes
Which is the faster way to copy data to another feature class: Feature Class to Feature Class or Insert Cursor
I created three sample random point datasets - 10 features, 100k features, 1m features - and tested the following options
Copy (arcpy.Copy_management())
Insert Cursor (arcpy.da.InsertCursor())
...
- 29.8k
7
votes
Prevent SAGA in QGIS 3.0 from creating certain outputs?
Should be possible to do if you feel like hacking your qgis install.
look for this file in your qgis install: https://github.com/qgis/QGIS/blob/master/python/plugins/processing/algs/saga/description/...
- 27.8k
7
votes
QGIS more than 3000 raster works slowly
You may see considerable benefit if you load them into a single virtual raster (vrt). You can do that through the processing toolbox, by searching for "build vrt"
- 27.8k
7
votes
Accepted
PostGIS ST_Intersects is slow and doesn't seem to use spatial index
Try
WITH
poly AS (
SELECT geom
FROM admin_boundary
WHERE id = <id>
)
UPDATE highway_only_motor AS a
SET has_business_value = true
FROM poly AS b
WHERE ...
- 30.4k
6
votes
Avoiding GeoServer/Java out of heap space error?
I've had the same error message on a Windows Server with GeoServer using Jetty. After some research I found the file C:\Program Files (x86)\GeoServer 2.10.1\wrapper\wrapper.conf where I edited these ...
- 161
6
votes
Accepted
Deleting many duplicate points using ArcGIS Desktop?
I found that this script copies unique points into separate originally empty shapefile about 5 times faster, compare to delete identical. I tested it on 200,000 points, with 100,000 of them being a ...
- 23.2k
6
votes
Accepted
LineString VS Multi-LineString Performance PostGIS
I wouldn't expect any significant difference in performance unless you have very unusual data. There are a few places in the code where optimizations are put in for edge cases like two-point ...
- 13.2k
6
votes
Displaying huge geotiffs (or vrts) with QGIS?
You seem to have two main concerns: VRT id slow with browsing and it is slow to build global overviews.
While I am sure that GDAL VRT used to be slow for me and my MapServer many years ago it may be ...
- 67.3k
6
votes
Accepted
Displaying huge geotiffs (or vrts) with QGIS?
Ok, well I solved both my problems...mainly by buying an NVMe SSD. My disk read/write has gone from 125 MB/s to 1200 MB/s.
Programatically, there are a few things you can do to help your read/write ...
- 2,894
Only top scored, non community-wiki answers of a minimum length are eligible