Timeline for Is SQL a good way to optimise data access to just one huge table?
Current License: CC BY-SA 4.0
7 events
| when toggle format | what | by | license | comment | |
|---|---|---|---|---|---|
| Sep 27, 2023 at 17:14 | comment | added | music2myear | Are there better ways to store your data? Definitely! Is SQL likely to be one of those? Most likely. But this is really broad. I recommend having someone more familiar with databases or data management, and hopefully as into astronomic bodies as you, look at the data and advise how it may better be stored and served. Storing the data is only one part: you also need to retrieve and present the data (the database is the "back-end", and there's a "front-end" user interface). There may already be solutions that exist, or this will require an entire system designed and built. | |
| Sep 23, 2023 at 11:42 | comment | added | Hannu |
Note, you can access an SQL database from python, select ... where ... sort by ... is not overly complex, and you will get a subset of the data that corresponds to your where ... - this is simpler than you might expect. e.g. pip install pyodbc migh tbe a starting point.
|
|
| Sep 23, 2023 at 10:13 | review | Close votes | |||
| Oct 14, 2023 at 3:04 | |||||
| Sep 23, 2023 at 10:03 | comment | added | davidgo | A couple of thoughts - - SQL (eg Postgres) is likely to be A LOT faster then Python. One factor to consider is how you will index the data. Indexes can massively impact speed of access and it does not seem you have considered that. On the flipside it likely takes more space on disk and more memory to parse the a giant CSV. Thete os nothing erong with a single big table. | |
| Sep 23, 2023 at 9:38 | comment | added | CommunityBot | Please edit the question to limit it to a specific problem with enough detail to identify an adequate answer. | |
| S Sep 23, 2023 at 9:27 | review | First questions | |||
| Sep 23, 2023 at 9:38 | |||||
| S Sep 23, 2023 at 9:27 | history | asked | condosz | CC BY-SA 4.0 |