Questions tagged [sharding]
Sharding is a technique of partitioning database tables by row ("horizontally"); typically this technique requires a key to be selected that determines how the rows are to be partitioned.
sharding
1,673
questions
0
votes
0
answers
25
views
Efficient method for sharding BigQuery table collection
I would like to ask for advice regarding the following task: assume a collection of BQ tables bearing names with structure name_YYYYMM and containing each a DATETIME type column called date_time whose ...
- 111
0
votes
0
answers
44
views
Create read-only shards from a postgres DB
I have an application that consists of a master application+DB and a bunch of edge servers. Each edge server syncs a subset of the master data via custom API calls. I would like to simplify this ...
- 142
0
votes
1
answer
58
views
Index based routing allocation not working - Elasticsearch 5.x
As a dependent question to an existing open thread, where I was doubting the strategy or implementation details of settings when modified for a cluster as well as an index. While I had tried to ...
- 30.5k
0
votes
0
answers
27
views
Cluster versus Index level allocation in Elasticsearch
(Setup) In one of our production cluster, we have four entities namely user, call, interaction and comment indexed on an Elasticsearch(5.4.1). As the scale of these entities is increasing, we are ...
- 30.5k
0
votes
0
answers
38
views
springboot+myabtisplus+sharding-jdbc can't use my custom algorithm, error is logic table doesn't exist
version
springboot version is 2.3.5.RELEASE
mybatis-plus 3.5.1
sharding-jdbc-spring-boot-starter 4.1.1
<parent>
<groupId>org.springframework.boot</groupId>
<artifactId>...
- 1
0
votes
1
answer
37
views
Cron job scheduled in spring app to distribute data with all instances
I have a design question
Cron job scheduled in spring app using @scheduled, I have 4 instances and I want the job to run in all the instances by distributing data. Say I need to process 1000 data, ...
0
votes
0
answers
38
views
Dynamic data source mapping in spring boot for data sharding
I am creating a spring boot application which will store data in Postgres. I have 3 Postgres instance which I am calling shard-0 shard-1 and shard-2, based on the shard key, I want to calculate and ...
- 451
1
vote
1
answer
136
views
Elastic maximum shards open best practics
does anyone knows how can i solve
Error: Validation Failed: 1: this action would add [2] shards, but this cluster currently has [3000]/[3000] maximum normal shards open
i know elasticsearch suggest ...
- 425
0
votes
1
answer
29
views
How failures and restore operations in sharding (consistent hashing)
In consistent hashing, suppose we are using username as for hashing
hashFunction(username) = nodeA
Now from what I understand, if there is any failure or a node is removed requests will be directed to ...
- 47
0
votes
0
answers
35
views
Unable to connect to mongod instance started with --configsvr option
I am trying to create a mongo sharded cluster. First of all, I want to create a config server replica set with docker compose.
My docker-compose.yml file
version: "3.8"
services:
mongo1:
...
- 1
0
votes
0
answers
34
views
Problems encountered when using _shard_num when querying clickhouse shard sets
I'm using a sharded set where my table is described like this
This is my _all_all table:
ENGINE = Distributed('level2_cluster', 'datas', 'data_animals_0_60_all', rand())
This is my _all table:
ENGINE ...
-3
votes
2
answers
86
views
Horizontal scaling strategy with 10,000 shards [closed]
My app has a User collection. Each document in the collection averages about .04 MB. At worst case, a document may slightly exceed .1 MB. Needless to say, these are small documents. However, each ...
0
votes
1
answer
29
views
Assigning a dedicated Primary node for write operations in MongoDB replica set
Distribution of my MongoDB cluster consisting of 15 nodes across 3 different data centers is as listed below:
DataCenter 1:
Router-1
ConfigServer-1
Shard1Node1 [Primary]
Shard2Node2 [Secondary]
...
- 11
0
votes
1
answer
379
views
Why does Elastic search limit the maximum shard number to 1k per node?
"The cluster shard limits prevent creation of more than 1000 non-frozen shards per node, and 3000 frozen shards per dedicated frozen node. Make sure you have enough nodes of each type in your ...
- 167
0
votes
1
answer
117
views
citus add node --> "fe_sendauth: no password supplied" error
I am trying to setup multi node schema based sharding for postgresql database using citus extension.
I have two azure virtual machines , one is working as worker node('20.40.43.246') and other as ...
- 329