how to rename an index in a cluster?

Question

I need to rename several indexes in a cluster (their name must be changed, I cannot use aliases).

I saw that there are no supported ways to do that, the closest I found is to rename the directory of the index, I tried this in a cluster.

The cluster has 3 machines A, B and C and the shards are replicated on each of them. I shut down elasticsearch on A, renamed /var/lib/elasticsearch/security/nodes/0/indices/oldindexname to /var/lib/elasticsearch/security/nodes/0/indices/newindexname and restarted A.

The state of the cluster was yellow and elasticsearch was doing some magic to restore a correct state. After some time I ended up with

• oldindexname being available and fully replicated (recovered from B and C I guess)
• newindexname being available (I can search it) but the head plugin shows that its shards are in an "Unassigned" state and that they are grayed out (not replicated)

During the recovery security.log showed the following message:

[2015-02-20 11:02:33,461][INFO ][gateway.local.state.meta ] [A.example.com] dangled index directory name is [newindexname], state name is [oldindexname], renaming to directory name

While newindexname is searchable, it is certainly not in a normal state.

I rolled back to the previous state by deleting newindexname. The cluster is back to green without any "Unassigned" entries.

Given that, how can I rename oldindexname to newindexname in a cluster?

Note: the ultimate solution I have in mind is to scroll-copy oldindex into newindex and delete oldindex afterwards. This is going to take time so if there is a more direct solution it would be great.

Answer 1

You can use REINDEX to do that.

Reindex does not attempt to set up the destination index. It does not copy the settings of the source index. You should set up the destination index prior to running a _reindex action, including setting up mappings, shard counts, replicas, etc.

1. First copy the index to a new name

POST /_reindex
{
  "source": {
    "index": "twitter"
  },
  "dest": {
    "index": "new_twitter"
  }
}

2. Now delete the Index

DELETE /twitter

Answer 2

Starting with ElasticSearch 7.4, the best method to rename an index is to copy the index using the newly introduced Clone Index API, then to delete the original index using the Delete Index API.

The main advantage of the Clone Index API over the use of the Snapshot API or the Reindex API for the same purpose is speed, since the Clone Index API hardlinks segments from the source index to the target index, without reprocessing any of its content (on filesystems that support hardlinks, obviously; otherwise, files are copied at the file system level, which is still much more efficient that the alternatives). Clone Index also guarantee that the target index is identical in every point to the source index (that is, there is no need to manually copy settings and mappings, contrary to the Reindex approach), and doesn't require a local snapshot directory be configured.

Side note: even though this procedure is much faster than previous solutions, it still implies down time. There are real use cases that justify renaming indices (for example, as a step in a split, shrink or backup workflow), but renaming indices should not be part of day-to-day operations. If your workflow requires frequent index renaming, then you should consider using Indices Aliases instead.

Here is an example of a complete sequence of operations to rename index source_index to target_index. It can be executed using some ElasticSearch specific console, such as the one integrated in Kibana. See this gist for an alternative version of this example, using curl instead of an Elastic Search console.

# Make sure the source index is actually open
POST /source_index/_open

# Put the source index in read-only mode
PUT /source_index/_settings
{
  "settings": {
    "index.blocks.write": "true"
  }
}

# Clone the source index to the target name, and set the target to read-write mode
POST /source_index/_clone/target_index
{
  "settings": {
    "index.blocks.write": null 
  }
}

# Wait until the target index is green;
# it should usually be fast (assuming your filesystem supports hard links).
GET /_cluster/health/target_index?wait_for_status=green&timeout=30s

# If it appears to be taking too much time for the cluster to get back to green,
# the following requests might help you identify eventual outstanding issues (if any)
GET /_cat/indices/target_index
GET /_cat/recovery/target_index
GET /_cluster/allocation/explain

# Delete the source index
DELETE /source_index

Answer 3

For renaming your index you can use Elasticsearch Snapshot module.

First you have to take snapshot of your index.while restoring it you can rename your index.

    POST /_snapshot/my_backup/snapshot_1/_restore
    {
     "indices": "jal",
     "ignore_unavailable": "true",
     "include_global_state": false,
     "rename_pattern": "jal",
     "rename_replacement": "jal1"
     }

rename_replacement :-New indexname in which you want backup your data.

Answer 4

If you can't REINDEX a workaround is to use aliases. From the official documentation:

APIs in elasticsearch accept an index name when working against a specific index, and several indices when applicable. The index aliases API allow to alias an index with a name, with all APIs automatically converting the alias name to the actual index name. An alias can also be mapped to more than one index, and when specifying it, the alias will automatically expand to the aliases indices. An alias can also be associated with a filter that will automatically be applied when searching, and routing values. An alias cannot have the same name as an index.

Be aware that this solution does not work if you're using More Like This feature. https://github.com/elastic/elasticsearch/issues/16560

Answer 5

As such there is no direct method to copy or rename index in ES (I did search extensively for my own project)

However a very easy option is to use a popular migration tool [Elastic-Exporter].

http://www.retailmenot.com/corp/eng/posts/2014/12/02/elasticsearch-cluster-migration/

[PS: this is not my blog, just stumbled upon and found it good]

Thereby you can copy index/type and then delete the old one.

Answer 6

Another different way to achieve the renaming or change the mappings for an index is to reindex using logstash. Here is a sample of the logstash 2.1 configuration:

input {
  elasticsearch {
   hosts => ["es01.example.com", "es02.example.com"]
   index => "old-index-name"
   size => 500
   scroll => "5m"
  }
}
filter {

 mutate {
  remove_field => [ "@version" ]
 }

 date {
   "match" => [ "custom_timestamp", "MM/dd/YYYY HH:mm:ss" ]
   target => "@timestamp"
 }

}
output {
 elasticsearch {
   hosts => ["es01.example.com", "es02.example.com" ]
   manage_template => false
   index => "new-index-name"
 }
}

Answer 7

As indicated in Elasticsearch reference for snapshot module,

The rename_pattern and rename_replacement options can be also used to rename index on restore using regular expression

Answer 8

For those landing here based on OpenSearch, here's how you do it...

PUT /<source-index>/_block/write

DELETE <target-index> (if one already exists)

PUT /<source-index>/_clone/<target-index>

Answer 9

Just in case someone still needs it. The successful, not official, way to rename indexes are:

1. Close indexes that need to be renamed
2. Rename indexes' folders in all data directories of master and data nodes.
3. Reopen old closed indexes (I use kofp plugin). Old indexes will be reopened but stay unassigned. New indexes will appear in closed state
4. Reopen new indexes
5. Delete old indexes

If you happen to get this error "dangled index directory name is", remove index folder in all master nodes (not data nodes), and restart one of the data nodes.