1

I have problem with btrfs incremental send and receive from local to remote machine.

My host Lxd is Ubuntu 16.04.3 LTS with lxd 2.0.10 and btrfs-progs v4.4

My 2 containers are centos7 (CentOS Linux release 7.3.1611 (Core) with

btrfs-progs-devel-4.4.1-1.el7.x86_64

btrfs-progs-4.4.1-1.el7.x86_64

mariadb-libs-5.5.52-1.el7.x86_64

mariadb-5.5.52-1.el7.x86_64

mariadb-server-5.5.52-1.el7.x86_64

The First mariadb centos7 container. (local btrfs)

I make btrfs sub volume /var/lib/mariadb/mysql for keeping mariadb database and make snapshot very day

The result btrfs snapshot example on The First mariadb centos7 container

ID 281 gen 195 top level 5 path mysql_201707210830

ID 288 gen 186 top level 5 path mysql_201707220830

ID 290 gen 191 top level 5 path mysql_201707230830

ID 292 gen 217 top level 5 path mysql

The Second mariadb centos7 container. (remote btrfs)

I make btrfs sub volume /var/lib/mariadb

and send btrfs sub volume snapshot from The First mariadb centos7 container start with mysql_201707210830 and increamental between mysql_201707210830 and mysql_201707220830 and increamental between mysql_201707220830 and mysql_201707230830

The result btrfs snapshot example on The Second mariadb centos7 container

ID 270 gen 68 top level 5 path mysql_201707210830

ID 274 gen 66 top level 5 path mysql_201707220830

ID 276 gen 71 top level 5 path mysql_201707230830

I start to test the result on Second mariadb centos7 container with following procedure (first of all "cd /var/lib/mariadb ").

  1. use command "btrfs sub snap mysql_201707210830 mysql" then "systemctl start mariadb" the result is fine mariadb works as expected. ( after this "systemctl stop mariadb" ,"btrfs sub del mysql" and "btrfs sub sync .")

    1. use command "btrfs sub snap mysql_201707220830 mysql" then "systemctl start mariadb" the result is fine mariadb works as expected. ( after this "systemctl stop mariadb" ,"btrfs sub del mysql" and "btrfs sub sync .")

    2. use command "btrfs sub snap mysql_201707230830 mysql" then "systemctl start mariadb" the result is not as expected!!!! mariadb cannot start.

Anyone please help me what step that I make the mistake?

Best Regards,

Siranee Jaraswachirakul.

Improve this question

1 Answer 1

Reset to default
0

With all help from the btrfs support team. Thanks you very much for the suggestions from "Chris Murphy" and "A L".

Finally I found the mistake steps which made the result incorrect. The tool to prove identical between source and destination snapshot send/receive is "rsync -avnc /var/lib/mariadb/mysql_yyyymmddhhmm/ user@ip_destination:/var/lib/mariadb/mysql_yyyymmddhhmm/"

On Sat, Aug 12, 2017 at 8:20 PM, wrote:

[root@backuplogC7 ~]# rsync -avnc /var/lib/mariadb/mysql_201708090830 [email protected]://var/lib/mariadb/mysql_201708090830

You need trailing / for the first directory with -a option.

rsync -a dir dir

is not the same command as

rsync -a dir/ dir

It's confusing but your command is trying to create mysql_201708090830 directory on the source, in the mysql_201708090830 on the destination. That is why everything mismatches. To make it mean "contents of" you need trailing slash on at least the origin.

-- Chris Murphy

I didn't remember which steps that I made a mistake and made the mysql had Received UUID.

The main point that made the btrfs send/receive incremental incorrect is the current sub volume "mysql" had "Received UUID" which should occur when receive at destination site (Remote) but in my case it appeared at source site (Local).

On 8/13/2017 12:52 PM, [email protected] wrote:

Hi "A L",

[root@backuplogC7 ~]# btrfs sub show /var/lib/mariadb/mysql /var/lib/mariadb/mysql Name: mysql UUID: 92f319c5-e132-3249-9b13-d39ee77a2b44 Parent UUID: - Received UUID: 3ad0334a-4063-654c-add6-b1cbcdeaa639 Creation time: 2017-06-21 13:27:41 +0700 Subvolume ID: 257 Generation: 539 Gen at creation: 9 Parent ID: 5 Top level ID: 5 Flags: - Snapshot(s): mysql_201708060830 mysql_201708070830 mysql_201708080830 mysql_201708090830 mysql_201708100830 mysql_201708110830 mysql_201708120830 mysql_201708130830

yes I think it has Received UUID because I restored the source from snapshot mysql_201708040830 for prove that the local snapshot was work.

How to clear the Received UUID ? What to do next? You need to make a read-write snapshot of /var/lib/mariadb/mysql and then remove the old subvolume and all its snapshots.

Example from https://github.com/digint/btrbk/blob/master/doc/FAQ.md

cd /mnt/btr_pool

mv mysubvolume mysubvolume.broken

btrfs subvolume snapshot mysubvolume.broken mysubvolume

You can do the same with each of your snapshots too, and send them as full snapshots (without -p).

~A

-- as recommend from https://github.com/digint/btrbk/blob/master/doc/FAQ.md --

"I'm getting an error: Aborted: "Received UUID" is set

You probably restored a backup with send-receive, and made it read/write using btrfs property set. This is bad, as all snapshots and backups will inherit this identical "Received UUID", which results in all these subvolumes will be treated as "containing same data".

To fix this, create a "proper" snapshot:

  • This is as your suggestion for the subvolume "mysql"

cd /mnt/btr_pool

mv mysubvolume mysubvolume.broken

btrfs subvolume snapshot mysubvolume.broken mysubvolume

Now, mysubvolume should have an empty "Received UUID". Note that in order to have a clean environment, you also need to fix all subvolumes (snapshots as well as backups) that you created with the broken subvolume.

Check if there are more broken subvolumes:

btrfs subvolume show mysubvolume.broken

btrfs subvolume list -a -R /mnt/btr_pool | grep <"Received UUID" from above>

btrfs subvolume list -a -R /mnt/btr_backup | grep <"Received UUID" from above>

  • This guide seem that I have to clear <"Received UUID" > only the subvolume "mysql" and the others ("mysql_201708070830" should using btrfs subvolume snapshot -r instead of btrfs subvolume snapshot. Is this correct?

Now clean all subvolume listed (same as above, but using btrfs subvolume snapshot -r now). Then delete all the broken subvolumes:

btrfs subvolume delete *.broken

Finally, you should have a clean environment, and btrbk will not complain any more.

Last procedure to repair the broken snapshot subvolumes are following.

I have done the following and it work as it should be right now.

[root@backuplogC7 mariadb]# btrfs subvolume snapshot mbroken_201708070830 rw_201708070830 Create a snapshot of 'mbroken_201708070830' in './rw_201708070830' [root@backuplogC7 mariadb]# btrfs sub list . ID 257 gen 542 top level 5 path mbroken ID 317 gen 576 top level 5 path mbroken_201708070830 ID 318 gen 568 top level 5 path mbroken_201708080830 ID 319 gen 569 top level 5 path mbroken_201708090830 ID 320 gen 570 top level 5 path mbroken_201708100830 ID 321 gen 571 top level 5 path mbroken_201708110830 ID 322 gen 572 top level 5 path mbroken_201708120830 ID 323 gen 573 top level 5 path mbroken_201708130830 ID 324 gen 543 top level 5 path mysql ID 348 gen 576 top level 5 path rw_201708070830 [root@backuplogC7 mariadb]# btrfs subvolume snapshot mbroken_201708080830 rw_201708080830 Create a snapshot of 'mbroken_201708080830' in './rw_201708080830' [root@backuplogC7 mariadb]# btrfs subvolume snapshot mbroken_201708090830 rw_201708090830 Create a snapshot of 'mbroken_201708090830' in './rw_201708090830' [root@backuplogC7 mariadb]# btrfs subvolume snapshot mbroken_201708100830 rw_201708100830 Create a snapshot of 'mbroken_201708100830' in './rw_201708100830' [root@backuplogC7 mariadb]# btrfs subvolume snapshot mbroken_201708110830 rw_201708110830 Create a snapshot of 'mbroken_201708110830' in './rw_201708110830' [root@backuplogC7 mariadb]# btrfs subvolume snapshot mbroken_201708120830 rw_201708120830 Create a snapshot of 'mbroken_201708120830' in './rw_201708120830' [root@backuplogC7 mariadb]# btrfs subvolume snapshot mbroken_201708130830 rw_201708130830 Create a snapshot of 'mbroken_201708130830' in './rw_201708130830' [root@backuplogC7 mariadb]# btrfs sub list . ID 257 gen 542 top level 5 path mbroken ID 317 gen 576 top level 5 path mbroken_201708070830 ID 318 gen 577 top level 5 path mbroken_201708080830 ID 319 gen 578 top level 5 path mbroken_201708090830 ID 320 gen 579 top level 5 path mbroken_201708100830 ID 321 gen 580 top level 5 path mbroken_201708110830 ID 322 gen 581 top level 5 path mbroken_201708120830 ID 323 gen 582 top level 5 path mbroken_201708130830 ID 324 gen 543 top level 5 path mysql ID 348 gen 576 top level 5 path rw_201708070830 ID 349 gen 577 top level 5 path rw_201708080830 ID 350 gen 578 top level 5 path rw_201708090830 ID 351 gen 579 top level 5 path rw_201708100830 ID 352 gen 580 top level 5 path rw_201708110830 ID 353 gen 581 top level 5 path rw_201708120830 ID 354 gen 582 top level 5 path rw_201708130830 [root@backuplogC7 mariadb]# btrfs subvolume list -a -R . | grep "3ad0334a-4063-654c-add6-b1cbcdeaa639" ID 257 gen 542 top level 5 received_uuid 3ad0334a-4063-654c-add6-b1cbcdeaa639 path mbroken ID 317 gen 576 top level 5 received_uuid 3ad0334a-4063-654c-add6-b1cbcdeaa639 path mbroken_201708070830 ID 318 gen 577 top level 5 received_uuid 3ad0334a-4063-654c-add6-b1cbcdeaa639 path mbroken_201708080830 ID 319 gen 578 top level 5 received_uuid 3ad0334a-4063-654c-add6-b1cbcdeaa639 path mbroken_201708090830 ID 320 gen 579 top level 5 received_uuid 3ad0334a-4063-654c-add6-b1cbcdeaa639 path mbroken_201708100830 ID 321 gen 580 top level 5 received_uuid 3ad0334a-4063-654c-add6-b1cbcdeaa639 path mbroken_201708110830 ID 322 gen 581 top level 5 received_uuid 3ad0334a-4063-654c-add6-b1cbcdeaa639 path mbroken_201708120830 ID 323 gen 582 top level 5 received_uuid 3ad0334a-4063-654c-add6-b1cbcdeaa639 path mbroken_201708130830 [root@backuplogC7 mariadb]# btrfs sub snap -r rw_201708070830 mysql_201708070830 Create a readonly snapshot of 'rw_201708070830' in './mysql_201708070830' [root@backuplogC7 mariadb]# btrfs sub snap -r rw_201708080830 mysql_201708080830 Create a readonly snapshot of 'rw_201708080830' in './mysql_201708080830' [root@backuplogC7 mariadb]# btrfs sub snap -r rw_201708090830 mysql_201708090830 Create a readonly snapshot of 'rw_201708090830' in './mysql_201708090830' [root@backuplogC7 mariadb]# btrfs sub snap -r rw_201708100830 mysql_201708100830 Create a readonly snapshot of 'rw_201708100830' in './mysql_201708100830' [root@backuplogC7 mariadb]# btrfs sub snap -r rw_201708110830 mysql_201708110830 Create a readonly snapshot of 'rw_201708110830' in './mysql_201708110830' [root@backuplogC7 mariadb]# btrfs sub snap -r rw_201708120830 mysql_201708120830 Create a readonly snapshot of 'rw_201708120830' in './mysql_201708120830' [root@backuplogC7 mariadb]# btrfs sub snap -r rw_201708130830 mysql_201708130830 Create a readonly snapshot of 'rw_201708130830' in './mysql_201708130830' [root@backuplogC7 mariadb]# btrfs sub list . ID 257 gen 542 top level 5 path mbroken ID 317 gen 576 top level 5 path mbroken_201708070830 ID 318 gen 577 top level 5 path mbroken_201708080830 ID 319 gen 578 top level 5 path mbroken_201708090830 ID 320 gen 579 top level 5 path mbroken_201708100830 ID 321 gen 580 top level 5 path mbroken_201708110830 ID 322 gen 581 top level 5 path mbroken_201708120830 ID 323 gen 582 top level 5 path mbroken_201708130830 ID 324 gen 584 top level 5 path mysql ID 348 gen 583 top level 5 path rw_201708070830 ID 349 gen 584 top level 5 path rw_201708080830 ID 350 gen 585 top level 5 path rw_201708090830 ID 351 gen 586 top level 5 path rw_201708100830 ID 352 gen 587 top level 5 path rw_201708110830 ID 353 gen 588 top level 5 path rw_201708120830 ID 354 gen 589 top level 5 path rw_201708130830 ID 355 gen 583 top level 5 path mysql_201708070830 ID 356 gen 584 top level 5 path mysql_201708080830 ID 357 gen 585 top level 5 path mysql_201708090830 ID 358 gen 586 top level 5 path mysql_201708100830 ID 359 gen 587 top level 5 path mysql_201708110830 ID 360 gen 588 top level 5 path mysql_201708120830 ID 361 gen 589 top level 5 path mysql_201708130830

[root@backuplogC7 mariadb]# btrfs subvolume list -a -R . | grep "3ad0334a-4063-654c-add6-b1cbcdeaa639" ID 257 gen 542 top level 5 received_uuid 3ad0334a-4063-654c-add6-b1cbcdeaa639 path mbroken ID 317 gen 576 top level 5 received_uuid 3ad0334a-4063-654c-add6-b1cbcdeaa639 path mbroken_201708070830 ID 318 gen 577 top level 5 received_uuid 3ad0334a-4063-654c-add6-b1cbcdeaa639 path mbroken_201708080830 ID 319 gen 578 top level 5 received_uuid 3ad0334a-4063-654c-add6-b1cbcdeaa639 path mbroken_201708090830 ID 320 gen 579 top level 5 received_uuid 3ad0334a-4063-654c-add6-b1cbcdeaa639 path mbroken_201708100830 ID 321 gen 580 top level 5 received_uuid 3ad0334a-4063-654c-add6-b1cbcdeaa639 path mbroken_201708110830 ID 322 gen 581 top level 5 received_uuid 3ad0334a-4063-654c-add6-b1cbcdeaa639 path mbroken_201708120830 ID 323 gen 582 top level 5 received_uuid 3ad0334a-4063-654c-add6-b1cbcdeaa639 path mbroken_201708130830 [root@backuplogC7 mariadb]# btrfs send /var/lib/mariadb/mysql_201708070830 | ssh 192.168.45.166 btrfs receive /var/lib/mariadb At subvol /var/lib/mariadb/mysql_201708070830 At subvol mysql_201708070830 [root@backuplogC7 mariadb]# btrfs sub show mysql_201708070830 /var/lib/mariadb/mysql_201708070830 Name: mysql_201708070830 UUID: 70ee3c31-126d-574a-814c-e3b4c81b414e Parent UUID: 1d5bb8eb-b0df-2549-8b62-552cfa517609 Received UUID: - Creation time: 2017-08-14 07:00:08 +0700 Subvolume ID: 355 Generation: 583 Gen at creation: 583 Parent ID: 5 Top level ID: 5 Flags: readonly Snapshot(s): [root@backuplogC7 mariadb]# rsync -avnc /var/lib/mariadb/mysql_201708070830/ [email protected]:/var/lib/mariadb/mysql_201708070830/ sending incremental file list ./

sent 3773 bytes received 19 bytes 1083.43 bytes/sec total size is 718361496 speedup is 189441.32 (DRY RUN) [root@backuplogC7 mariadb]# btrfs send -p /var/lib/mariadb/mysql_201708070830 /var/lib/mariadb/mysql_201708080830 | ssh 192.168.45.166 btrfs receive /var/lib/mariadb At subvol /var/lib/mariadb/mysql_201708080830 At snapshot mysql_201708080830 [root@backuplogC7 mariadb]# rsync -avnc /var/lib/mariadb/mysql_201708080830/ [email protected]:/var/lib/mariadb/mysql_201708080830/ sending incremental file list ./

sent 3769 bytes received 19 bytes 688.73 bytes/sec total size is 718361496 speedup is 189641.37 (DRY RUN) [root@backuplogC7 mariadb]# btrfs send -p /var/lib/mariadb/mysql_201708080830 /var/lib/mariadb/mysql_201708090830 | ssh 192.168.45.166 btrfs receive /var/lib/mariadb At subvol /var/lib/mariadb/mysql_201708090830 At snapshot mysql_201708090830 [root@backuplogC7 mariadb]# rsync -avnc /var/lib/mariadb/mysql_201708090830/ [email protected]:/var/lib/mariadb/mysql_201708090830/ sending incremental file list ./

sent 3773 bytes received 19 bytes 583.38 bytes/sec total size is 718361496 speedup is 189441.32 (DRY RUN) [root@backuplogC7 mariadb]# btrfs send -p /var/lib/mariadb/mysql_201708090830 /var/lib/mariadb/mysql_201708100830 | ssh 192.168.45.166 btrfs receive /var/lib/mariadb At subvol /var/lib/mariadb/mysql_201708100830 At snapshot mysql_201708100830 [root@backuplogC7 mariadb]# rsync -avnc /var/lib/mariadb/mysql_201708100830/ [email protected]:/var/lib/mariadb/mysql_201708100830/ sending incremental file list ./

sent 3773 bytes received 19 bytes 689.45 bytes/sec total size is 718361496 speedup is 189441.32 (DRY RUN) [root@backuplogC7 mariadb]# btrfs send -p /var/lib/mariadb/mysql_201708100830 /var/lib/mariadb/mysql_201708110830 | ssh 192.168.45.166 btrfs receive /var/lib/mariadb At subvol /var/lib/mariadb/mysql_201708110830 At snapshot mysql_201708110830 [root@backuplogC7 mariadb]# rsync -avnc /var/lib/mariadb/mysql_201708110830/ [email protected]:/var/lib/mariadb/mysql_201708110830/ sending incremental file list ./

sent 3773 bytes received 19 bytes 689.45 bytes/sec total size is 718361496 speedup is 189441.32 (DRY RUN) [root@backuplogC7 mariadb]# btrfs send -p /var/lib/mariadb/mysql_201708110830 /var/lib/mariadb/mysql_201708120830 | ssh 192.168.45.166 btrfs receive /var/lib/mariadb At subvol /var/lib/mariadb/mysql_201708120830 At snapshot mysql_201708120830 [root@backuplogC7 mariadb]# rsync -avnc /var/lib/mariadb/mysql_201708120830/ [email protected]:/var/lib/mariadb/mysql_201708120830/ sending incremental file list ./

sent 3773 bytes received 19 bytes 689.45 bytes/sec total size is 718361496 speedup is 189441.32 (DRY RUN) [root@backuplogC7 mariadb]# btrfs send -p /var/lib/mariadb/mysql_201708120830 /var/lib/mariadb/mysql_201708130830 | ssh 192.168.45.166 btrfs receive /var/lib/mariadb At subvol /var/lib/mariadb/mysql_201708130830 At snapshot mysql_201708130830 [root@backuplogC7 mariadb]# rsync -avnc /var/lib/mariadb/mysql_201708130830/ [email protected]:/var/lib/mariadb/mysql_201708130830/ sending incremental file list ./

sent 3773 bytes received 19 bytes 689.45 bytes/sec total size is 718361496 speedup is 189441.32 (DRY RUN) [root@backuplogC7 mariadb]#

Improve this answer

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .