How to recover data from adaptive storage while rooted phone stuck at boot?

Question

So , i'm using Galaxy Mega (i9152) with a CM13 ROM which i flashed through TWRP. For the sake of running PoGO , i had disabled root through the settings of the SuperSu app in CM13 (the game still didn't run , kept tripping safety net). The phone was running fine up till 2 days ago , after which it started going into bootloop.

so , i googled on how to resolve boot loop and saw that fixing file permissions through TWRP might resolve the issue. i did that BUT i also rooted the phone when TWRP prompted me to install SuperSU , citing that the phone was not rooted. After that , the phone was stuck at Boot Animation (the CM android face) , possibly because i rooted the phone through TWRP.

I had formatted my SD card as internal which meant that it would not work on any other device. All my pics and stuff is on that SD card now. I researched extensively on google to find out a way to decrypt the encrypted SD card and found out that i can decrypt it through linux. i have also recovered the .key file from /data/misc/vold folder and have converted the code into the appropriate 16 byte format through terminal commands.

I have a linux system and i have tried to decrypt it. here's what happens when i run parted -l. (i also wanted to post output of fdisk -l , but my reputation score won't let me post more than 2 links)

since the names "android_expand" and "android_meta" are still showing , i have hope and good reason to believe that my data's still intact !

but when i try to mount it ...

i don't know what to do now , since this is all i knew , all i had :/ , i have no idea what filesystem the encrypted card is. i tried ext4 as described in the guide. maybe if there was some way i could make the fstab file cycle through ALL possible filesystems , hitting the right one and finally mounting my SD card. maybe the mount feature has such an 'auto' feature to auto detect the filesystems ?

I have tried to reflash the SAME ROM without factory resetting , but it gave me an error saying : "error executing updater binary" and also "cannot install on top of incompatible data"

i have cleared the Cache and Dalvik Cache SEVERAL times.

i have backed up the DATA portion , through TWRP , by using ANOTHER SD card in the phone . Also i have backed up SYSTEM portion in the internal storage of the phone. I am just worried about my pics and stuff on SD card. I can't see my encrypted SD card in TWRP , it just shows me O MB in the SD card option in TWRP , possibly because the SD card is encrypted. i have no idea how to access my files in the encrypted SD card through the TWRP file manager. i don't want to lose my data. just please help me recover data from this SD card that has been encrypted by Android Marshmallow. i don't know if my SD card will be detected in the OS as before if i am able to somehow reflash the EXACT SAME rom i was using prior to this disaster.

Just to clarify , i am using my Laptop's internal SD card reader for reading card with Linux (i have ubuntu 16.04 LTS), with an sd card adapter supplied with the card (it's a Kingston 16 GB , SDHC class 4)

if i can somehow access my SD card and retrieve all my data from it , my problem will be solved. please help me out.

P.S : also , at the VERY beginning of this ordeal , by mistake , i MIGHT have created partitions on the SD card , while using the fdisk. i hope that did not wipe the data off the sd card or corrupt it. i CTRL+C'ed out of the utility without saving changes ... so that might be of some hope.

EDIT (1-Dec-2019): This is NOT a duplicate, as can be seen by the solution i provided below. This question represents a much more serious case with adopted storage, where your phone is having problems booting or your OS is corrupted. Kindly let this question remain as it is, as it will prove helpful to advanced users and others who have their files stuck in adopted storage while facing difficulties in accessing their phone's Operating System.

Answer 1

before starting , i'd recommend making a FULL backup of your SD card , through a raw image ( google the 'dd' command in linux)

i followed the guide here ; Corrupt SD card formatted as internal storage

BUT with some IMPORTANT MODIFICATIONS of my own !

1. Phone was stuck in BOOTLOOP and later stuck at BOOT Animation, no access to any usual file manager utility. Mind you , this is a VERY Grievous situation , since there's no trivial method of gaining access to your sd card (it shows as 0mb in TWRP file manager) for backing up your precious data , pics and memories. Also , this is the only part which is a bit shaky since i am not completely sure if my phone was rooted or not (since my problem changed from bootloop to boot animation when i tried to install to SuperSu through TWRP) and retrieving the encryption key is central to solving this. I had to get the .key file , since that was pivotal to breaking the SD card's encryption. So amidst all the panic , i was able to browse , through TWRP file manager , to the /data/misc/vold folder and found the .key file. This was a MAJOR breakthrough !

2. Next , to extract the file. i used another , UNENCRYPTED SD card and transferred the .key file to that card , through TWRP file manager , read that SD card into the laptop and got the .key file onto Ubuntu (16.04) . Decoded the .key file and got the 16 byte key. there's 2 ways to do this ... either open the file through a hex editor (i used BLESS) and copy off the 16 pairs of hex digits you see OR use this command ;

hexdump -e '1/1 "%.2x"' expand_8838e738a18746b6e435bb0d04c15ccd.key

make sure you're in the SAME directory as the .key file when executing this command (advanced linux users will work their way around this but i want it to be as beginner friendly as possible)

so , i got the 32 digit / 16 byte key.

3. NEXT , decrypting the SD card AND making Linux recognise it. when i read it in , through the card reader , it wouldn't show in Nautilus (the Ubuntu filemanager) but upon running fdisk-l and parted -l , i saw

"parted -l output"

http://pasteboard.co/5klviklR8.jpg

"fdisk -l output"

http://pasteboard.co/POGvyBOxO.png

notice the android_meta and android_expand , those names gave me hope , that my data was still intact. ESPECIALLY , since at the beginning of the problem , i had created partitions and partition tables in the sd card through the fdisk utility (but i ctrl+c'ed out so it's possible that the changed were not written , PHEW! )

the unknown filesystem was the biggest pain in this whole fiasco ... since according to the few guides and discussions regarding this issue , SD cards formatted as internal by marshmallow were supposed to be ext4 filesystem. but that was not the case , at least not here.

4. So , to decrypt the card , i used

dmsetup create crypt1 --table "0 `blockdev --getsize /dev/mmcblk0p2` crypt aes-cbc-essiv:sha256 "insert your 16-byte key here w/out double quotes" 0 /dev/mmcblk0p2 0"

if you get some ioctl related error use this format instead (this is the EXACT format i used today , with my key and everything)

dmsetup create crypt7 --table "0 `blockdev --getsize /dev/mmcblk0p2` crypt aes-cbc-essiv:sha256 07147CFFB77F249A5DBD2AD204610E7D  0 /dev/mmcblk0p2 0"

please note that the dmsetup utility should be installed in linux. (google it ) . also , note that i went for the parition 'mmcblk0p2' , since that was the bigger one in size and had the obvious inference of containing all the data !

this went well , completed without error. the next bit , though , ABSOLUTELY drove me crazy and took around 20 hours (including my sleep time ) to be properly resolved.

5. the GADDAMN file system ! it was written EVERYWHERE that adoptive storage is formatted as ext4 , so when i ran the command

# mount -t ext4 /dev/mapper/crypt1 /mnt/1/

(please note that mnt/1/ should exist as directory in your root folder , if not , use mkdir to create it)

it yielded the damn error

wrong fs or bad option.

and this got me thinking ... if the sd card is NOT ext4 type , then HOW ON EARTH am i supposed to know what file system it is out of all the possible ones !

i searched and searched for a command that could force cycling through ALL possibilities in a brute-force method or any entry i could make in the /etc/fstab file that could AUTOMATICALLY detect the filesystem somehow, but to no avail...

5 . In the end , i recalled ... that maybe , just MAYBE the filesystem was f2fs (flash-friendly file system). And that proved to be a telling recall ...

for this code ;

mount -t f2fs /dev/mapper/crypt1 /mnt/1

proved to be the WINNING GOAL !

i was then able to cd into /mnt/1 and saw some very familiar android-esque files and from there on , it was Eternal BLISS !