I have two Azure Storage accounts. On one of the Storage account, there are almost 100 containers and some blobs in those containers. I want to transfer all these containers along with the blobs in it to other Storage account.
I have seen many tools copying the blobs from one container to other, but didn't come across any that copies the whole containers aswell. Please help with this.
Thanks
It's a bit late but this PowerShell script will do it...
#Server side storage copy
$SourceStorageAccount = "sourceAccountName"
$SourceStorageKey = "sourceAccountAPIKey"
$DestStorageAccount = "destinationAccountName"
$DestStorageKey = "destinationAccountAPIKey"
$SourceStorageContext = New-AzureStorageContext -StorageAccountName $SourceStorageAccount -StorageAccountKey $SourceStorageKey
$DestStorageContext = New-AzureStorageContext -StorageAccountName $DestStorageAccount -StorageAccountKey $DestStorageKey
$Containers = Get-AzureStorageContainer -Context $SourceStorageContext
foreach($Container in $Containers)
{
$ContainerName = $Container.Name
if (!((Get-AzureStorageContainer -Context $DestStorageContext) | Where-Object { $_.Name -eq $ContainerName }))
{
Write-Output "Creating new container $ContainerName"
New-AzureStorageContainer -Name $ContainerName -Permission Off -Context $DestStorageContext -ErrorAction Stop
}
$Blobs = Get-AzureStorageBlob -Context $SourceStorageContext -Container $ContainerName
$BlobCpyAry = @() #Create array of objects
#Do the copy of everything
foreach ($Blob in $Blobs)
{
$BlobName = $Blob.Name
Write-Output "Copying $BlobName from $ContainerName"
$BlobCopy = Start-CopyAzureStorageBlob -Context $SourceStorageContext -SrcContainer $ContainerName -SrcBlob $BlobName -DestContext $DestStorageContext -DestContainer $ContainerName -DestBlob $BlobName
$BlobCpyAry += $BlobCopy
}
#Check Status
foreach ($BlobCopy in $BlobCpyAry)
{
#Could ignore all rest and just run $BlobCopy | Get-AzureStorageBlobCopyState but I prefer output with % copied
$CopyState = $BlobCopy | Get-AzureStorageBlobCopyState
$Message = $CopyState.Source.AbsolutePath + " " + $CopyState.Status + " {0:N2}%" -f (($CopyState.BytesCopied/$CopyState.TotalBytes)*100)
Write-Output $Message
}
}
Some credit for the script goes to this blog post...
http://windowsitpro.com/azure/copy-content-one-azure-storage-account-another
...I've ammended his code removing nasty long dashes and added code to loop through the containers instead of specifying them manually.
You can try the following script based on Azcopy. It will list all containers in a storage account, and copy them to another storage account one by one by AzCopy .
To copyeverything
Copy all containers, directories, and blobs to another storage account by using the azcopy copy command.
azcopy copy 'https://<source-storage-account-name>.blob.core.windows.net/<SAS-token>' 'https://<destination-storage-account-name>.blob.core.windows.net/' --recursive
azcopy copy 'https://mysourceaccount.blob.core.windows.net/?sv=2018-03-28&ss=bfqt&srt=sco&sp=rwdlacup&se=2019-07-04T05:30:08Z&st=2019-07-03T21:30:08Z&spr=https&sig=CAfhgnc9gdGktvB=ska7bAiqIddM845yiyFwdMH481QA8%3D' 'https://mydestinationaccount.blob.core.windows.net' --recursive
I used Mick's script in a scheduled Azure Automation Runbook job to backup blob storage.
I made this minor change to it to check if blob already exists in the destination container before attempting copying over:
#Do the copy of everything
foreach ($Blob in $Blobs)
{
$BlobName = $Blob.Name
$blob = Get-AzureStorageBlob -Blob $BlobName -Container $ContainerName -Context
$DestStorageContext -ErrorAction Ignore
if (-not $blob)
{
Write-Output "Copying $BlobName from $ContainerName"
$BlobCopy = Start-CopyAzureStorageBlob -Context $SourceStorageContext -SrcContainer $ContainerName -SrcBlob $BlobName -DestContext $DestStorageContext -DestContainer $ContainerName -DestBlob $BlobName
$BlobCpyAry += $BlobCopy
}
}
Similar to above but using Bash with Azure CLI and AZCopy - Code is on Github and associated video on YouTube.
https://github.com/J0hnniemac/yt-blobsync
#!/bin/bash
cd /home
app_id=""
tenant=""
sourceurl="https://<>.blob.core.windows.net"
destinationurl="https://<>.blob.core.windows.net"
pemfile="/home/service-principal.pem"
sourceaccount=$(echo $sourceurl | awk -F/ '{print $3}' | awk -F. '{print $1}')
destinationaccount=$(echo $destinationurl | awk -F/ '{print $3}' | awk -F. '{print $1}')
echo $app_id
echo $tenant
echo $sourceurl
echo $destinationurl
echo $sourceaccount
echo $destinationaccount
az login --service-principal --password $pemfile --username $app_id --tenant $tenant
# list storage containers
az storage container list --auth-mode login --account-name $sourceaccount -o=table | awk 'NR>1 {print $1}' | grep networking-guru > src.txt
az storage container list --auth-mode login --account-name $destinationaccount -o=table | awk 'NR>1 {print $1}' | grep networking-guru > dst.txt
grep -vf dst.txt src.txt > diff.txt
for blob_container in $(cat diff.txt);
do
echo $blob_container;
newcmd="az storage container create --auth-mode login --account-name $destinationaccount -n $blob_container --fail-on-exist"
echo "---------------------------------"
echo $newcmd
eval $newcmd
done
echo "performing AZCOPY login"
azcopy login --service-principal --certificate-path $pemfile --application-id $app_id --tenant-id $tenant
echo "performing AZCOPY sync for each container"
for blob_container in $(cat src.txt);
do
#Create timestame + 30 Minutes for SAS token
end=`date -u -d "30 minutes" '+%Y-%m-%dT%H:%MZ'`
sourcesas=`az storage container generate-sas --account-name $sourceaccount --as-user --auth-mode login --name $blob_container --expiry $end --permissions acdlrw`
echo $sourcesas
# remove leading and trailing quotes from SAS Token
sourcesas=$(eval echo $sourcesas)
echo $sourcesas
src="$sourceurl/$blob_container?$sourcesas"
dst="$destinationurl/$blob_container"
echo $src
echo $dst
synccmd="azcopy sync \"$src\" \"$dst\" --recursive --delete-destination=true"
echo $synccmd
eval $synccmd
done
I highly recommend you take a look at Azure Storage Explorer https://azure.microsoft.com/en-us/features/storage-explorer/. It's a lot easier than azcopy, and you can even copy and paste between containers.
This is from 2020 and the credit goes to @Mick. I just removed the separate blob copy. And it overwrites the whole container in one line. If the code executes this is guaranteed to work. So no check is given. NB: This will not copy folders and instead give error
#Server side storage copy
$SourceStorageAccount = "sourceAccountName"
$SourceStorageKey = "sourceAccountAPIKey"
$DestStorageAccount = "destinationAccountName"
$DestStorageKey = "destinationAccountAPIKey"
$SourceStorageContext = New-AzureStorageContext -StorageAccountName $SourceStorageAccount -StorageAccountKey $SourceStorageKey
$DestStorageContext = New-AzureStorageContext -StorageAccountName $DestStorageAccount -StorageAccountKey $DestStorageKey
$Containers = Get-AzureStorageContainer -Context $SourceStorageContext
foreach($Container in $Containers)
{
$ContainerName = $Container.Name
if (!((Get-AzureStorageContainer -Context $DestStorageContext) | Where-Object { $_.Name -eq $ContainerName }))
{
Write-Output "Creating new container $ContainerName"
New-AzureStorageContainer -Name $ContainerName -Permission Off -Context $DestStorageContext -ErrorAction Stop
}
Write-Output "start copy $ContainerName"
az storage blob copy start-batch --destination-container "$ContainerName" --account-name "$DestStorageAccount" --account-key "$DestStorageKey" --source-account-name "$SourceStorageAccount" --source-account-key "$SourceStorageKey" --source-container "$ContainerName"
}
Select-AzSubscription -Subscription "your subscription name" # Set subscription id
# Variables
$SourceStorageAccount = "SourceStorageAccount" # Replace with required Source Storage Account
$DestStorageAccount = "DestStorageAccount" # Replace with required Destination Storage Account
$SourceResourceGroupName = "SourceResourceGroupName" # Replace with required source resource group name
$DestResourceGroupName = "DestResourceGroupName" # Replace with required destination resource group name
$SourceStorageKey = Get-AzStorageAccountKey -Name $SourceStorageAccount -ResourceGroupName $SourceResourceGroupName
$DestStorageKey = Get-AzStorageAccountKey -Name $DestStorageAccount -ResourceGroupName $DestResourceGroupName
$SourceStorageContext = New-AzStorageContext -StorageAccountName $SourceStorageAccount -StorageAccountKey $SourceStorageKey.Value[0]
$DestStorageContext = New-AzStorageContext -StorageAccountName $DestStorageAccount -StorageAccountKey $DestStorageKey.Value[0]
$Containers = Get-AzStorageContainer -Context $SourceStorageContext
foreach($Container in $Containers)
{
$ContainerName = $Container.Name
if ($ContainerName -eq "snow-cmdb"){
$ContainerName = $Container.Name+"copy3"
if (!((Get-AzureStorageContainer -Context $DestStorageContext) | Where-Object { $_.Name -eq $ContainerName }))
{
Write-Output "Creating new container $ContainerName"
New-AzureStorageContainer -Name $ContainerName -Permission Off -Context $DestStorageContext -ErrorAction Stop
}
$Blobs = Get-AzureStorageBlob -Context $SourceStorageContext -Container $ContainerName
$BlobCpyAry = @() #Create array of objects
#Copy every thing
foreach ($Blob in $Blobs)
{
$BlobName = $Blob.Name
Write-Output "Copying $BlobName from $ContainerName"
$BlobCopy = Start-CopyAzureStorageBlob -Context $SourceStorageContext -SrcContainer $ContainerName -SrcBlob $BlobName -DestContext $DestStorageContext -DestContainer $ContainerName -DestBlob $BlobName
$BlobCpyAry += $BlobCopy
}
#Check Status
foreach ($BlobCopy in $BlobCpyAry)
{
#Could ignore all rest and just run $BlobCopy | Get-AzureStorageBlobCopyState output with % copied
$CopyState = $BlobCopy | Get-AzureStorageBlobCopyState
$Message = $CopyState.Source.AbsolutePath + " " + $CopyState.Status + " {0:N2}%" -f (($CopyState.BytesCopied/$CopyState.TotalBytes)*100)
Write-Output $Message
}
}
}
Did you see this tool?: https://azure.microsoft.com/en-us/documentation/articles/storage-use-azcopy/, it may copy all blobs from one container to another.
Though, when I needed to move content between accounts, I ended up with downloading all content to VM and then uploading it back to new account.
