Upload on-premises content to SharePoint Online

This is a step by step guide on how to use the SharePoint Online Migration PowerShell cmdlets to migrate content from an on-premises file share to Office 365.

SharePoint Online Migration PowerShell cmdlets are designed to move on-premises content from file shares. Requiring minimal CSOM calls, it leverages Azure temporary BLOB storage to scale to the demand of large migration of data content.

Here are the steps for using SPO Migration powershell to upload your on-premises data into SharePoint Online:

Step 1: Install the SharePoint Online Management Shell.

Step 2: Setup your working directory

Step 3: Determine your locations and credentials

Step 4: Create a new content package from an on-premises file share

Step 5: Convert the content package for your target site

Step 6: Submit content to import

Step 7: Processing and Monitoring your SPO Migration (optional)

Prerequisites

Supported Operating Systems: Windows 7 Service Pack 1, Windows 8, Windows Server 2008 R2 SP1, Windows Server 2008 Service Pack 2, Windows Server 2012, Windows Server 2012 R2
Windows PowerShell 4.0

Note: Permissions: You must be a site collection administrator on the site you are targeting.

Before you begin

Provision your Office 365 with either your existing active directory or one of the other options for adding accounts to Office 365. See See Office 365 integration with on-premises environments and Add users to Office 365 for business for more information.
Install the SharePoint Online Management Shell and set up your working directory.

Step 1: Install the SharePoint Online Management Shell

The first step is install the SharePoint Online Management shell.

Uninstall all previous versions of the SharePoint Online Management Shell.
Install from here: SharePoint Online Management Shell.
Open SharePoint Online Management Shell and select Run as Administrator.

Return to top

Step 2: Setup your working directory

Before you start the migration process, you need to setup your working directory by creating two empty folders. These folders to not require a lot of disk space as they will only contain XML.

Create a Temporary package folder.
Create a Final package folder.

Return to top

Step 3: Determine your locations and credentials

In this step you need to identify your locations and credentials, including the location of your source files, target files and web location.

On your local computer, open the SharePoint Online Management Shell. Run the following commands substituting your values.

$cred = (Get-Credential admin@contoso.com)

$sourceFiles = '\\fileshare\users\charles'

$sourcePackage = 'C:\migration\CharlesDocumentsPackage_source'

$targetPackage = 'C:\migration\CharlesDocumentsPackage_target'

$targetWeb = 'https://contoso-my.sharepoint.com/personal/charles_contoso_com'

$targetDocLib = 'Documents'

Return to top

Step 4: Create a new content package from an on-premises file share

In this step, you will create a new migration package from a file share. To create a content package from a file share, the New-SPOMigrationPackage command reads the list of content targeted by the source path and will generate XML to perform migration.

The following parameters are required unless marked optional:

SourcefilesPath: points to the content you want to migrate
OutputPackagePath: points to your Temporary folder
TargetWebUrl: point to your destination web
TargetDocumentLibraryPath: point to the document library inside the web.
IgnoreHidden: option to skip hidden files (optional)
ReplaceInvalidCharacters: will fix invalids characters when possible (optional)

Example:

This example shows how to create a new package from a file share, ignoring hidden files and replacing unsupported characters in a file/folder name.

New-SPOMigrationPackage -SourceFilesPath $sourceFiles -OutputPackagePath $sourcePackage -TargetWebUrl $targetWeb -TargetDocumentLibraryPath $targetDocLib -IgnoreHidden –ReplaceInvalidCharacters

Return to top

Step 5: Convert the content package for your target site

After you have created the content package, use the ConvertTo-SPOMigrationTargetedPackage command to convert the xml generated in your temporary folder. It saves a new set of targeted migration package metadata files to the target directory. This is the final package.

Note: Your target site collection administrator credentials are used to gather data to connect to the data site collection.

There are six required parameters to enter (others are optional

ParallelImport : Tell the tool to optimise performance by using parallel threads.
SourceFiles: points to the directory location where the package's source content files exist
SourcePackagePath: points to your Temporary package folder
OutputPackagePath: points to your final package folder
Credentials: SPO credential that has admin rights to the destination site
TargetWebUrl: point to your destination web
TargetDocumentLibraryPath: the path to your destination library

Example:

This example shows how to convert a package to a targeted one by looking up data in the target site collection. It uses the –ParallelImport parameter to boost file share migration performance.

$finalPackages = ConvertTo-SPOMigrationTargetedPackage -ParallelImport -SourceFilesPath $sourceFiles -SourcePackagePath $sourcePackage -OutputPackagePath $targetPackage -Credentials $cred -TargetWebUrl $targetWeb -TargetDocumentLibraryPath $targetDocLib

Return to top

Step 6: Submit content to import

In this step, the Invoke-SPOMigrationEncryptUploadSubmit command creates a new migration job in the target site collection, and then returns a GUID representing the JobID. This command will upload encrypted source files and manifests into temporary Azure blob storage per job.

There are four required parameters to enter (others are optional):

TargetwebURL: Points to the web of the destination
SourceFilesPath : Points to the files to import
SourcePackagePath : Points to the final manifest of the files to import
Credentials: The SharePoint Online credentials that have Site Collection Administrator rights to the destination site

Example 1:

This example shows how to submit package data to create a new migration job.

$job = Invoke-SPOMigrationEncryptUploadSubmit -SourceFilesPath $sourceFiles -SourcePackagePath $spoPackagePath -Credentials $cred -TargetWebUrl $targetWebUrl

Example 2:

This example shows how to submit package data to create new migration jobs for parallel import.

$jobs = $finalPackages | % {Invoke-SPOMigrationEncryptUploadSubmit -SourceFilesPath $_.FilesDirectory.FullName -SourcePackagePath $_.PackageDirectory.FullName -Credentials $cred -TargetWebUrl $targetWeb}

For each submitted job, the Invoke cmdlet returns these properties as part of a job:

JobId: the ID of the job in SPO
ReportingQueueUri: the SPO Azure queue that stores the real-time progress messages of the migration.
Encryption: The encryption key and method used during uploading the content to Azure. This is required when you decrypt the queue messages and import logs.

If you're using your own Azure storage account, then use Set-SPOMigrationPackageAzureSource and Submit-SPOMigrationJob to upload content into your storage.

Return to top

(Optional) Step 7: Processing and Monitoring your SPO Migration

After the job is submitted, only Azure and SPO are interacting to fetch and migrate the content into the destination. This process is timer-job based, which means it's in a queue on a first come first served basis. This does not prevent other jobs from being queued up by the same person.

There is a potential of a 1 minute delay if there are no other jobs running.

Checking job status

You can check the status of your job by viewing the real time updates posted in the Azure storage account queue by using the Encryption.EncryptionKey returned in step 6.

Viewing logs

If you're using your own Azure storage account, you can look into the manifest container in the Azure Storage for logs of everything that happened. At this stage, it is now safe to delete those containers if you don't want to keep them as backup in Azure.

If there were errors or warnings, .err and .wrn files will be created in the manifest container.

If you're using the temporary Azure storage created by Invoke-SPOMigrationEncryptUploadSubmit in step 6, the import log SAS URL can be obtained by decrypting the Azure queue message with the "Event" value "JobLogFileCreate". With the import log SAS URL, you can download the log file and decrypt it with the same encryption key as returned in Step 6.

Return to top

Scripting Scenarios for Reuse

The following is a sample script you can use that includes the complete steps from determining your locations and credentials to submitting your package data to create a new migration job.

$userName = "admin@contoso.onmicrosoft.com"

$sourceFiles = "d:\data\documents"

$packagePath = "d:\data\documentPackage"

$spoPackagePath = "d:\data\documentPackageForSPO"

$targetWebUrl = "https://contoso.sharepoint.com/sites/finance"

$targetLibrary = "Documents"

$cred = Get-Credential $userName

New-SPOMigrationPackage -SourceFilesPath $sourceFiles -OutputPackagePath $packagePath -TargetWebUrl $targetWebUrl -TargetDocumentLibraryPath $targetLibrary -IgnoreHidden -ReplaceInvalidCharacters

# Convert package to a targeted one by looking up data in target site collection

$finalPackages = ConvertTo-SPOMigrationTargetedPackage -SourceFilesPath $sourceFiles -SourcePackagePath $packagePath -OutputPackagePath $spoPackagePath -TargetWebUrl $targetWebUrl -TargetDocumentLibraryPath $targetLibrary -Credentials $cred

# Submit package data to create new migration job

$job = Invoke-SPOMigrationEncryptUploadSubmit -SourceFilesPath $sourceFiles -SourcePackagePath $spoPackagePath -Credentials $cred -TargetWebUrl $targetWebUrl

This sample shows how to get the returned information of a job, which comes in the form of a GUID.

$job = $jobs[0] $job.JobId Guid ---- 779c4b3b-ec24-4705-bb58-c38f4329418c

This sample shows how to get the $job.ReportingQueueURi.AbosoluteUri.

# To obtain the $job.ReportingQueueUri.AbsoluteUri https://spodm1bn1m013pr.queue.core.windows.net/953pq20161005-f84b9e51038b4139a179f973e95a6d6f?sv=2014-02-14&sig=TgoUcrMk1Pz8VzkswQa7owD1n8TvLmCQFZGzyV7WV8M%3D&st=2016-10-04T07%3A00%3A00Z&se=2016-10-26T07%3A00%3A00Z&sp=rap

This sample demonstrates how to obtain the encryption key and the sample return.

$job.Encryption EncryptionKey EncryptionMethod ------------- ---------------- {34, 228, 244, 194...} AES256CBC

Important: All messages are encrypted in the queue. If you want to read from the ReportingQueue, you must have the EncryptionKey.

Best Practices and Limitations

Description	Recommendation
Package size	10-20 GB Use –ParallelImport switch for File Share migration which will automatically split the big package into smaller ones.
File size	2 GB
Target size	Target site should remain non-accessible to users until migration is complete
SharePoint Online limits	SharePoint Online and OneDrive for Business: software boundaries and limitsSharePoint Online: software boundaries and limits

Azure Limits

Resource	Default/Limit
TB per storage account	500 TB
Max size of single blob container, table, or queue	500 TB
Max number of blob containers, blobs, file shares, tables, queues, entities, or messages per storage account	Only limit is the 500 TB storage account capacity
Target throughput for single blob	Up to 60 MB per second, or up to 500 requests per second

Microsoft Office Tutorials

Thursday, September 21, 2017