Automating Sharepoint List And Library Migration With Powershell

The Need for Automation

Manually migrating SharePoint lists and document libraries from one site to another or between SharePoint on-premises and SharePoint Online can be an extremely tedious and error-prone process. When faced with migrating multiple SharePoint lists and libraries containing hundreds or thousands of list items and documents, attempting the migration by hand is impractical.

PowerShell provides system administrators a flexible, scriptable way to programmatically migrate SharePoint lists and libraries between old and new SharePoint environments. By leveraging SharePoint Online Management Shell and PowerShell scripts, massive amounts of SharePoint list and library data can be rapidly exported, transformed, and imported between SharePoint sites with far less effort than manual migration.

Preparing for Migration

The first step when planning a PowerShell-based migration of SharePoint lists and libraries is to clearly identify the source SharePoint on-premises or SharePoint Online site and target destination site URLs. All credentials and permissions must be confirmed for accessing both the source and destination, with the user context for the migration having read access on all source content databases and write access on the destination libraries and lists.

The SharePoint Online Management Shell or PnP PowerShell module must be installed before running SharePoint Online PowerShell cmdlets. The management shell handles authentication and provides access for querying SharePoint Online APIs and executing commands.

Querying for Lists and Libraries

SharePoint PowerShell allows querying a site to retrieve all lists and document libraries or filtering on specific types like custom lists, document libraries, wiki pages, discussion boards, and more. Useful metadata like the list item count, date created, or other properties can also be queried on the lists and libraries during migration analysis.

Here is an example PowerShell snippet for getting all custom lists from the SharePoint site:

$lists = Get-PnPList | Where-Object { $_.BaseTemplate -eq "CustomList" }

Document libraries containing Office documents, PDFs, images, and other files can be queried using similar Filter parameters.

Exporting List and Library Contents

Once the source lists and libraries have been identified, the list items and documents can be exported to a portable intermediary format like CSV for simplified data transfer.

Critical metadata like authors, timestamps, and version histories can all be preserved alongside the core list item data using built-in column mappings in the SharePoint PowerShell export cmdlets. Document libraries will also automatically extract attached file binaries, with the export including the full file content alongside list metadata.

For extremely large lists and libraries exceeding list throttling thresholds, batched exports retrieving 5000 items at a time will ensure all data including attachments is reliably migrated without timeouts or throttling issues.

Importing Data to the Destination

Before list import, the target SharePoint Online site or on-premises environment must contain the destination libraries and lists matching the appropriate templates and column definitions to receive the migrated data.

PowerShell scripts can dynamically create custom lists and document libraries on the destination if they do not already exist. Column schema between old and new lists can be programmatically compared, aligning column names, types and mandatory settings to allow seamless import of list contents without manual list re-creation.

Using the PowerShell bulk CVS import capability, even millions of list items can be rapidly inserted into the new SharePoint lists in batches avoiding threshold limits. Document binaries from library exports will attach directly as files back into the new SharePoint library or list upon import completion.

Verifying Successful Migration

Once PowerShell data migration scripts finish runtime, both source and destination sites can be re-queried to validate list item totals match exactly, confirming full completion of the migration process. Data samples from the newly migrated lists and libraries should be extensively tested to ensure not only totals match, but specific documents, list items and associated metadata exported and imported successfully.

Special handling may be required to re-link lookup columns, managed metadata terms, user profile data or other complex fields between the two SharePoint environments for certain list types.

Example Migration Script

Below is a walkthrough of a sample PowerShell script migrating a custom Contacts list from an on-premises SharePoint 2013 site collection to Office 365 SharePoint Online using a CSV intermediary and the SharePoint PnP PowerShell module:

#Parameters
$SourceSiteUrl = "https://sp2013.contoso.com" 
$SourceListName = "Contacts"
  
$TargetSiteUrl = "https://contoso.sharepoint.com/sites/newintranet"
$TargetListName = "Contacts"

#Install PnP Module
Install-Module SharePointPnPPowerShellOnline 

#Connect to SPO Service  
Connect-PnPOnline $TargetSiteUrl –Credentials (Get-Credential)   

#Get Source List Items
$SourceListItems = Get-PnPListItem –SiteUrl SourceSiteUrl –List SourceListName –Query "<Fetch All Items>"

#Export Source Items to CSV 
Export-Csv $SourceListItems –Path C:\Temp\Contacts.csv

#Create Target List (if doesn't exist)
$TargetList = Get-PnPList | Where-Object {$_.Title -eq $TargetListName}  
if ($TargetList -eq $null) {
    New-PnPList –Title $TargetListName –Template Contacts 
}

#Import CSV rows to Target List
Import-Csv C:\Temp\Contacts.csv | ForEach { 
    $ListItem = ConvertTo-SharePointBatchFormat $_ 
    Add-PnPListItem –List $TargetListName -Values $ListItem 
}
  
#Report Results
Write-Output "Contacts list migration complete!"

This covers the core technique of batching export CSV, creating the destination list if needed, and iterating through each CSV row to insert as new SharePoint list items on the target. Additional validation and error handling should be added to production migration scripts.

Additional Tips and Tricks

Some additional best practices to improve SharePoint list and library data migration with PowerShell include:

• Using batched queries minimizes timeouts and throttling triggering errors
• Reduce open PowerShell sessions against a single tenant to avoid resource overload
• Schedule recurring incremental migration scripts via Azure Automation or similar
• Custom PowerShell handlers can migrate unique metadata columns or filtered subsets of data