Azure & Office 365

You can use below PowerShell to find all duplicate files(regardless of document libraries) on SharePoint online site.

Make sure you update the parameters such as SiteURL and ReportOutput. This may take time based on number of items in SharePoint site. I have kept it running for 3 days and it worked like a charm. I have seen other Powershell’s which don’t work if you have MFA enabled whereas this script works regardless you have MFA enabled or disabled.

#Before running the below script, kindly follow below steps :
#1. Open you PowerShell ISE in your system and run it as administrator
 
#cls2. Install the New PnP PowerShell Module using below commands:
        Install-Module PnP.PowerShell
 
#3. Register a new Azure AD Application and Grant Access to the tenant
        Register-PnPManagementShellAccess
 
#Then paste and run below pnp script:

#Parameters
$SiteURL = "https://tenantname.sharepoint.com/sites/Sitename"
$Pagesize = 2000
$ReportOutput = "C:\Temp\Sitename.csv"
 
#Connect to SharePoint Online site
Connect-PnPOnline $SiteURL -Interactive
  
#Array to store results
$DataCollection = @()
 
#Get all Document libraries
$DocumentLibraries = Get-PnPList | Where-Object {$_.BaseType -eq "DocumentLibrary" -and $_.Hidden -eq $false -and $_.ItemCount -gt 0 -and $_.Title -Notin("Site Pages","Style Library", "Preservation Hold Library")}
 
#Iterate through each document library
ForEach($Library in $DocumentLibraries)
{   
    #Get All documents from the library
    $global:counter = 0;
    $Documents = Get-PnPListItem -List $Library -PageSize $Pagesize -Fields ID, File_x0020_Type -ScriptBlock `
        { Param($items) $global:counter += $items.Count; Write-Progress -PercentComplete ($global:Counter / ($Library.ItemCount) * 100) -Activity `
             "Getting Documents from Library '$($Library.Title)'" -Status "Getting Documents data $global:Counter of $($Library.ItemCount)";} | Where {$_.FileSystemObjectType -eq "File"}
   
    $ItemCounter = 0
    #Iterate through each document
    Foreach($Document in $Documents)
    {
        #Get the File from Item
        $File = Get-PnPProperty -ClientObject $Document -Property File
 
        #Get The File Hash
        $Bytes = $File.OpenBinaryStream()
        Invoke-PnPQuery
        $MD5 = New-Object -TypeName System.Security.Cryptography.MD5CryptoServiceProvider
        $HashCode = [System.BitConverter]::ToString($MD5.ComputeHash($Bytes.Value))
  
        #Collect data       
        $Data = New-Object PSObject
        $Data | Add-Member -MemberType NoteProperty -name "FileName" -value $File.Name
        $Data | Add-Member -MemberType NoteProperty -Name "HashCode" -value $HashCode
        $Data | Add-Member -MemberType NoteProperty -Name "URL" -value $File.ServerRelativeUrl
        $Data | Add-Member -MemberType NoteProperty -Name "FileSize" -value $File.Length       
        $DataCollection += $Data
        $ItemCounter++
        Write-Progress -PercentComplete ($ItemCounter / ($Library.ItemCount) * 100) -Activity "Collecting data from Documents $ItemCounter of $($Library.ItemCount) from $($Library.Title)" `
                     -Status "Reading Data from Document '$($Document['FileLeafRef']) at '$($Document['FileRef'])"
    }
}
#Get Duplicate Files by Grouping Hash code
$Duplicates = $DataCollection | Group-Object -Property HashCode | Where {$_.Count -gt 1}  | Select -ExpandProperty Group
Write-host "Duplicate Files Based on File Hashcode:"
$Duplicates | Format-table -AutoSize
 
#Export the duplicates results to CSV
$Duplicates | Export-Csv -Path $ReportOutput -NoTypeInformation

Comments are closed.