SharePoint Online/2016/2013: How To Upload Large Files Using PowerShell Automation

Uploading large files to SharePoint On-Premise or Online is an obvious problem during data migration from any external systems like Lotus Notes.

Here is one of such errors which we might encounter while trying to upload a file of size greater than 250 MB-

1

In this article I will explain a data upload strategy where we can split a large file into multiple chunks of smaller size.

Solution Architecture Diagram

For better understanding we can refer to the following solution architecture diagram-

2

Based on this diagram we can conclude the following facts-
1. This solution can be hosted on multiple servers to launch parallel uploads
2. This solution can consume data from Network File Shares
3. Once data file is retrieved (say of size 300 MB), this solution will split the file (100 MB) automatically based on the pre-configured chunk size (which should not exceed the size limit of 250 MB)
4. Each chunk then appended to the file uploaded in multiple iterations

In order to start with this demo we would need a SharePoint Document Library in SharePoint Online (or On-Premise) Site as shown below-

3

Another prerequisite to demo is to have files of various sizes that we can use to upload to the document library.

I made use of following command line utility to generate files of various sizes. This utility takes destination folder path and size of the file in KBs as input.

Here is the usage example of the command line utility-

fsutil file createnew "C:\Prashant\Self Paced Training\Sample Files\2GB.txt" 2147483648

Similarly I have generated other files too as shown below-

4

Now let’s dive down into the code to understand the actual implementation.

Step 1: Declare a variables to hold the document library name & folder path. For production use I recommend to have these values in an external configuration file.

Step 2: Reading files from the folder specified in path variable in Step 1

Step 3: Loop through all the files and pass each file to the “UploadLargeFiles” function along with the document library name

5

Step 4: Generate unique upload id & get file name of the file to be uploaded

Step 5: Get handle on document library object and load the root folder (or any target folder) with in the document library

Step 6: Calculate the block size to be uploaded and total file size (as shown in the architecture diagram)

Step 7: Read the bytes from the source file and set the read buffer based on the block size

6

Step 8: Read the bytes based on the buffer limit that we set in earlier steps

7

Step 9: Check if this is the first chunk that is being uploaded, if yes then add a new file to SharePoint Document Library, get the file content based on the buffer size for the chunk and call “StartUpload” function that is defined under “Microsoft.SharePoint.Client.File” class. This will add the file to the document library but with small bunch of content only.

Step 10: Check if this is not the first chunk that is being uploaded, if yes then find the file in document library and get the handle on it

Step 11: If this is another chunk of data which is not the last chunk, this chunk will be appended to the same file by using “ContinueUpload” function that is defined under “Microsoft.SharePoint.Client.File” class. This will append the content to the file identified by Upload Id that we have initialized in earlier steps.

Step 12: If this is last chunk of data, this chunk will be appended to the same file by using “FinishUpload” function that is defined under “Microsoft.SharePoint.Client.File” class. This will append the content to the file identified by Upload Id that we have initialized in earlier steps and commits the changes to the file. Once this function completes successfully the changes will be made persistent to the file.

8

Step 13: Perform exception handling and call the “UploadLargeFileToLibrary”

9

I recommend to read the documentation on Microsoft.SharePoint.Client.File class and understand functions carefully before using it.

Once we execute this script we can see the following information-

  1. File Name to be uploaded
  2. Chunk size
  3. Total Time taken to upload the files

It is important to note that total time taken to upload the files may vary depending on the hosting environment.

File Size to be uploaded: 10 MB

10

File Size to be uploaded: 50 MB

11

File Size to be uploaded: 500 MB

12

File Size to be uploaded: 2 GB

13

Once the script executed successfully we can see the respective files uploaded to the SharePoint Online Site as shown below-

14

That is all for this demo.

This article is equally applicable for both SharePoint Online & On-Premise Versions.

Hope you find it helpful.

Advertisements

SharePoint 2016/2013/Online- How to Apply Password Encryption for Component as Service using PowerShell

Recently I have developed a couple of PowerShell based components that will serve as data crawlers for federated data sources like External Web Services, SQL Server Databases, and Excel Workbooks & SharePoint Lists.

In order to authenticate the Service Accounts against all of these sources I had no choice but to embed the User Name and Passwords with in the PowerShell Code in plain text. It gets even worst when few of the Web Services could support only “Basic Authentication”.

Saving passwords in plain text to code files could lead us to the Compliance Issues and could get the solutions rejected eventually.

In order to fix this issue I have implemented a couple of mechanism to deal with each type of Authentication requirements.

In this article I will discuss the mechanism to authenticate the requests to SharePoint Lists.

In order to simplify this demo let’s consider a simple scenario where I am having a list “MyLocations” as shown below and I need to export its metadata using a PowerShell based component.

1

To keep the content crisp I will walk you through the specific section from code and skipping all the CSOM specific code which you can refer in my earlier articles if you like.

I have intentionally divided this implementation into two separate code files in order to keep the passwords safe from the developers. Intent is to get the Encryption File generated by the SharePoint Admins and provided these files to developers for so that they can use it in code directly as shown below.

In the following code snippet you can see the commands to encrypt password “12345678” and export it to a text file “BANSALP.txt”

2

This file would look like as shown below:

34

This way you can store passwords for all required service accounts in different text files without violating Security Compliance.

Now in order to pass this encrypted password to SharePoint for authentication we can make use of “System.Management.Automation.PSCredential” Class as shown below.

Here “Get-Content” Command let is used to read the content from “BANSALP.txt” file and “ConvertTo-SecureString” Command let to get the encrypted password as secure string

5

Once credential Object has been created we can assign this credential object to SharePoint Client Context “Credentials” Property

6

With this Client Context SharePoint Authenticates the incoming request based on the ACL of the requestor

Following is the outcome of the call that we have send to SharePoint:

7

I have exported the metadata to a “csv” file as well that would look like this.

8

Hope you find it helpful.

SharePoint 2016/2013/Online: Sites Health Monitoring by PowerShell Automation

It is not uncommon that we need to perform health monitoring on SharePoint Sites to ensure that we have stable and healthy SharePoint Farms and to ensure maximum possible availability of all the sites.

In this article we will discuss a simple yet powerful automation technique using PowerShell that will try ping the Site and if gets failed it will send the email notifications to the designated SharePoint Admins for the Site.

With this background let’s start with the demo…

In this demo we have two SharePoint Sites in Question as follows-

The On-Premise site is up and running while SharePoint Online Site is down and unavailable due to some technical reasons

In order to automate this monitoring process I have written a simple function in PowerShell as explained below-

Step 1: Create an Object of System.Net.WebClient Class. This object will provide us the methods to deal with Site Pages

Step 2: Make use of DownloadString Method of this class to download the html of the respective Site Page

Step 3: Check for Possible Error Messages that SharePoint Page would generally contains in case of Site is not accessible by making use of Wild Card Search using Contains Method

Step 4 & 5: We are sending emails to the SharePoint Admins Informing that a specific site is down, so that they can take appropriate actions accordingly.

1

Step 6: This is the initiation function that is having an array of SharePoint Site URLs that requires monitoring.

Step 7: This function will call for Step 1 to Step 5 for each SharePoint Site URL in a periodic manner and keep on circulating the notifications in case of health issues with SharePoint Sites.

2

Once this automation script executes we can see the following email notification for SharePoint Online Site which is currenly down:

4

And there will be no emails for SharePoint On-Premise Site since it is healthy and active as shown below:

3

That is all for this demo.

Hope you find it helpful.

SharePoint 2016/2013/Online: How download SharePoint List Item Attachments by PowerShell Automation

Downloading List Item Attachments is normally required when we are dealing with migration scenarios and sometimes for Custom Backups.

In this article we will explore the PowerShell to download the List Item attachments.

In order to setup this demo I have created a list and added 1 List Item with some of the attachments as shown below:

1

Also I have one folder in the local machine as shown below:

2

In the upcoming section we will do the Step by Step Code Analysis to get better understanding on the core functionality.

Step 1: In this step we will initialize the SharePoint Client Context by passing required credentials

3

Step 2: In this step we will initialize the SharePoint List Object by referencing it using current Client Context and setting up the disk location where the attachments will get downloaded

Step 3: In this step will first check the status of any pending request in the queue, and if there is any pending query then first execute it

Step 4: In this Step we will read the file from SharePoint List Item Attachments collection & save it to the disk location as specified in Step 2

4

Step 5: In this step finally will be call the “DownloadListItemAttachments” Function that will execute the logic we have seen in above steps

5

Once you execute the PowerShell it will ask you for the credentials as per the SharePoint Hosting Environment (On Premise or Online).

Provide the required credentials

6

If the PowerShell scripts executes successfully it will download all the attachment in the disk location as shown below:

7

8

During the real production scenarios it is always advisable to keep all attachments of a specific item in a separate folder govern by a proper naming convention to support smooth tracking of the content later on.

That is all for this demo.

Hope you find it helpful.

SharePoint 2016/2013/Online: How to Upload Files to SharePoint Document Libraries by PowerShell Automation

File upload operations are quite frequently used and often tricky when the SharePoint governance allows only SharePoint Client Side Technologies.

Recently I have come across a requirement in one of my assignments where an automated process is required to Sync the network shares with SharePoint.

This provides me the opportunity to write down the automation process using PowerShell +CSOM combination.

In this demo we will explore the PowerShell code to upload the file to SharePoint Online/On Premise Implementations which represents just a part of the complete automation process.

In order to setup this demo I have created as document library in SharePoint Online Site as shown below:

123

Let’s consider that we have a File Share somewhere on the network having files to be uploaded to the SharePoint Document Library

4

Now let’s explore the code step by step:

Step 1: In this step Client connection has been setup and the Client Context has been initiated.

The below is representing the connection to SharePoint Online Site but this will be little different in case if you want to initiate the connection with SharePoint On Premise site there will be a slight difference in the code.

For details you can visit to my earlier blogs at below URLs:

https://howtodowithsharepoint.wordpress.com/category/sharepoint-2013/sharepoint-online/

https://howtodowithsharepoint.wordpress.com/category/powershell-with-csom/

5

Step 2: In this step we will instantiate the Document Library Object that will provide us the handle on the desired document library in SharePoint Online Site.

This handle will let us allow getting reference of the desired folder in the document library. In this demo we are referring the Root Folder of the document library

Step 3: In this step we will explore the File System folder by using Get-ChildItem cmdlet on the using the specific Folder path

Step 4: In this step we will call another supporting function to upload the files to SharePoint Document Library

6

Inside UploadFile function-

Step 5: In this step File Object has been initialized using “System.IO” namespace. The file object will provide us the handle on files in the disk folder, open the file and read the content of the file in form of File Stream

Step 6: In this step we will add the file to the SharePoint Document Library folder (Root Folder in this case) as File Stream that we get it in Step 5

Step 7: Load File object & execute query to get it file actually uploaded to SharePoint Document Library

Step 8: Check in the file that has been uploaded in the Step 7

Step 9: This is showing the function call to “UploadFileToLibrary” function

7

That is all for the code.

In order to test this script we can either choose SharePoint (2016/2013/Online) Management Shell or Windows PowerShell.

Since in this demo I demonstrating the PowerShell in Conjunction with CSOM that I am going to run on client machines where we don’t have SharePoint Management Shell installed so I am going to use Windows PowerShell only.

Search for Windows PowerShell and launch the Windows PowerShell Command Prompt as shown below:

8

Refer the script & execute it

9

Provide password to when asked for

10

Once the PowerShell Script executes successfully, we can see the success messages printed on the Command Prompt

11

After the successful execution of the PowerShell script we can go back to the SharePoint Online Site and navigate to the document library that is the destination for these newly uploaded documents and sure enough we will get the documents

12

In case if you want to achieve additional automation scenarios you can execute this script as part of the Scheduled Task and that will further reduce human errors by executing the process automatically for you.

That is all for this demo.

Hope you find it helpful.

SharePoint 2016/2013/Online: How to Download SharePoint Files by PowerShell Automation

In this article we will explore the PowerShell way of downloading the files in SharePoint Document Libraries.

By adding some flavor of automation mechanism this technique could be very useful to the scenarios where we need to download documents in bulk in a schedule based job.

In order to setup this demo I have created a document library in my SharePoint Online Site, though this equally applicable to On Premise implementations of SharePoint.

I have added a new document library in SharePoint Online Site by the name “Prashant Documents” with a set of documents uploaded to it.

0.0.png

I have also set up a local folder on the disk drive which will act as destination folder for the files to be downloaded

0.1.png

Step 1: Initiate the Client Context

In order to get more information on Setting up Client Context & other Environmental Configurations, you may visit one of the earlier articles SharePoint Online: How to Install SharePoint Online Management Shell

1.png

Step 2: Instantiate & Load SharePoint Document Library Object

Step 3: Load the Root Folder in SharePoint Document Library which is containing the set of documents to be downloaded

Step 4: Using “OpenBinaryDirect” Method that is available in “Microsoft.SharePoint.Client.File” Class to read the file from SharePoint Document Library and save the File Stream to the disk location by using “CopyTo” method which is available in “System.IO.File” Class

2

Step 5: Call “DownloadFilesFromFolder” function that will download the files to the disk location

3

Step 6: On execution of this function Windows PowerShell Credential Prompt will appear where we have to specify the password to the login account

4.png

Step 7: Once the script executed successfully we can see the documents downloaded into the disk location

5.png

Hope you find it helpful.

SharePoint 2016/2013: How to Install SharePoint Client Components SDK

In this article we will look for another necessary component that we will be required when preparing client machines for CSOM based PowerShell Development.

Microsoft has packaged all the SharePoint Client Side Programming Dlls using “SharePoint <Version> (2013/2016) Client Components” package which are required to start with CSOM based PowerShell Development.

In order to get the client component for SharePoint search “SharePoint <Version> (2013/2016)  Client Components SDK”, and sure enough you will get the related links

1

Follow the link to Download Page

Click on “Download” Button

2

Choose the version of MSI file depending on the type of Operating System you are using. In my case I am using 64Bit OS so I am downloading the 64Bit version of the MSI file.

3

Once the MSI file has been downloaded, we can start the installation by Right clicking & choosing Install option from the menu

4

Select Next on the next Step of the installation wizard

5

Then click Install

6

Wait till the installer is getting completed

7

Once the installation has been completed, there will be new folder structure has been created just like as in SharePoint Server.

You will find all the SharePoint Client Side Dlls at the location your machine in ISAPI Folder on your machine as shown below:

8

You can refer these files in your PowerShell Scripts that are based on CSOM.

Hope you find it helpful.