PowerShell Automation: How to Send Email Notifications

While developing some of the automation tools I was required to send emails to the intended users sometimes to inform them about the operational progress or otherwise.

In this article I will take you through the steps to develop PowerShell functions that can send emails to the intended participants.

Let’s look into the code base used for this article-

Step – 1: This will be the final step where we are calling the “Send-Email” function, implementation of which can be seen in the upcoming steps.

Step – 2: In this step I have shown the distribution list comprising of intended participants, here in this article I choose to make it hard coded with in the code which is an absolutely bad idea J and should be avoided. Rather we should have external content source (CSVs, SharePoint Lists) to configure this list of intended recipients.

Step – 3: In this step we have declared a variable holding subject for the email

Step – 4: In this step we prepare the body of the email. It is worth noting that you can use any valid HTML to prepare the mail body. Make sure you don’t use CSS Classes as they won’t work here. You can use inline styles to format the html.

Step – 5: In this step we will call the helper function and pass on the data we have prepared in previous steps. This function has the responsibility to send the email to the required recipients.

1

 

Step – 6: In this step we get the Sender’s Id. As a part of the best practice this should be a generic Id.

Step – 7: In this we get SMTP Server Address to relay through the mail

Step – 8: In this step we are creating the array of recipient since “Send-MailMessage” function uses Array as parameter return type to “-To” parameter.

Step – 9: In this step we are calling “Send-MailMessage” function by passing required parameters as shown below-

2

That’s all for the code.

On successful execution we will get a new email in the Inbox as shown below-

3

That is all for this simple demo.

Hope you find it helpful.

SharePoint Online/2016/2013: How To Upload Large Files Using PowerShell Automation

Uploading large files to SharePoint On-Premise or Online is an obvious problem during data migration from any external systems like Lotus Notes.

Here is one of such errors which we might encounter while trying to upload a file of size greater than 250 MB-

1

In this article I will explain a data upload strategy where we can split a large file into multiple chunks of smaller size.

Solution Architecture Diagram

For better understanding we can refer to the following solution architecture diagram-

2

Based on this diagram we can conclude the following facts-
1. This solution can be hosted on multiple servers to launch parallel uploads
2. This solution can consume data from Network File Shares
3. Once data file is retrieved (say of size 300 MB), this solution will split the file (100 MB) automatically based on the pre-configured chunk size (which should not exceed the size limit of 250 MB)
4. Each chunk then appended to the file uploaded in multiple iterations

In order to start with this demo we would need a SharePoint Document Library in SharePoint Online (or On-Premise) Site as shown below-

3

Another prerequisite to demo is to have files of various sizes that we can use to upload to the document library.

I made use of following command line utility to generate files of various sizes. This utility takes destination folder path and size of the file in KBs as input.

Here is the usage example of the command line utility-

fsutil file createnew "C:\Prashant\Self Paced Training\Sample Files\2GB.txt" 2147483648

Similarly I have generated other files too as shown below-

4

Now let’s dive down into the code to understand the actual implementation.

Step 1: Declare a variables to hold the document library name & folder path. For production use I recommend to have these values in an external configuration file.

Step 2: Reading files from the folder specified in path variable in Step 1

Step 3: Loop through all the files and pass each file to the “UploadLargeFiles” function along with the document library name

5

Step 4: Generate unique upload id & get file name of the file to be uploaded

Step 5: Get handle on document library object and load the root folder (or any target folder) with in the document library

Step 6: Calculate the block size to be uploaded and total file size (as shown in the architecture diagram)

Step 7: Read the bytes from the source file and set the read buffer based on the block size

6

Step 8: Read the bytes based on the buffer limit that we set in earlier steps

7

Step 9: Check if this is the first chunk that is being uploaded, if yes then add a new file to SharePoint Document Library, get the file content based on the buffer size for the chunk and call “StartUpload” function that is defined under “Microsoft.SharePoint.Client.File” class. This will add the file to the document library but with small bunch of content only.

Step 10: Check if this is not the first chunk that is being uploaded, if yes then find the file in document library and get the handle on it

Step 11: If this is another chunk of data which is not the last chunk, this chunk will be appended to the same file by using “ContinueUpload” function that is defined under “Microsoft.SharePoint.Client.File” class. This will append the content to the file identified by Upload Id that we have initialized in earlier steps.

Step 12: If this is last chunk of data, this chunk will be appended to the same file by using “FinishUpload” function that is defined under “Microsoft.SharePoint.Client.File” class. This will append the content to the file identified by Upload Id that we have initialized in earlier steps and commits the changes to the file. Once this function completes successfully the changes will be made persistent to the file.

8

Step 13: Perform exception handling and call the “UploadLargeFileToLibrary”

9

I recommend to read the documentation on Microsoft.SharePoint.Client.File class and understand functions carefully before using it.

Once we execute this script we can see the following information-

  1. File Name to be uploaded
  2. Chunk size
  3. Total Time taken to upload the files

It is important to note that total time taken to upload the files may vary depending on the hosting environment.

File Size to be uploaded: 10 MB

10

File Size to be uploaded: 50 MB

11

File Size to be uploaded: 500 MB

12

File Size to be uploaded: 2 GB

13

Once the script executed successfully we can see the respective files uploaded to the SharePoint Online Site as shown below-

14

That is all for this demo.

This article is equally applicable for both SharePoint Online & On-Premise Versions.

Hope you find it helpful.

TypeScript: How to Include External JavaScript Frameworks to TypeScript Projects

While working in Typescript Projects I found it tricky to include external JavaScript Frameworks like JQuery.js, Moment.js and so on.

I tried to look over on google to find some help around it but don’t find any step by step tutorial to get it implemented.

The purpose of this blog post is help Typescript newbies with projects involving external JavaScript Frameworks.

1

Note: I am using Visual Studio 2015 as development IDE and these steps are specific to Visual Studio only.

In order to start with this demo, first we will create a “HTML Application with Typescript” project using Visual Studio Typescript Template

Enter “Project Name, Location and Solution Name” as required and click OK.

2

Once the Project has been created successfully we can add the require typescript code in “App.ts” file while html elements in “index.html” file

For the sake of this demo I have added simple code that requires JQuery.js

In the index.html file we can see JQuery code construct as “$(document).ready ()”, which requires “JQuery.js” as dependency.

3

In “App.ts” file I have added “$(#content).append()” which requires JQuery to be included into the typescript.

4

Now let’s try to run this project by pressing “F5” and see what we get.

And Boom. We get this error since we don’t have JQuery file included in this project as shown in the following screen shots-

5

We can further see the error message generated by runtime as below-

6

In type script projects that required external JavaScript Frameworks, we need to first install the Type Definition for the respective framework.

For example in this demo since we need to include JQuery to the project so what we need to first install Type Definition for JQuery

We can install JQuery Type Definition by using “NuGet Package Manager” from the “Tools” Menu as shown in the screen shot below

7

Once the Package Manager launches, search for “jQuery for SharePoint”

Search results will show you “jquery.TypeScript.DefinitelyTyped” which needs to be installed for this project

Select the Project and Click “Install” to install the type definitions as shown below-

8

Click “OK”, when asked to proceed with modifications to the project as shown below-

9

This will install the type definitions for this project and that we can see in the output window as shown below-

10

Once type definition has been installed we can see a new folder structure in the solution explorer which will  be having new definition file for JQuery “jquery.d.ts” as shown below-

11

Next Step is to include the reference of type definition in the code files (*.ts) like shown below-

Add /// <reference path=”scripts/typings/jquery/jquery.d.ts” /> at the top of (*.ts) files

12

Now in the html files, wherever you need to consume JQuery Framework add the reference of JQuery file as we usually do in normal web application projects

Add <script src=”https://ajax.googleapis.com/ajax/libs/jquery/3.2.1/jquery.min.js”></script> to the html files

13

Once all the changes has been made to the code files, let’s run the project again using “F5” and this time we can see it through without any errors.

14

Though this demo talks about including JQuery but this holds true for other JavaScript Frameworks.

Hope you find it helpful.

TypeScript: How To Install TypeScript Plugin for Visual Studio 2015

In this article we see the simple step by step process to install Typescript Plugin for Visual Studio.

Important to note that Typescript is already included with the base installation of Visual Studio 2015 but using the Typescript Plugin we can get the latest version of the Typescript SDK to be used with Visual Studio. Though we will consider Visual Studio 2015 in this demo but process remains the same for Visual Studio 2015 and Visual Studio 2017.

We can download the Typescript Plugin from the following location-

http://www.typescriptlang.org/index.html#download-links

As highlighted select required version of Visual Studio that you want to install this plugin on to.

We will choose Visual Studio 2015 for this demo

1

On clicking the link will take us to the Plugin download page as shown below-

Click download to start the download

2

You can run or save the file as required

3

On execution of the Plugin executable, Install wizard triggers and we can see the following screens

Agree the License Terms and click Install

4

Monitor the progress and proceed

5

Close the Setup once it is completed.

6

Now launch Visual Studio, create new project and we can see a project template under TypeScript Node as shown below:

7

We can use this template to start with a TypeScript Project.

Hope you find it helpful.

SharePoint Framework: How to Prepare Environment for Development with SharePoint Framework

Microsoft have introduced another development methodology called as SharePoint Framework, which is claimed more flexible than earlier approaches like Farm Solutions, Sandbox Solutions, SharePoint Add-ins.

SharePoint Framework is compatible with a complete Web Development Stack with an inherent support to all modern JavaScript frameworks like React, Knockout, Type Script and so on.

In order to get a better insight I would recommend you to refer to the Webinar by Andrew Connell.

In this article, we will look for steps required to get developer machine ready for starting with SharePoint Development using SharePoint Framework.

Following image displays the steps to set up the development machine:

1

Step 1: Installation of Node JS

Search for “node js” or and navigate the link as shown below to reach to the official download page for Node JS.

You can also follow the direct link to download Node JS: https://nodejs.org/en/

2

As per recommendation from the Technical Community, it is advisable to go for LTS Version of Node JS.

Download the latest version available as shown below:

3

Run the installer and follow the default options as shown below:

Click Next

4

Accept license agreement and Click Next

5

Leave the default installation directory and Click Next

6

Continue with default features and Click Next

7

Click Install to start installation of Node JS

8

Wait until installation has been completed

9

Click Finish once the installation has been completed

10

Now in order to test the installation, launch Windows PowerShell Command Prompt with Administrator privilege

Type command “npm -v”, If installation is successful you can see the version of Package Manager (NPM).

Note: NPM is the Package Manager that used for dependency management, so that you no longer have to manually download and manage your scripts.

11

Step 2: Visual Studio Code

Search for “visual studio code” or and navigate the link as shown below to reach to the official download page for “Visual Studio Code”.

You can also follow the direct link to download Visual Studio Code:  https://code.visualstudio.com/

12

Since my development environment is using Windows Server 2012 R2 as base operating system, I can download the suitable version as per my needs.

1314

Once downloaded run the installer and proceed as follows-

Click Next to initiate the installation

15

Accept the license agreement and click Next to continue

16

Keep the defaults as it is and click Next to continue

17

18

Select required options and click Next

19

Click Install to start the installation

20

Wait until installation has been completed

21

Click Finish to complete the installation process

22

On completion, install will launch the Visual Studio Code window to ensure installation is successful.

23

Step 3: Install Yeoman  & Gulp

Yeoman scaffolds out a new application, writing your build configuration (e.g Gulpfile) and pulling in relevant build tasks and package manager dependencies (e.g npm) that you might need for your build.

Gulp is the Build System used to build, preview and test your project.

In order to install Yeoman & Gulp launch windows PowerShell command prompt with Administrator privilege

Enter command “npm  install  -g  yo  gulp” and enter

This command will perform global installation of Yeoman and Gulp

24

Wait until installation has been completed

25

262728

Once done successfully it will return with command prompt without any error message.

Step: 4 Install SharePoint Yeoman Plugin (SharePoint Client Side Libraries)

In order to install SharePoint Client Side Libraries enter following command

“npm install  -g  @microsoft/sharepoint”

29

Wait until the installation has been completed

3031

Step: 5 Install SPFX Developer Certificate

Local Workbench Site, which is used to test the WebParts locally with using SharePoint Context, will notify you with the error that it is not configured to load scripts from localhost.

In order to avoid this error message we need to install SPFX Developer Certificate by using the following command

“gulp trust-dev-cert”

32

Wait till installation has been completed

Click Yes on the Security Warning to accept installation of certificate

34

With this Step 5, SharePoint Framework and all related dependencies are installed successfully and it is the time verifies the installation and in the following steps, we will perform validations.

Step: 6 Create SharePoint Project

Enter following command “yo  @microsoft/sharepoint”

35

This command will initiate the Project Configuration for SPFX based SharePoint Solution

This command will take input from the user to configure the project attributes like project name, description, directory, development environment and so on.

36

Input all information required and enter to proceed with Project configuration.

Wait till Project has been created

3738

Once project has been created successfully, we will get command prompt returned without error.

Step: 7 Execute First Project

Now this is the time to finally run the Project and testify the SharePoint Framework.

In order to run the project enter the following command

“gulp  serve”                      (case sensitive so be careful)

39

Wait until the project initialize

40

Once the Project initialized successfully, we can see SharePoint Workbench launch WebPart placeholder, mimicking SharePoint Web Page.

On this page, you can add the WebPart that has been deployed locally without using SharePoint context.

41

On click of Add button we can see available WebParts listed, select the WebPart in question and add it to the Page

42

Once added we can see the WebPart on the Page, this ensures that we have done all configuration as expected

43

In order to open the Project Code we can right click the solution directory choose option “Open with Code”

44

And we can see the Visual Studio Code launched with project dependent files

45

Command Summary

46

That is all for this demo.

Hope you find it helpful.

SharePoint 2013/2016: How to Find Duplicate Records in SharePoint List

During one of my assignments I have come across a situation where we need to fix data issues in SharePoint Lists.

One of the issues that we found was presence of duplicate data. In order to fix that problem in hand I had developed a Powershell Script to find out duplicate data based on a specific or a group of columns.

For the sake of demo, I have added a SharePoint List with some duplicate records in it as shown below:

1

Now let’s look into the code to understand implementation details-

In Step 1 we are getting references of the Site and Web where the SharePoint List resides

In Step 2 we are splitting the list of columns based on which we want to find out the duplicate data

We can see there are two input variables “ColumnToValidate” and “ColumnToDisplay”. “ColumnToValidate” provides columns based on which duplicity needs to be checked while “ColumnToDisplay” contains the list of columns that needs to be the part of data export.

In Step 3 we are creating the export folder that will hold the CSV files exported with duplicate records

In Step 4 we are creating the list object that will give the handle on the list which needs to be validated

2

In Step 5 we are getting list of Items from SharePoint List and grouping them based on the validation columns

In Step 6 we are creating the directory for export files

In Step 7 we are exporting all the groups which is having item count greater than 1 (this logic identifies the duplicate items)

3

That is all for the code.

Now we will see the variation in outputs depending on the columns specified for duplicacy check

In Step 8 we specify the validation and display columns, for the first execution we will check duplicate values in “Title” column

In Step 9 we are calling the “Get-DuplicateListItems” function to find the duplicate values

4

After the function executed successfully we can see the following output.

In Step 10 we can see the output of this excution and can see 6 items found which duplicate in Title Column

5

In Step 11 we can see the CSV file that is exported by the execution considering “Title” Column to be validated.

6

In Step 12 we can see the output file and can notice duplicate values in “Title” Column

7

In Step 13 we have changed the list of columns to be validated. In this second execution I have added another column “Role”.

Now the list will be validated for duplicity based on the combination of “Title & Role” Columns

8

In Step 14 we can see the output of this excution and can see 4 items found which duplicate in “Title & Role” Columns

9

In Step 15 we can see the CSV file that is exported by the execution considering “Title & Role” Column to be validated

10

In Step 16 we can see the output file and can notice duplicate values in “Title & Role” Column

11

In Step 17 we have changed the list of columns to be validated. In this second execution I have added another column “Location”.

Now the list will be validated for duplicity based on the combination of “Title & Role & Location” Columns

12

In Step 18 we can see the output of this excution and can see 2 items found which duplicate in “Title & Role & Location” Columns

13

In Step 19 we can see the CSV file that is exported by the execution considering “Title & Role & Location” Column to be validated

14

In Step 20 we can see the output file and can notice duplicate values in “Title & Role & Location” Column

15

This is a very simple technique that can be used to fix one of the issues with SharePoint List data.

Hope you find it helpful.

SharePoint 2016/2013/Online: How to Optimize SharePoint Custom Pages Using HTML5 IndexedDB API

In this article we will discuss another obvious performance issues with SharePoint Solutions involving large volume of data transactions surfacing SharePoint Custom Pages.

This could become more prominent if we have strict governance in place and we are not allowed to make use of advanced server side options (Custom Web Service End Point, MTA Enabled Modules etc.).

In one of the recent assignment I came across a similar scenario where I need to crawl data from an external Web Service end Point and surface data on SharePoint Pages. Since the anticipated data volume was huge and traditional caching approaches like Cookies wont’ work due to size limitations.

In pursuit of the solution I have gone through the “HTML5 Web Storage APIs” that allows you to setup an In-Browser Transactional Database System called “IndexedDB”.

Here is a quick introduction of IndexedDB for details I must recommend you to visit IndexedDB

“IndexedDB is a transactional database system, like an SQL-based RDBMS. However, unlike SQL-based RDBMSes, which use fixed-column tables, IndexedDB is a JavaScript-based object-oriented database. IndexedDB lets you store and retrieve objects that are indexed with a key; any objects supported by the structured clone algorithm can be stored. Operations performed using IndexedDB are done asynchronously, so as not to block applications.”

I also want to thanks to “Raymond Camden” for his detailed research on Storage Limits for IndexedDB and believe you must refer this link to understand the limits carefully before getting into concrete implementations.

Now let’s try to understand the implementation details by using following diagram:

Solution Architecture Diagram & Explanation

1

In this solution the SharePoint Page will try to look for the required data in Local Indexed DB created to support this page. If data is not found in local database, page will issue the request for data from SharePoint List.

Since we are dealing with “100,000” Items present in SharePoint List, I made use of “REST API + OData Continuation” data access strategy to overcome SharePoint List Threshold Limits. This mechanism will access only 100 List Items at a time and it is safe to extend this limit up to 2000 items per fetch.

Each fetch will a JSON Object that will be persisted into Indexed DB as individual record. I opt this strategy to reduce the page load time. If the number of items are not much you can add each item as separate record.

Every subsequent data call will be automatically diverted to the local database as primary source.

Additionally we can add “Auto Refresh Modules” to keep the local database fresh with SharePoint List Changes and sync the changes with Indexed DB “Asynchronously”.

Ideally speaking for a complete solution “Auto Refresh Modules” are must to have.

So this all about execution summary for this solution.

Now let’s have look at implementation details as follows-

I have created a SharePoint List with two columns and “100,000” Items added to it as shown below.

Demo

This list will be acting as data source for the page. In actual scenarios this source could be a Web Service End Point which can provide voluminous data on demand.

2

3

Before getting into code let’s see how this Page will behave on execution. Demonstrating the page in action will be helpful later when we get a deep dive in code.

If we run the page we will see this page took about “3 minutes” to get execution completed.

The first execution cycle will include the following actions:

  1. Initialize IndexedDB Database
  2. Query SharePoint List
  3. Add REST API Response to IndexedDB
  4. Load page with data from IndexedDB

Since we are adding data to the store asynchronously, overall application will remain functional even it is taking 3 minutes to complete.

4

Following screen shot showing data adding to IndexedDB asynchronously

5

We can also review the Indexed DB initialized as the part of this request using “Developer Tools or F12 Key” with in the browser as shown below-

6

We can explore each item in the each of the JSON Object as shown below-

7

Now refresh the page to see the execution again and we can see roughly “1 second” to complete the page request.

The subsequent execution cycle will include the following actions:

  1. Query IndexedDB for data
  2. Load page with data from IndexedDB

So we can see how we can trim the execution path by using a well-defined strategy.

8

Code Analysis

Let’s do the code analysis to understand the concrete implementation.

In Step 1 we are enclosing some of the literals as variables and will refer theses variables later in the code

9

In Step 2 we are checking if respective Indexed Database is initialized already or not and if not Initialize the Database. In this demos let’s call this database as “Products”

10

In Step 3 “onsuccess” event handler will get executed and database object will get stored in a global variable “SharePointOptimization.sharePointStore”. This variable will be acting as start point for all the operations on the database in future.

In Step 4 default error handling module is assigned as callback function to “onerror”, “onblocked”, “onabort” event handler

11

In Step 5 we are querying SharePoint List using REST API

12

In Step 6 we are making use of OData Continuation Techniques to overcome SharePoint List Threshold restrictions.

In this step we also call “AddDataToStore” function that will add SharePoint List Items coming as JSON Object to the Local Indexed Database.  It is important to recall that in this demo I am storing 1 JSON Object as 1 record in database and each object contains information for 100 List Items.

13

In Step 7 we are adding JSON Objects to IndexedDB. In order to do that we need to perform following operations-

  • Initialize Transaction with Read Write Operation Permissions
  • Get Handle on “Products” Database inside IndexedDB Data Stores
  • Call asynchronous “add” method to add JSON Object to “Products” Store

In Step 8 we are calling “QuerySharePoint” function to query data from SharePoint List in case data is not available in Local Database.

14

Steps 9, 10, 11 explains about “ReadSPStore” function where we will read the data from Local Data Store (IndexedDB)

In Step 9 following operations are performed-

  • Initialize Transaction with Read Operation Permissions
  • Get Handle on “Products” Database inside IndexedDB Data Stores
  • Call asynchronous “count” method to get total number of JSON Object available in “Products” Store

In Step 10 following operations are performed-

  • Check for get count request status
  • If success Initialize Indexed DB Cursor by calling asynchronous “openCursor” function

In Step 11 following operations are performed-

  • Check for get cursor request status
  • If success read the record from IndexedDB and add to the local array variable
  • Call “continue” function as long as there are items left in local store
  • Once all data is read and save to the local array pass this array to “RenderUI” function to render this data on the interface as required

15

In Step 12 we can plug any UI engine to produce more intuitive UI as applicable, for the sake of this demo I am writing out the Count of Store records * 100 (since each record contains 100 Items) to show the total number of items stored in the local store.

16

Steps 13, 14, 15 show you a helper function to check if local store contains required data or not. It helps to decide if we need to read data from Local Store or SharePoint List

“GetProductCount” function is quite similar to the “ReadSPStore” function except it perform a lesser number of operations

17

In Step 16 we will initialize Local SharePoint Store by calling “InitializeSharePointStore” function

18

In Step 17 we can see some of the UI elements to build a basic UI for this demo

19

Point of caution

Before implementing this mechanism make sure you have identified all the compatibility issues around this corner.

I would recommend you to refer the following site every now and then to make sure you are using features supported by the targeted browsers.

http://caniuse.com/#search=IndexedDb

20

Since I have made use of artifacts which are compatible with SharePoint Online Development Guidelines so we can use this approach with pages hosted in SharePoint Online as well.

That is all for this demo.

Hope you find it helpful.