SharePoint 2016: How to configure Search Service Application using PowerShell

In this article we will discuss how to provision Search Service Application for SharePoint 2016 using PowerShell

During this whole article I will guide you steps involved in provisioning Search Application along with the corresponding PowerShell Script

Here the steps that we will explore in the upcoming section-

1

2

3

Though we can provision Search Application from Central Admin as well but the purpose here is to demonstrate the use of PowerShell commands required to provision Search Application so that’s what we will do.

First let’s go to Central Admin to ensure that there is no existing instance of Search Service Application has been provisioned earlier

Under Application Management -> Click on “Manage Service Applications”

4

Here we can see what all Service Application Instances provisioned earlier and we can see there are none

5

Now launch SharePoint Management Shell to run the required PowerShell commands

6

We will run the PowerShell commands in the following order to make sure each Sub component provisioned as desired.

Step 1: Provision Service Application Instance

In this Step we will first provision Search Service Application using the following cmdlet-

$sa = New-SPEnterpriseSearchServiceApplication -Name "Search Service Application" -DatabaseName "SearchDB" –ApplicationPool  "SecurityTokenServiceApplicationPool"

Make sure Application Pool should exists before you run this command else it will fail since this cmdlet wont’ add the Application Pool automatically

7

Step 2: Provision Application Proxy

Then we need to provision Application Proxy by using following cmdlet referring the Service Application Instance provisioned in the previous step

New-SPEnterpriseSearchServiceApplicationProxy -Name "Search Service Application Proxy" -SearchApplication $sa

8

Step 3: Validate Service Instance

Next step is to validate that the Service Instance is online and to do so we can use the following cmdlet

Get-SPEnterpriseSearchServiceInstance -local

9

Step 4: Clone Topology

Next step is to clone the topology which is required in order to make any changes to the search topology in a search installation that has items in the search index

As per Microsoft recommendation around this you need to modify this new topology object, which is a clone of the active topology, by adding or removing search components. After you have made the changes to the clone topology object, you have to activate this clone to have this topology in action.

$clone = $sa.ActiveTopology.Clone()

10

Step 5: Get Search Service Instance Server Name

This Server name is required in the upcoming steps to it is wise to make use of the following cmdlet to retrieve the server name

Get-SPEnterpriseSearchServiceInstance| Select Server

11

Step 6: Get Search Service Instance

Then we get handle over the search service instance running on the respective server using the following cmdlet

$si = Get-SPEnterpriseSearchServiceInstance | ?{$_.Server -match "SP-2016-Dev"}

This reference object will be used in the upcoming steps

12

Step 7: Provision Admin Component

Next step is to provision a new Admin Component for the given topology and search service instance using the following cmdlet. This cmdlet is using reference to search service instance we get in the earlier steps

New-SPEnterpriseSearchAdminComponent -SearchTopology $clone -SearchServiceInstance $si

13

Step 8: Provision Processing Component

Next step is to provision new content processing component for the given topology and search service instance using the following cmdlet

New-SPEnterpriseSearchContentProcessingComponent -SearchTopology $clone -SearchServiceInstance $si

14

Step 9: Provision Analytics Component

Then we have to provision new analytics processing component for the given topology and search service instance using the following cmdlet

New-SPEnterpriseSearchAnalyticsProcessingComponent -SearchTopology $clone -SearchServiceInstance $si

15

Step 10: Provision Crawl Component

Then we have to provision new crawl component for the given topology and search service instance using the following cmdlet

New-SPEnterpriseSearchCrawlComponent -SearchTopology $clone -SearchServiceInstance $si

16

Step 11: Provision Index Component

Then we have to provision new index component for the given topology and search service instance using the following cmdlet

New-SPEnterpriseSearchIndexComponent -SearchTopology $clone -SearchServiceInstance $si -IndexPartition 0 -RootDirectory C:\SearchIndex\

RootDirectory: Specifies the root directory that will hold the index location for the new search index component. If you plan to isolate the index on dedicated discs in order to avoid I/O contention that may leads to performance degradation as it might be a risk that index filling up the OS disk and ruin the overall server performance.

17

Step 12: Provision Query Component

And finally we have to provision new query processing component for the given topology and search service instance using the following cmdlet

New-SPEnterpriseSearchQueryProcessingComponent -SearchTopology $clone -SearchServiceInstance $si

18

Step 13: Activate New Topology

Once all the components has been added to the new topology, activate it by using following cmdlet

$clone.Activate()

19

Step 14: Clean Inactive Topologies

Finally we have to clean all the inactive topologies associated with a search service application. We can perform this clean by using the following the code using “Remove-SPEnterpriseSearchTopology” cmdlet

foreach($tp in (Get-SPEnterpriseSearchTopology -SearchApplication $sa | ?{$_.State -eq "Inactive"}))
{    
   Remove-SPEnterpriseSearchTopology -Identity $tp -Confirm:$false
}

20

Step 15: Change Default Content Access Account

And we can change the default content access account by using the following code

$sa = Get-SPEnterpriseSearchServiceApplication

$content = New-Object Microsoft.Office.Server.Search.Administration.Content($sa)

$content.SetDefaultGatheringAccount("<Enter User Name>", (ConvertTo-SecureString "<Enter Password>" -AsPlainText -Force))

21

Now it is time to validate if search service application has been provisioned correctly.

Go to Central Admin -> Manage Service Applications

We can see a new Search Application has been provisioned successfully by clicking the Search Service Application Link

22

And we can see all the component are provisioned and running as expected.

23

Also we can validate the SQL Databases that has been provisioned during creating new search service applications as shown in the following screenshots

242526

Hope you find it helpful.

Advertisements

PowerShell Automation: How to Send Email Notifications

While developing some of the automation tools I was required to send emails to the intended users sometimes to inform them about the operational progress or otherwise.

In this article I will take you through the steps to develop PowerShell functions that can send emails to the intended participants.

Let’s look into the code base used for this article-

Step – 1: This will be the final step where we are calling the “Send-Email” function, implementation of which can be seen in the upcoming steps.

Step – 2: In this step I have shown the distribution list comprising of intended participants, here in this article I choose to make it hard coded with in the code which is an absolutely bad idea J and should be avoided. Rather we should have external content source (CSVs, SharePoint Lists) to configure this list of intended recipients.

Step – 3: In this step we have declared a variable holding subject for the email

Step – 4: In this step we prepare the body of the email. It is worth noting that you can use any valid HTML to prepare the mail body. Make sure you don’t use CSS Classes as they won’t work here. You can use inline styles to format the html.

Step – 5: In this step we will call the helper function and pass on the data we have prepared in previous steps. This function has the responsibility to send the email to the required recipients.

1

 

Step – 6: In this step we get the Sender’s Id. As a part of the best practice this should be a generic Id.

Step – 7: In this we get SMTP Server Address to relay through the mail

Step – 8: In this step we are creating the array of recipient since “Send-MailMessage” function uses Array as parameter return type to “-To” parameter.

Step – 9: In this step we are calling “Send-MailMessage” function by passing required parameters as shown below-

2

That’s all for the code.

On successful execution we will get a new email in the Inbox as shown below-

3

That is all for this simple demo.

Hope you find it helpful.

SharePoint Online/2016/2013: How To Upload Large Files Using PowerShell Automation

Uploading large files to SharePoint On-Premise or Online is an obvious problem during data migration from any external systems like Lotus Notes.

Here is one of such errors which we might encounter while trying to upload a file of size greater than 250 MB-

1

In this article I will explain a data upload strategy where we can split a large file into multiple chunks of smaller size.

Solution Architecture Diagram

For better understanding we can refer to the following solution architecture diagram-

2

Based on this diagram we can conclude the following facts-
1. This solution can be hosted on multiple servers to launch parallel uploads
2. This solution can consume data from Network File Shares
3. Once data file is retrieved (say of size 300 MB), this solution will split the file (100 MB) automatically based on the pre-configured chunk size (which should not exceed the size limit of 250 MB)
4. Each chunk then appended to the file uploaded in multiple iterations

In order to start with this demo we would need a SharePoint Document Library in SharePoint Online (or On-Premise) Site as shown below-

3

Another prerequisite to demo is to have files of various sizes that we can use to upload to the document library.

I made use of following command line utility to generate files of various sizes. This utility takes destination folder path and size of the file in KBs as input.

Here is the usage example of the command line utility-

fsutil file createnew "C:\Prashant\Self Paced Training\Sample Files\2GB.txt" 2147483648

Similarly I have generated other files too as shown below-

4

Now let’s dive down into the code to understand the actual implementation.

Step 1: Declare a variables to hold the document library name & folder path. For production use I recommend to have these values in an external configuration file.

Step 2: Reading files from the folder specified in path variable in Step 1

Step 3: Loop through all the files and pass each file to the “UploadLargeFiles” function along with the document library name

5

Step 4: Generate unique upload id & get file name of the file to be uploaded

Step 5: Get handle on document library object and load the root folder (or any target folder) with in the document library

Step 6: Calculate the block size to be uploaded and total file size (as shown in the architecture diagram)

Step 7: Read the bytes from the source file and set the read buffer based on the block size

6

Step 8: Read the bytes based on the buffer limit that we set in earlier steps

7

Step 9: Check if this is the first chunk that is being uploaded, if yes then add a new file to SharePoint Document Library, get the file content based on the buffer size for the chunk and call “StartUpload” function that is defined under “Microsoft.SharePoint.Client.File” class. This will add the file to the document library but with small bunch of content only.

Step 10: Check if this is not the first chunk that is being uploaded, if yes then find the file in document library and get the handle on it

Step 11: If this is another chunk of data which is not the last chunk, this chunk will be appended to the same file by using “ContinueUpload” function that is defined under “Microsoft.SharePoint.Client.File” class. This will append the content to the file identified by Upload Id that we have initialized in earlier steps.

Step 12: If this is last chunk of data, this chunk will be appended to the same file by using “FinishUpload” function that is defined under “Microsoft.SharePoint.Client.File” class. This will append the content to the file identified by Upload Id that we have initialized in earlier steps and commits the changes to the file. Once this function completes successfully the changes will be made persistent to the file.

8

Step 13: Perform exception handling and call the “UploadLargeFileToLibrary”

9

I recommend to read the documentation on Microsoft.SharePoint.Client.File class and understand functions carefully before using it.

Once we execute this script we can see the following information-

  1. File Name to be uploaded
  2. Chunk size
  3. Total Time taken to upload the files

It is important to note that total time taken to upload the files may vary depending on the hosting environment.

File Size to be uploaded: 10 MB

10

File Size to be uploaded: 50 MB

11

File Size to be uploaded: 500 MB

12

File Size to be uploaded: 2 GB

13

Once the script executed successfully we can see the respective files uploaded to the SharePoint Online Site as shown below-

14

That is all for this demo.

This article is equally applicable for both SharePoint Online & On-Premise Versions.

Hope you find it helpful.

SharePoint 2013/2016: How to Find Duplicate Records in SharePoint List

During one of my assignments I have come across a situation where we need to fix data issues in SharePoint Lists.

One of the issues that we found was presence of duplicate data. In order to fix that problem in hand I had developed a Powershell Script to find out duplicate data based on a specific or a group of columns.

For the sake of demo, I have added a SharePoint List with some duplicate records in it as shown below:

1

Now let’s look into the code to understand implementation details-

In Step 1 we are getting references of the Site and Web where the SharePoint List resides

In Step 2 we are splitting the list of columns based on which we want to find out the duplicate data

We can see there are two input variables “ColumnToValidate” and “ColumnToDisplay”. “ColumnToValidate” provides columns based on which duplicity needs to be checked while “ColumnToDisplay” contains the list of columns that needs to be the part of data export.

In Step 3 we are creating the export folder that will hold the CSV files exported with duplicate records

In Step 4 we are creating the list object that will give the handle on the list which needs to be validated

2

In Step 5 we are getting list of Items from SharePoint List and grouping them based on the validation columns

In Step 6 we are creating the directory for export files

In Step 7 we are exporting all the groups which is having item count greater than 1 (this logic identifies the duplicate items)

3

That is all for the code.

Now we will see the variation in outputs depending on the columns specified for duplicacy check

In Step 8 we specify the validation and display columns, for the first execution we will check duplicate values in “Title” column

In Step 9 we are calling the “Get-DuplicateListItems” function to find the duplicate values

4

After the function executed successfully we can see the following output.

In Step 10 we can see the output of this excution and can see 6 items found which duplicate in Title Column

5

In Step 11 we can see the CSV file that is exported by the execution considering “Title” Column to be validated.

6

In Step 12 we can see the output file and can notice duplicate values in “Title” Column

7

In Step 13 we have changed the list of columns to be validated. In this second execution I have added another column “Role”.

Now the list will be validated for duplicity based on the combination of “Title & Role” Columns

8

In Step 14 we can see the output of this excution and can see 4 items found which duplicate in “Title & Role” Columns

9

In Step 15 we can see the CSV file that is exported by the execution considering “Title & Role” Column to be validated

10

In Step 16 we can see the output file and can notice duplicate values in “Title & Role” Column

11

In Step 17 we have changed the list of columns to be validated. In this second execution I have added another column “Location”.

Now the list will be validated for duplicity based on the combination of “Title & Role & Location” Columns

12

In Step 18 we can see the output of this excution and can see 2 items found which duplicate in “Title & Role & Location” Columns

13

In Step 19 we can see the CSV file that is exported by the execution considering “Title & Role & Location” Column to be validated

14

In Step 20 we can see the output file and can notice duplicate values in “Title & Role & Location” Column

15

This is a very simple technique that can be used to fix one of the issues with SharePoint List data.

Hope you find it helpful.

SharePoint 2016/2013/Online- How to Apply Password Encryption for Component as Service using PowerShell

Recently I have developed a couple of PowerShell based components that will serve as data crawlers for federated data sources like External Web Services, SQL Server Databases, and Excel Workbooks & SharePoint Lists.

In order to authenticate the Service Accounts against all of these sources I had no choice but to embed the User Name and Passwords with in the PowerShell Code in plain text. It gets even worst when few of the Web Services could support only “Basic Authentication”.

Saving passwords in plain text to code files could lead us to the Compliance Issues and could get the solutions rejected eventually.

In order to fix this issue I have implemented a couple of mechanism to deal with each type of Authentication requirements.

In this article I will discuss the mechanism to authenticate the requests to SharePoint Lists.

In order to simplify this demo let’s consider a simple scenario where I am having a list “MyLocations” as shown below and I need to export its metadata using a PowerShell based component.

1

To keep the content crisp I will walk you through the specific section from code and skipping all the CSOM specific code which you can refer in my earlier articles if you like.

I have intentionally divided this implementation into two separate code files in order to keep the passwords safe from the developers. Intent is to get the Encryption File generated by the SharePoint Admins and provided these files to developers for so that they can use it in code directly as shown below.

In the following code snippet you can see the commands to encrypt password “12345678” and export it to a text file “BANSALP.txt”

2

This file would look like as shown below:

34

This way you can store passwords for all required service accounts in different text files without violating Security Compliance.

Now in order to pass this encrypted password to SharePoint for authentication we can make use of “System.Management.Automation.PSCredential” Class as shown below.

Here “Get-Content” Command let is used to read the content from “BANSALP.txt” file and “ConvertTo-SecureString” Command let to get the encrypted password as secure string

5

Once credential Object has been created we can assign this credential object to SharePoint Client Context “Credentials” Property

6

With this Client Context SharePoint Authenticates the incoming request based on the ACL of the requestor

Following is the outcome of the call that we have send to SharePoint:

7

I have exported the metadata to a “csv” file as well that would look like this.

8

Hope you find it helpful.

Create Windows Scheduler Task by PowerShell Automation

In this article we will see simple steps to configure windows scheduler tasks using PowerShell Automation.

This article can be useful in conjunction with some of the earlier articles that I have written on Process Automation using PowerShell

Before getting into the code, let see the Windows Task Manager to understand where we can find tasks which are getting created by the code

Type “Task Manager”

1

You can see the highlighted section where you can find newly created Tasks

2

The code is pretty simple and based on standard PowerShell Command-lets as described below:

3

In Step 1, we are defining Trigger for the Task by using “New-ScheduledTaskTrigger” Command-Let. It could be any permissible Time Unit as shown below

4

In Step 2 & 3 we are setting up User Name and Password that the Task will be used as Owner Account to run the defined action

In Step 4 we are defining the action by using “New-ScheduledTaskAction” Command-Let that needs to be executed by the task when trigger reached, in this demo this task will execute a PowerShell script

Finally in Step-5 we are registering the tasks by using “Register- ScheduledTask” Command-Let. This Command-Let takes Name, Trigger, UserName, Password and Action as input parameter (as defined in above steps)

Once the script gets executed a new task will be added to the Windows Task Scheduler

5

We can see the new task added in Task Scheduler Window as shown below:

6

If we double click the this task, it will open the task configuration screen that we can use verify the task properties as explained below-

General Tab: Here we can see the name of the Task & User Account that will be used to execute the defined action

7

Triggers Tab: Here we can see the Trigger defined for this task

8

Actions Tab: Here we can see the action defined for this task. In this demo we have specified a PowerShell Script to be executed whenever respective trigger achieved

9

In case if you make any changes to this task Windows will present you a password dialog to confirm the User Credentials, If asked enter the respective credentials

10

This is a simple implementation that allows you to configure Windows Tasks using PowerShell Automation.

That is all for this demo.

Hope you find it helpful.

SharePoint 2016/2013 : Event Log Monitoring by PowerShell Automation

This article is based on the requirement I recently encounter where I was required to monitor a specific exception type and if it occurs the Admins should be notified at the same time.

In this demo I am considering a scenario that if ever we have encountered an Event ID “1101” that would mean the SharePoint Site or any related service is down and in that case the Administrators will get Email Notifications automatically.

1

In order to accomplish this let start with writing a PowerShell function “Monitor-Event-Logs” as shown in Step 1

In Step 2 we are making use of “Get-EventLog” commandlet by instructing it to get the top 1 latest Application Log where the Event ID = “1101”

We can check for Event Object for null and if it returns the data  we prepare the Email Content comprising of relevant data in as shown in Step 3

In Step 4 we are sending Email Notification to the Administrators by using another generic function “Send-Email”

23

There is an external function “Execute-Process” that will call the “Monitor-Event-Logs” function by passing required Event ID as shown in Step 5

In Step 6 we are initializing the required variables pointing to the email ids of the respective contact persons

Finally in Step 7 we will call the “Execute-Process” function that will drive the whole mechanics.

4

Once the function gets executed we can see the email arrived notifying the error to the administrators as shown below:

56

To make this process more intuitive I got this script scheduled using Windows Task Scheduler to run on a specific time intervals and scan the logs for specific Event IDs.

In my actual implementation I design the “Monitor-Event-Logs” function to accept an array of Event IDs to be monitored so you can try it that way depending on your requirements.

Hope you find it helpful.