SharePoint Online: How Develop Analytics for SharePoint Online using POWER BI

In this article we discuss to use data stored in SharePoint Online Site to develop analytics using PowerBI.

Here is the architecture diagram to understand it in a better way:

1

For details on this architecture you can visit: POWER BI ANALYTICS POWERED BY R INTEGRATION

In order to start with this demo, we will start with two lists namely Products & Product Category having some test data to work with as shown below:

12

Now launch the PowerBI desktop

3

Click on Get Data -> More…

4

From “Online Services” category select “SharePoint Online List”

Click “Connect”

5

Specify the URL of SharePoint Online Site

Click OK

6

Select “Product” & “Product Categories” from Navigator

Click Load to load the data in the memory of report designer

7

Let the designer query the list metadata and build cache

8

Under Fields section we can “Product” & “Product Categories” added to the designer

9

Right Click on Product List and Select “Edit Query” to modify the query for fetching the data from lists

10

In the Query Editor, click on “Choose Columns” Menu -> Choose Columns

11

Select the desired columns and from the list to be included into the report. This step is useful from performance point of view as well so always try to include only relevant columns in the query

Click OK

12

We get the new result set updated based on the query modifications

13

Now modify any of the available lookup fields or Composite fields like URL in SharePoint to include the required property into the final result set

Click on the icon next to “Product Url” as indicated below

14

Select required properties like I am selecting “Url” and deselecting “Description”

Make sure you select “Use original column name as prefix” so that new property name generate based on the original column name

Click OK

15

Once we are done we are done with the modification we can see a new column appears with the name “Product Url.Url”

16

Once all the changes have been made click “Close & Apply” command button in the ribbon

17

Wait till the query changes have been saved successfully

18

Once all the changes have been saved we can see the selected query fields under “Fields” section

19

Now we establish Parent Child Relationship between “Products” & “Product Categories” list

Click “Manage Relationships” option from the Ribbon

20

On the “Manage relationships” screen Click “New” to add a new relationship between “Products” & “Product Categories” list

21

Select Primary Key & Foreign Key Columns on the “Create Relationship” screen as shown below

Select “ProductCategoryId” column from “Products” List as Foreign Key and “Id” column from “Product Categories” List as Primary Key

Also select “Cardinality” as required as “Many to one”

Once done click “Ok” to complete the process

22

And we can be able to see the new relationships created as shown below.

Click “Close” to close the “Manage Relationship” dialog

23

Once this is done we can develop reports using any available visualizations as highlighted below-

24

Here is the sample dashboard that I have developed based on the data available in SharePoint Lists

25

Hope you find it helpful.

Advertisements

SharePoint Developer Tools: How To Test & Debug SharePoint REST API Endpoints (POST Requests)

This is the second are article in the series of using Fiddler as Debugging & Testing Tool for SharePoint REST API EndPoints.

You can read the article on GET Request here:

SHAREPOINT DEVELOPER TOOLS: HOW TO TEST & DEBUG SHAREPOINT REST API ENDPOINTS (GET REQUESTS)

POST requests are different in nature than GET requests. They require more authentication layers to get through in order to push the data to SharePoint Lists and Libraries.

In order to run the POST request successfully we need an additional request header “X-RequestDigest” which is not but the User Authentication Token.

In order to request this token from SharePoint we need to make of “contextInfo” endpoint that will return the “FormDigestValue” containing the required user authentication token.

Now let see how we can request Authentication Token from SharePoint

Get Authorization Token

http://<Host Name>_api/contextinfo

1

2

Once we get the Authentication Token from SharePoint, we can add this token information in the Request Header of each of the POST requests

Request Headers

Accept: application/json;odata=verbose
Content-Type: application/json;odata=verbose
X-RequestDigest: 0xE1AE266A42214DA2940689826F68426D10620220CEDD3093CA2C234993E4ECA265BA57D357E8D3BD32F56660613CADBF72495F2C858B38F7C9B9C3CAD797F6D5,06 Feb 2017 01:22:08 -0000

Once we are ready with Request Headers we can start issuing POST Requests as shown below-

Add Data to List

Let’s consider we have a list called Categories as shown below-

3

First see the XML return based on querying schema for Categories List using following URL

http://<Host Name>/_api/Web/Lists/getByTitle('Categories')

4

Then we will see the XML return based on querying for Categories List Items using following URL

http://<Host Name>/_api/Web/Lists/getByTitle('Categories')/Items

5

Next step is to prepare the Request Body and we have to include following properties to add the items.

Please note that I am taking properties that are required for this list to add the category and add any desired number of properties to the Request Body as per the schema of the target list.

Request Body

"__metadata": { type: " SP.Data.CategoriesListItem" },
Title: "Category From Fiddler",
CategoryID: 9,
Description: “New Category Added from Fiddler”

6

Once we execute this request we can inspect the response to ensure that the request item has been added successfully to the Categories List.

7

Also we can validate this new item added by browsing Categories List

8

Update List Item

http://<Host Name>/_api/Web/Lists/getByTitle('Categories')/Items(9)

For update request you have to include “eTag” value that was returned with the item during the initial query to the Request Body. SharePoint uses this value to determine if there is any updates made to the item since it is last queried.

“If-Match: *” can be used to match any “eTag” value resulting in the operation being performed regardless of the actual value.

“X-Http-Method: PATCH” is to override the existing values

So the request body would be like this

IF-MATCH: *
X-Http-Method: PATCH
{
    "__metadata": {
    type: "SP.Data.CategoriesListItem"
},
Title: "Category From Fiddler - Updated",
Description: "New Category Added from Fiddler - Updated"
};

9

Once the request executed successfully we can see the item is updated in the Categories List

10

Delete List Item

http://<Host Name>/_api/Web/Lists/getByTitle('Categories')/Items(9)

Delete operation is more or less similar to Update operations.

11

In case of delete we will use of “X-Http-Method: DELETE” in the Request Body

Request Body

IF-MATCH: *
X-Http-Method: DELETE

12

Once the request executed successfully we can validate the item is deleted from the list.

13

Add New List

http://<Host Name>/_api/Web/Lists

Adding a new SharePoint List involve a little bit more of configuration information in Request body apart from request headers

Request Headers

Accept: application/json;odata=verbose
Content-Type: application/json;odata=verbose

Request Body

Content-Length: 0
{
"__metadata": { type: "SP.List" },
"AllowContentTypes":true,
"ContentTypesEnabled":true,
"Description":"This is Task List Created using Fiddler",
"BaseTemplate": 107,
"Title": "Task List By Fiddler"
}

14

Once this request has been executed successfully we can see the Response Body holding information about newly added SharePoint List

15

Also we can see this new list added to SharePoint by browsing the respective site

16

17

Also we can verify the “AllowContentTypes” & “ContentTypesEnabled” properties are configured as expected by browsing the Advanced Properties of the new List as shown below-

18

Delete List

http://<Host Name>/_api/Web/Lists/getByTitle('Task%20List%20By%20Fiddler')

Deleting a list is rather simpler than adding it. It takes “X-Http-Method: DELETE” to be added to the request header and rest will be done for you.

 Request Headers

Accept: application/json;odata=verbose
Content-Type: application/json;odata=verbose
Content-Length: 0
IF-MATCH: *
X-Http-Method: DELETE

19

Once the request has been completed, it will delete the required list from SharePoint Lists Collection.

20

Hope you find it helpful.

SharePoint Developer Tools: How to Test & Debug SharePoint REST API Endpoints (GET Requests)

In this article we will understand how utilize a famous developer productivity tool called fiddler as REST API Test Client for SharePoint (though the target system could be anything with a valid REST API Endpoint)

Fiddler is primarily used as a Web Proxy that can allow you intercept REST API Request – Response Cycle. The usage of this tool has increase with shift in modern SharePoint development paradigms that favors more if Client Side Development Techniques/Strategies/Platforms rather than traditional Farm Solutions.

In this upcoming section of this article I will guide on how to use Fiddler to test REST API Call against SharePoint Data.

In this article we will explore only GET type of Requests only.

To start with this demo launch Fiddler and go to “Rules” Menu and Select “Automatically Authenticate”, this will let Fiddler to authenticate you against SharePoint based on the User Token stored once.

1

If this setting is not enabled you might encounter “401 UNAUTHORIZED” as shown below-

2

Also notice the request headers that are required to execute the SharePoint REST API Endpoint

GET Requests

http://<Host Name>/_api/<SharePoint Endpoint>

Request Headers
Accept: application/json;odata=verbose
Content-Type: application/json;odata=verbose

Get Web Object

http://<Host Name>/_api/web

  • Click on “Compose” Tab
  • Select request type as “GET” from dropdown
  • Specify the Request URL as http://<Host Name>/_api/web
  • Click on “Execute” Button

3

Once the request is issued using Fiddler “Composer“, we can see the request details in the left pane

4

When you click on the request in the left pane we can see the details breakdown in the Right Pane

For instance we can click on “Inspectors” tab and then click on “JSON” tab.

JSON Tab will display the response received from SharePoint in JSON Format.

5

Similarly we can execute other GET Requests as shown in upcoming Screen Shots-

Get List Object

http://<Host Name>/_api/Web/Lists

6

7

Get Lists which are not hidden and have Items

http://<Host Name>/_api/Web/Lists?$select=Title,Hidden,ItemCount&orderby=ItemCount&$filter=((Hidden eq false) and (ItemCount gt 0))

Encoded Version of Request URL

http://<Host Name>/_api/Web/Lists?$select=Title,Hidden,ItemCount&orderby=ItemCount&$filter=((Hidden%20eq%20false)%20and%20(ItemCount%20gt%200))

8

9

Get Web filtered by Title

http://<Host Name>/_api/Web/?$select=Title

10

11

Get Web and Lists using Look Properties Expanding Lists Collection

 http://<Host Name>/_api/Web/?$select=Title,Lists/Title,Lists/Hidden,Lists/ItemCount&$expand=Lists

12

13

Get Web and Lists using Look Properties Expanding Users Collection

http://sp-2016-ddev/_api/Web/?$select=Title,CurrentUser/Id,CurrentUser/Title&$expand=CurrentUser/Id

14

15

That is all for this demo.

Hope you find it helpful.

SHAREPOINT ONLINE/OFFICE 365: HOW TO COPY EMAIL ATTACHMENTS TO SHAREPOINT DOCUMENT LIBRARY USING MICROSOFT FLOWS

In this article we will see how we can utilize Microsoft Flows to copy email attachments from Outlook Emails to SharePoint Document Library.

To continue with this demo we can start with adding a document library to SharePoint site, let’s call it “Outlook Attachments”

1

Now go to Microsoft Flow designer and start with adding a workflow

As I explained in my earlier articles we need to start with selecting a trigger for this workflow

Since we need to copy the email attachments to document library so we have two Cloud Services “Office 365 Outlook” and “SharePoint Online” to deal with

Select “Office 365 Outlook” as connector and “Office 365 Outlook – When a new email arrives” as trigger to kick off this workflow

2

Choose the Outlook folder or subfolder that you want to monitor. Here I am selecting “Inbox” as target folder. This means workflow will trigger only if the email arrives in Inbox folder and rest of the emails will be ignored.

3

Set “Has Attachment” = Yes, this settings will further filter out the incoming emails based on if they have attachments or not.

We are only monitoring only emails with attachments and that’s what these filter will achieve for us.

Set “Include Attachments” = Yes, this is required so that Workflow runtime include the binary information of each the attachments and make it available to be written to SharePoint Library.

4

Add another action with “SharePoint” as connector

Select “SharePoint – Create file” as trigger, this action will be responsible to write down the attachment binaries to SharePoint Library

5

Enter Site URL where we have “Outlook Attachments” document library sitting

Select document library name from the lookup in the Folder Path Field

6

Select “Name” property in the File Name field, this property represents the name of the attachment

7

Select “Content” in the File Content Field, this property will hold the binary data for the attachment being written to the SharePoint Document Libraries.

Here it is important to note that if we does not set the “Include Attachments” = Yes, then Content property will be null and will cause not write operations takes place

8

Add another action to the workflow, which will be responsible to send email after the attachment has been copied to the SharePoint Library

9

Select “Office 365 Outlook” as connector

Select “Office 365 Outlook – Send an email” as trigger, this trigger is responsible for sending emails

10

Choose or any valid email id in “To”, here I am just selecting the email of me

Specify Subject as applicable

Specify the Body content as required. This content can consist of dynamic tokens provided by Workflow runtime as shown below-

11

Saves the Workflow and complete it.

In the following screens we can see the complete workflow steps in one go-

Step-1

12

Step-2

13

Step-3

14

Now let’s test the workflow by sending an email with attachments to myself.

15

Once I received the email containing attachments in Inbox Folder, Workflow gets triggered

16

Just after that we can see workflow entered into the debug mode as shown below-

17

And as soon as it is completed we can see the completion notification email as shown below-

18

And we can also see the attachments copied to the “Outlook Attachments” SharePoint Library as shown below-

19

Hope you find it helpful.

SHAREPOINT ONLINE/OFFICE 365: HOW TO DEVELOP APPROVAL WORKFLOWS ON LIST ITEM UPDATE EVENT USING MICROSOFT FLOWS

In this article we will see how to deal with SharePoint Online List Item update event.

For the demo in this article I will consider a business scenario where on update of any Product in the Product list, an approval process will execute and an email is send to approver for the selection of Price Range for the Product just updated.

To start with a demo I have a Product List in SharePoint Online Site as shown below-

2

Go to “Flow” Menu and Click on “Create a Flow”

3

Here we have two choices either we can use any available workflow template that suites our requirements or we can create a new one from scratch

4

For this demo we will create a Flow using blank template

Go to Flow Site by clicking “See you flow” links

5

Click on “Create from blank” this will launch Workflow Designer

6

On the Workflow Designer, Select the Trigger

We can select trigger from the any of the available services as shown highlighted below-

7

Select SharePoint Online as target service & select for the Trigger “SharePoint-When an existing item is modified”

8

Provide the Title to the Workflow “Product Details Modification Approval Request”

9

Create SharePoint context by providing Site URL and List Name as shown-

10

You can optionally rename the Step by selecting “Rename” option as shown below-

11

Here I have changed the name of Step to “Get SharePoint Context”

12

Add another action to the workflow by clicking “Add an action” option

13

Send email by selecting “Office 365 Outlook” Service and “Office 365 Outlook-Send approval email” action

14

Rename the name of Step to “Request Approval”

15

Choose or approver’s email in “To”, here I am just selecting the email of the person who has originally created the Product in list by choosing “Created By Email” field

Specify Subject as applicable

Specify User Options that allows approver to select the approval options from within the mail

Specify the Body content as required. This content can consist of dynamic tokens provided by Workflow runtime as shown below-

16

Update List Item by selecting “SharePoint Online” Service and “SharePoint-Update item” action

17

Rename the step to “Update item”

Specify Site URL & List Name

In Id field select “ID” token provided by workflow runtime. This token represents the List item Id of the item that triggered this workflow to execute.

It is important to note that values to all mandatory fields in the list must be provided. If you don’t want to change the values you can choose same values again.

18

For Title field we have selected “Title” token that represent current value present in list for the Title field of this item

19

In “Product Price Range” field choose “SelectedOption” token that will hold the value selected by the approver from within the mail

20

Once all the Steps are configure as required we can save the workflow by clicking “Save flow”

21

Click “Done” to return back to the workflow dashboard

22

On the workflow dashboard we can see the details of services used with in this workflow

23

Now modify and the Product in Product List and notice the “Product Price Range” field which is currently set to “Low”

24

Once the Product details has been changed, this will trigger the workflow that we have developed

25

On execution the workflow we can see a new email hitting the inbox of the approver

26

Let’s consider the approver submitted “High” as approval option

27

This action will update the “Product Price Range” field for the list item to “High” as shown below-

28

Hope you find it helpful.

SharePoint Online/Office 365: How To Develop Approval Workflows On List Item Add Event Using Microsoft Flows

Microsoft introduces Flows that enables development of Workflows for SharePoint Online and other more than 50 services over the cloud.

In this article we will see the demonstration of Microsoft Flow around SharePoint Online and will deal with List Item Add Event.

For this demo let’s consider we have a Product Catalog List which required Approval to happen with each new item added.

We have a Product List as shown below-

2

Go to “Flow” Menu and Click on “Create a Flow”

3

Here we have two choices either we can use any available workflow template that suites our requirements or we can create a new one from scratch

4

For this demo we will create a Flow using blank template

Go to Flow Site by clicking “See you flow” links

5

Click on “Create from blank” this will launch Workflow Designer

6

On the Workflow Designer, Select the Trigger

We can select trigger from the any of the available services as shown highlighted below-

7

Provide the title of the Workflow

Select SharePoint Trigger “SharePoint-When a new item is created” that will fire when a new item is added to SharePoint Online List

8

Provide Site Url and List name to setup SharePoint context

Add an action to the Workflow by clicking “Add an action”

9

Select approval action by selecting “Outlook.com – Send approval email”

10

Select “Approver Email” in “To”, since this “Approver” is a People Field on SharePoint,

Workflow designer is intelligent enough to represent each property as a separate field

11

Add a condition to the workflow by clicking “Add a condition”

12

In the condition we are check if Approver Approved the Production or not.

If Approved then send approval email to requestor else delete the Product and send Email to requestor

13

Choose “Created By Email” field in “To” as it will hold the Email Id of the Requestor (Person added the Product)

Choose “Approver Email” field in “CC” to send a copy of the mail to Approver also

14

If not approved, delete the Product by choosing action “SharePoint – Delete item”

15

 

In the “Delete item” action specify SharePoint Site Url and List Name

Select “ID” token for the field “Id”. This token will hold the ID of the List Item that trigged the Workflow

16

After deleting the item we, send email to Requestor by choosing “Created By Email” in “To” field
Specify the body of email as desired
Choose “Approver Email” field in “CC” to send a copy of the mail to Approver also

17

Once all the actions are configured as required save the workflow by clicking “Save flow”

18

With this we are all done and now it is time to test the workflow

To test the Workflow, add a new item in the Product List. Specify Title and Approver and Save the Product.

Save action will trigger the Workflow that we just created

19

As a result of Workflow execution Approve will get the Email with “Approve” and “Reject” options

20

Beauty of this implementation is that Email contains clickable options for approver to submit the approver’s choices from within the email

Here Approver rejects the Product

21

As a result of this Rejection, the workflow will delete the Product from the List and send an Email Notification to Requestor and Approver informing same.

22

Product has been deleted from the list

23

Now again add a new Product to the list that will again trigger the event to execute the workflow

24

Product has been added

25

As a result of Workflow execution Approve will get the Email with “Approve” and “Reject” options

26

This time Approver approved the Product

27

As a result of this Approval, the workflow sends an Email Notification to Requestor and Approver informing same.

28

Hope you find it helpful.

SharePoint Online/2016/2013: How To Upload Large Files Using PowerShell Automation

Uploading large files to SharePoint On-Premise or Online is an obvious problem during data migration from any external systems like Lotus Notes.

Here is one of such errors which we might encounter while trying to upload a file of size greater than 250 MB-

1

In this article I will explain a data upload strategy where we can split a large file into multiple chunks of smaller size.

Solution Architecture Diagram

For better understanding we can refer to the following solution architecture diagram-

2

Based on this diagram we can conclude the following facts-
1. This solution can be hosted on multiple servers to launch parallel uploads
2. This solution can consume data from Network File Shares
3. Once data file is retrieved (say of size 300 MB), this solution will split the file (100 MB) automatically based on the pre-configured chunk size (which should not exceed the size limit of 250 MB)
4. Each chunk then appended to the file uploaded in multiple iterations

In order to start with this demo we would need a SharePoint Document Library in SharePoint Online (or On-Premise) Site as shown below-

3

Another prerequisite to demo is to have files of various sizes that we can use to upload to the document library.

I made use of following command line utility to generate files of various sizes. This utility takes destination folder path and size of the file in KBs as input.

Here is the usage example of the command line utility-

fsutil file createnew "C:\Prashant\Self Paced Training\Sample Files\2GB.txt" 2147483648

Similarly I have generated other files too as shown below-

4

Now let’s dive down into the code to understand the actual implementation.

Step 1: Declare a variables to hold the document library name & folder path. For production use I recommend to have these values in an external configuration file.

Step 2: Reading files from the folder specified in path variable in Step 1

Step 3: Loop through all the files and pass each file to the “UploadLargeFiles” function along with the document library name

5

Step 4: Generate unique upload id & get file name of the file to be uploaded

Step 5: Get handle on document library object and load the root folder (or any target folder) with in the document library

Step 6: Calculate the block size to be uploaded and total file size (as shown in the architecture diagram)

Step 7: Read the bytes from the source file and set the read buffer based on the block size

6

Step 8: Read the bytes based on the buffer limit that we set in earlier steps

7

Step 9: Check if this is the first chunk that is being uploaded, if yes then add a new file to SharePoint Document Library, get the file content based on the buffer size for the chunk and call “StartUpload” function that is defined under “Microsoft.SharePoint.Client.File” class. This will add the file to the document library but with small bunch of content only.

Step 10: Check if this is not the first chunk that is being uploaded, if yes then find the file in document library and get the handle on it

Step 11: If this is another chunk of data which is not the last chunk, this chunk will be appended to the same file by using “ContinueUpload” function that is defined under “Microsoft.SharePoint.Client.File” class. This will append the content to the file identified by Upload Id that we have initialized in earlier steps.

Step 12: If this is last chunk of data, this chunk will be appended to the same file by using “FinishUpload” function that is defined under “Microsoft.SharePoint.Client.File” class. This will append the content to the file identified by Upload Id that we have initialized in earlier steps and commits the changes to the file. Once this function completes successfully the changes will be made persistent to the file.

8

Step 13: Perform exception handling and call the “UploadLargeFileToLibrary”

9

I recommend to read the documentation on Microsoft.SharePoint.Client.File class and understand functions carefully before using it.

Once we execute this script we can see the following information-

  1. File Name to be uploaded
  2. Chunk size
  3. Total Time taken to upload the files

It is important to note that total time taken to upload the files may vary depending on the hosting environment.

File Size to be uploaded: 10 MB

10

File Size to be uploaded: 50 MB

11

File Size to be uploaded: 500 MB

12

File Size to be uploaded: 2 GB

13

Once the script executed successfully we can see the respective files uploaded to the SharePoint Online Site as shown below-

14

That is all for this demo.

This article is equally applicable for both SharePoint Online & On-Premise Versions.

Hope you find it helpful.