SharePoint Developer Tools: How To Test & Debug SharePoint REST API Endpoints (POST Requests)

This is the second are article in the series of using Fiddler as Debugging & Testing Tool for SharePoint REST API EndPoints.

You can read the article on GET Request here:

SHAREPOINT DEVELOPER TOOLS: HOW TO TEST & DEBUG SHAREPOINT REST API ENDPOINTS (GET REQUESTS)

POST requests are different in nature than GET requests. They require more authentication layers to get through in order to push the data to SharePoint Lists and Libraries.

In order to run the POST request successfully we need an additional request header “X-RequestDigest” which is not but the User Authentication Token.

In order to request this token from SharePoint we need to make of “contextInfo” endpoint that will return the “FormDigestValue” containing the required user authentication token.

Now let see how we can request Authentication Token from SharePoint

Get Authorization Token

http://<Host Name>_api/contextinfo

1

2

Once we get the Authentication Token from SharePoint, we can add this token information in the Request Header of each of the POST requests

Request Headers

Accept: application/json;odata=verbose
Content-Type: application/json;odata=verbose
X-RequestDigest: 0xE1AE266A42214DA2940689826F68426D10620220CEDD3093CA2C234993E4ECA265BA57D357E8D3BD32F56660613CADBF72495F2C858B38F7C9B9C3CAD797F6D5,06 Feb 2017 01:22:08 -0000

Once we are ready with Request Headers we can start issuing POST Requests as shown below-

Add Data to List

Let’s consider we have a list called Categories as shown below-

3

First see the XML return based on querying schema for Categories List using following URL

http://<Host Name>/_api/Web/Lists/getByTitle('Categories')

4

Then we will see the XML return based on querying for Categories List Items using following URL

http://<Host Name>/_api/Web/Lists/getByTitle('Categories')/Items

5

Next step is to prepare the Request Body and we have to include following properties to add the items.

Please note that I am taking properties that are required for this list to add the category and add any desired number of properties to the Request Body as per the schema of the target list.

Request Body

"__metadata": { type: " SP.Data.CategoriesListItem" },
Title: "Category From Fiddler",
CategoryID: 9,
Description: “New Category Added from Fiddler”

6

Once we execute this request we can inspect the response to ensure that the request item has been added successfully to the Categories List.

7

Also we can validate this new item added by browsing Categories List

8

Update List Item

http://<Host Name>/_api/Web/Lists/getByTitle('Categories')/Items(9)

For update request you have to include “eTag” value that was returned with the item during the initial query to the Request Body. SharePoint uses this value to determine if there is any updates made to the item since it is last queried.

“If-Match: *” can be used to match any “eTag” value resulting in the operation being performed regardless of the actual value.

“X-Http-Method: PATCH” is to override the existing values

So the request body would be like this

IF-MATCH: *
X-Http-Method: PATCH
{
    "__metadata": {
    type: "SP.Data.CategoriesListItem"
},
Title: "Category From Fiddler - Updated",
Description: "New Category Added from Fiddler - Updated"
};

9

Once the request executed successfully we can see the item is updated in the Categories List

10

Delete List Item

http://<Host Name>/_api/Web/Lists/getByTitle('Categories')/Items(9)

Delete operation is more or less similar to Update operations.

11

In case of delete we will use of “X-Http-Method: DELETE” in the Request Body

Request Body

IF-MATCH: *
X-Http-Method: DELETE

12

Once the request executed successfully we can validate the item is deleted from the list.

13

Add New List

http://<Host Name>/_api/Web/Lists

Adding a new SharePoint List involve a little bit more of configuration information in Request body apart from request headers

Request Headers

Accept: application/json;odata=verbose
Content-Type: application/json;odata=verbose

Request Body

Content-Length: 0
{
"__metadata": { type: "SP.List" },
"AllowContentTypes":true,
"ContentTypesEnabled":true,
"Description":"This is Task List Created using Fiddler",
"BaseTemplate": 107,
"Title": "Task List By Fiddler"
}

14

Once this request has been executed successfully we can see the Response Body holding information about newly added SharePoint List

15

Also we can see this new list added to SharePoint by browsing the respective site

16

17

Also we can verify the “AllowContentTypes” & “ContentTypesEnabled” properties are configured as expected by browsing the Advanced Properties of the new List as shown below-

18

Delete List

http://<Host Name>/_api/Web/Lists/getByTitle('Task%20List%20By%20Fiddler')

Deleting a list is rather simpler than adding it. It takes “X-Http-Method: DELETE” to be added to the request header and rest will be done for you.

 Request Headers

Accept: application/json;odata=verbose
Content-Type: application/json;odata=verbose
Content-Length: 0
IF-MATCH: *
X-Http-Method: DELETE

19

Once the request has been completed, it will delete the required list from SharePoint Lists Collection.

20

Hope you find it helpful.

Advertisements

SharePoint Developer Tools: How to Test & Debug SharePoint REST API Endpoints (GET Requests)

In this article we will understand how utilize a famous developer productivity tool called fiddler as REST API Test Client for SharePoint (though the target system could be anything with a valid REST API Endpoint)

Fiddler is primarily used as a Web Proxy that can allow you intercept REST API Request – Response Cycle. The usage of this tool has increase with shift in modern SharePoint development paradigms that favors more if Client Side Development Techniques/Strategies/Platforms rather than traditional Farm Solutions.

In this upcoming section of this article I will guide on how to use Fiddler to test REST API Call against SharePoint Data.

In this article we will explore only GET type of Requests only.

To start with this demo launch Fiddler and go to “Rules” Menu and Select “Automatically Authenticate”, this will let Fiddler to authenticate you against SharePoint based on the User Token stored once.

1

If this setting is not enabled you might encounter “401 UNAUTHORIZED” as shown below-

2

Also notice the request headers that are required to execute the SharePoint REST API Endpoint

GET Requests

http://<Host Name>/_api/<SharePoint Endpoint>

Request Headers
Accept: application/json;odata=verbose
Content-Type: application/json;odata=verbose

Get Web Object

http://<Host Name>/_api/web

  • Click on “Compose” Tab
  • Select request type as “GET” from dropdown
  • Specify the Request URL as http://<Host Name>/_api/web
  • Click on “Execute” Button

3

Once the request is issued using Fiddler “Composer“, we can see the request details in the left pane

4

When you click on the request in the left pane we can see the details breakdown in the Right Pane

For instance we can click on “Inspectors” tab and then click on “JSON” tab.

JSON Tab will display the response received from SharePoint in JSON Format.

5

Similarly we can execute other GET Requests as shown in upcoming Screen Shots-

Get List Object

http://<Host Name>/_api/Web/Lists

6

7

Get Lists which are not hidden and have Items

http://<Host Name>/_api/Web/Lists?$select=Title,Hidden,ItemCount&orderby=ItemCount&$filter=((Hidden eq false) and (ItemCount gt 0))

Encoded Version of Request URL

http://<Host Name>/_api/Web/Lists?$select=Title,Hidden,ItemCount&orderby=ItemCount&$filter=((Hidden%20eq%20false)%20and%20(ItemCount%20gt%200))

8

9

Get Web filtered by Title

http://<Host Name>/_api/Web/?$select=Title

10

11

Get Web and Lists using Look Properties Expanding Lists Collection

 http://<Host Name>/_api/Web/?$select=Title,Lists/Title,Lists/Hidden,Lists/ItemCount&$expand=Lists

12

13

Get Web and Lists using Look Properties Expanding Users Collection

http://sp-2016-ddev/_api/Web/?$select=Title,CurrentUser/Id,CurrentUser/Title&$expand=CurrentUser/Id

14

15

That is all for this demo.

Hope you find it helpful.

SharePoint 2016/2013/Online: How to Optimize SharePoint Custom Pages Using HTML5 IndexedDB API

In this article we will discuss another obvious performance issues with SharePoint Solutions involving large volume of data transactions surfacing SharePoint Custom Pages.

This could become more prominent if we have strict governance in place and we are not allowed to make use of advanced server side options (Custom Web Service End Point, MTA Enabled Modules etc.).

In one of the recent assignment I came across a similar scenario where I need to crawl data from an external Web Service end Point and surface data on SharePoint Pages. Since the anticipated data volume was huge and traditional caching approaches like Cookies wont’ work due to size limitations.

In pursuit of the solution I have gone through the “HTML5 Web Storage APIs” that allows you to setup an In-Browser Transactional Database System called “IndexedDB”.

Here is a quick introduction of IndexedDB for details I must recommend you to visit IndexedDB

“IndexedDB is a transactional database system, like an SQL-based RDBMS. However, unlike SQL-based RDBMSes, which use fixed-column tables, IndexedDB is a JavaScript-based object-oriented database. IndexedDB lets you store and retrieve objects that are indexed with a key; any objects supported by the structured clone algorithm can be stored. Operations performed using IndexedDB are done asynchronously, so as not to block applications.”

I also want to thanks to “Raymond Camden” for his detailed research on Storage Limits for IndexedDB and believe you must refer this link to understand the limits carefully before getting into concrete implementations.

Now let’s try to understand the implementation details by using following diagram:

Solution Architecture Diagram & Explanation

1

In this solution the SharePoint Page will try to look for the required data in Local Indexed DB created to support this page. If data is not found in local database, page will issue the request for data from SharePoint List.

Since we are dealing with “100,000” Items present in SharePoint List, I made use of “REST API + OData Continuation” data access strategy to overcome SharePoint List Threshold Limits. This mechanism will access only 100 List Items at a time and it is safe to extend this limit up to 2000 items per fetch.

Each fetch will a JSON Object that will be persisted into Indexed DB as individual record. I opt this strategy to reduce the page load time. If the number of items are not much you can add each item as separate record.

Every subsequent data call will be automatically diverted to the local database as primary source.

Additionally we can add “Auto Refresh Modules” to keep the local database fresh with SharePoint List Changes and sync the changes with Indexed DB “Asynchronously”.

Ideally speaking for a complete solution “Auto Refresh Modules” are must to have.

So this all about execution summary for this solution.

Now let’s have look at implementation details as follows-

I have created a SharePoint List with two columns and “100,000” Items added to it as shown below.

Demo

This list will be acting as data source for the page. In actual scenarios this source could be a Web Service End Point which can provide voluminous data on demand.

2

3

Before getting into code let’s see how this Page will behave on execution. Demonstrating the page in action will be helpful later when we get a deep dive in code.

If we run the page we will see this page took about “3 minutes” to get execution completed.

The first execution cycle will include the following actions:

  1. Initialize IndexedDB Database
  2. Query SharePoint List
  3. Add REST API Response to IndexedDB
  4. Load page with data from IndexedDB

Since we are adding data to the store asynchronously, overall application will remain functional even it is taking 3 minutes to complete.

4

Following screen shot showing data adding to IndexedDB asynchronously

5

We can also review the Indexed DB initialized as the part of this request using “Developer Tools or F12 Key” with in the browser as shown below-

6

We can explore each item in the each of the JSON Object as shown below-

7

Now refresh the page to see the execution again and we can see roughly “1 second” to complete the page request.

The subsequent execution cycle will include the following actions:

  1. Query IndexedDB for data
  2. Load page with data from IndexedDB

So we can see how we can trim the execution path by using a well-defined strategy.

8

Code Analysis

Let’s do the code analysis to understand the concrete implementation.

In Step 1 we are enclosing some of the literals as variables and will refer theses variables later in the code

9

In Step 2 we are checking if respective Indexed Database is initialized already or not and if not Initialize the Database. In this demos let’s call this database as “Products”

10

In Step 3 “onsuccess” event handler will get executed and database object will get stored in a global variable “SharePointOptimization.sharePointStore”. This variable will be acting as start point for all the operations on the database in future.

In Step 4 default error handling module is assigned as callback function to “onerror”, “onblocked”, “onabort” event handler

11

In Step 5 we are querying SharePoint List using REST API

12

In Step 6 we are making use of OData Continuation Techniques to overcome SharePoint List Threshold restrictions.

In this step we also call “AddDataToStore” function that will add SharePoint List Items coming as JSON Object to the Local Indexed Database.  It is important to recall that in this demo I am storing 1 JSON Object as 1 record in database and each object contains information for 100 List Items.

13

In Step 7 we are adding JSON Objects to IndexedDB. In order to do that we need to perform following operations-

  • Initialize Transaction with Read Write Operation Permissions
  • Get Handle on “Products” Database inside IndexedDB Data Stores
  • Call asynchronous “add” method to add JSON Object to “Products” Store

In Step 8 we are calling “QuerySharePoint” function to query data from SharePoint List in case data is not available in Local Database.

14

Steps 9, 10, 11 explains about “ReadSPStore” function where we will read the data from Local Data Store (IndexedDB)

In Step 9 following operations are performed-

  • Initialize Transaction with Read Operation Permissions
  • Get Handle on “Products” Database inside IndexedDB Data Stores
  • Call asynchronous “count” method to get total number of JSON Object available in “Products” Store

In Step 10 following operations are performed-

  • Check for get count request status
  • If success Initialize Indexed DB Cursor by calling asynchronous “openCursor” function

In Step 11 following operations are performed-

  • Check for get cursor request status
  • If success read the record from IndexedDB and add to the local array variable
  • Call “continue” function as long as there are items left in local store
  • Once all data is read and save to the local array pass this array to “RenderUI” function to render this data on the interface as required

15

In Step 12 we can plug any UI engine to produce more intuitive UI as applicable, for the sake of this demo I am writing out the Count of Store records * 100 (since each record contains 100 Items) to show the total number of items stored in the local store.

16

Steps 13, 14, 15 show you a helper function to check if local store contains required data or not. It helps to decide if we need to read data from Local Store or SharePoint List

“GetProductCount” function is quite similar to the “ReadSPStore” function except it perform a lesser number of operations

17

In Step 16 we will initialize Local SharePoint Store by calling “InitializeSharePointStore” function

18

In Step 17 we can see some of the UI elements to build a basic UI for this demo

19

Point of caution

Before implementing this mechanism make sure you have identified all the compatibility issues around this corner.

I would recommend you to refer the following site every now and then to make sure you are using features supported by the targeted browsers.

http://caniuse.com/#search=IndexedDb

20

Since I have made use of artifacts which are compatible with SharePoint Online Development Guidelines so we can use this approach with pages hosted in SharePoint Online as well.

That is all for this demo.

Hope you find it helpful.

 

SharePoint 2013 : How to overcome List Threshold Limits using OData Continuation

While working with SharePoint Large List we all must have encountered the issues with restricted access to the list due to List Threshold Limits.

In this article we see the solution to this problem that can be implemented using OData Continuation in conjunction with REST API.

In order to demonstrate this solution I am making use of SharePoint Hosted App as in shown in below steps-

  • Create SharePoint App Project by choosing “App for SharePoint 2013” Project Template

1

  • Specify Host Web URL
  • Select SharePoint -Hosted as hosting model

2

Wait while Visual Studio configure the Project for you

3

In order to showcase the data access from a Large SharePoint List, I have added a list with 1 Lakh items in it and default Threshold limits are still intact

4

Now add some HTML for the UI in the Start Page of the App (default.aspx)

Step 1: Adding container for the HTML Button

Step 2: Adding HTML Button that will execute the code in action

5

So the final UI would look like as below-

6.1

Now add some JavaScript code in App.js File

Step 3: Bind the event handler with the HTML Button in “document.ready()” function

Step 4: Adding a helper function “getQueryStringParameter()” that will help us to extract the Query String Parameters

6

In the Callback function “oDataContinuations” we have following steps executing-

Step 5: Getting value of “SPHostUrl” by using “getQueryStringParameter()” helper function

Step 6: Getting value of “SPAppWebUrl” by using “getQueryStringParameter()” helper function

Step 7: Loading “SP.RequestExecutor.js ” JavaScript file as we need to issue a Cross Domain Call from SharePoint App to Host Web

Step 8: Prepare the URL to execute the Cross Domain Call to the Large List present in the Host Web

Step 9: Instantiate the object of Request Executor based on the App URL

Step 10: Calling “executeAsync” method of Request Executor Object

This function needs a JSON object with configuration values like

url: Specify the Url to execute Cross Domain Call

method: Specify the Request access method (GET or POST)

datatype: Specify the expected return type of the response (JSON)

headers: Specify the Request Headers

Step 11: On successful execution of this request, response object would return back with a property “__next” which contains the direct URL to the next set of records which can be queried from the list, check for this property and make sure it exists before proceed further

7.1

Step 12: Get the Url to the next result set

Step 13: Call the “getItems()” method recursively with the new Url returned by response object “__next” property

7

With this we are all done with the code.

But before we move any further we need to allow the permission for the App on Host Web since we need to query the data from the list which is hanging inside the Host Web.

Step 1: Go to App Manifest File => Permission Tab

Step 2: Specify Read permission

8

Build the Solution and Deploy it

9

Trust the App when asked for

10

Provide the credentials when asked for

11

Click on “Manage Threshold Limits”

12

See to the Result Panel and sure enough you will find records adding continuously based on the Paginated REST Calls driven by Odata Continuation

13

This is a simple demonstration to showcase how we can effectively make use of Paginated REST Calls driven by Odata Continuation to overcome issues related to List Threshold Limits.

Hope you find it helpful.

SharePoint 2013: Export To Excel Using REST API

In this blog post we will discuss how can we export the SharePoint List Data using a custom solution approach utilizing SharePoint REST API.

Though this feature is also available OOB through SharePoint List/Libraries Ribbon Control, but limited under the scenarios mentioned below.

This solution can be utilized to overcome the OOB limitations under following scenarios:

  1. When we need to trigger export process on a custom events
  2. When we need to export filtered set of data at runtime
  3. When we need to automate the export process by specifying batch size on the fly

In order to demonstrate the solution I have setup the environment as below:

  1. Create a Large List with 10,000 Items
  2. Create a new WebPart Page with Content Editor WebPart added to it with a text file referenced from Site Assets Library. This page will take care of Presentation part of the solution, which is quite simple in this case.
  3. Add a Text File to Site Assets Library to which we will add the necessary code to achieve this solution.

Once the list is created and items are added to it, it will look like as below.

1

Please note that I intentionally keep the number of items more than the list default threshold limits (5000 items) in order to make sure that this solution remains workable even if the list threshold has been crossed.

Also notice that we only have one View (All Items) which is created already OOB as shown below:

2

Now first of all we need to analyze the super simple UI that we have created using WebPart Page.

This UI has got one Button “Create & Export List View”.

3

Now let’s look for the HTML that builds this super simple UI.

  1. “viewTitle” Div: Will display the title that we want to show while rendering the List View on this Page.
  2. “printView” Div: Will display the List View which will render as HTML
  3. “Create & Export List View” Button: Will Trigger the following actions:
    1. Create a new View in SharePoint List
    2. Export View Data to Excel Sheet
    3. Render View as HTML on the Custom UI

4

Next thing is to talk about the core plumbing that will do all the magic.

We need to add the JQuery reference to our JS File as shown below:

5

In our JS Code we will be having two functions:

CreateViews: This function will create a new view based on the filter query and row limits.

Let’s analyze the code for this function to understand the different sections as shown below:

  1. Request Headers: Specify the needed Request Headers for the REST Call.
  2. View Name: Generate Dynamic View Name
  3. URL: Prepare URL for REST Call to query the List Views
  4. REST Call Body To Created New View in SharePoint List:
    1. Specify Metadata used to create the SPView
    2. Specify View Name
    3. Specify if this is Personal View
    4. Specify the ViewQuery to get the filtered records in the View
    5. Specify the RowLimit to specify the maximum number of records that a view can hold. (I have specified 10,000)
    6. Call Success: Execute another method to Export and Render the List as HTML

6

We got another function “getViewAsHTML“ which will be executed if the above call gets successful.

In this function we got two noteworthy things:

  1. Calling renderAsHTML() Function : This function will take care the rendering of List Views as HTML.
  2. Export URL: This is a bit of tricky to get prepared the Export URL but fortunately with a very simple solution as explained below-

7

The simple solution to this tricky problem is to issue a dummy request to SharePoint to export the list data using following steps:

  1. Launch Fiddler
  2. Navigate to SharePoint List
  3. Click on Export to Excel Ribbon Command Button
  4. Analyze the request intercepted by Fiddler

8

9

Copy the URL issued to SharePoint by Command Button and analyze it.

10

On analysis we found that we can build this URL dynamically by providing List ID and View ID at runtime and that’s exactly what we are doing in the step 2 of getViewAsHTML() function above:

11

And that’s it.

We are all done with the functionality, now it is the time to test all our hard work.

Now launch the WebPart Page with our UI present on it.

Click “Create & Export List View” button

12

And sure enough once the execution is completed we will see three actions occur as follows:

  1. A new View has been created in the list as shown below:

13

  1. List View gets Render as HTML in “printView” Div as we discussed above
  2. Export Query File is ready to be downloaded

14

Save the Query File

15

Open the Query File and Enable the Data Connection with SharePoint

16

And we will get all items hold by the view based on the query specified, in this current View I took all 10,000 items as shown below:

17

18

Though this is quite a simple approach which can be used to easily cater the custom data export requirements in SharePoint.

Hope this will help someone in need…

SharePoint 2013 : How to develop Custom Search Apps Using REST API

In this article we are going to deal with Search Scenarios using enhanced REST API in SharePoint 2013. To make the story more interesting I am going to make use of new App Model offered by SharePoint 2013. Here we will be going to explore SharePoint Hosted App which is one of the three types of Apps offered under SharePoint 2013 App Model. In this article we will cover a simple scenario where we will create a Custom Search (SharePoint Hosted) App based on SharePoint 2013 Search and REST API provided by SharePoint 2013, that will perform search on Crawl Data aggregated by the Search Crawler.

Prerequisites

  1. App Development Environment must be configured properly.
  2. SharePoint 2013 Search Service Application must be created and configured.
  3. Sample Data Sources must be created with in the Scope targeted by the crawler.
  4. Full Crawl must be finished.
  5. Fiddler Web Proxy should be installed. You can get Fiddler from http://www.telerik.com/download/fiddler

Following are the steps involved in the development of Custom Search App using REST API:

Step-1 : Start Visual Studio 2012/2013 and Select “App for SharePoint 2013” project template as shown below

1

Step-2 : Modify the Default.aspx Page and add the UI Elements for Search App. For the sake of simplicity I am using basic HTML for the presentation but we can design the UI as compelling as it needs to be by making use of enhanced UI support from HTML5 & CSS3.

2

Step-3: Next thing is to hook up the “Click” event of the button “btnExecuteSearchREST” with an event handler function as shown below:

3

Step-4:   Now lets’ do the step wise analysis of the actual plumbing

6

  1. Get the Search Term entered by the user and stored it in the variable “queryText”.
  2. Define Request Headers for the Web Request, Please Note that we mention “odata=verbose” as a part of the value for “Accept” Property, this is a new convention for Web Requests using REST API in SharePoint 2013 and it must be followed else it would produce the following error :

4

  1. Set “url” property to the URL of the site which is hosting this App using “_spPageContextInfo.webAbsoluteUrl”, you must have to memorize this object as you have to use it on every single page in SharePoint where you want to use the SharePoint Context. This simple object provides you a lot of useful information to work on as shown below

5

  1. Specify the Search Query using new End Point (_api/search/query/<searchTerm>) provided by SharePoint 2013.
  2. Read the results from the response object and save it to “rows” variable, based on the data structure returned by the response object. We can identify the data structure by analyzing the response object using Fiddler Web Proxy as shown after few steps down the line.
  3. We need to loop through the result set to read the Values based on the Keys returned.
  4. Lastly we need to display the records as required.

Optional: At this point you can make use of some of the JavaScript Frameworks like “Knockout.js” which is based on MVVM Pattern and has an excellent feature of dependency tracking, which allows you to write complex UI logic with much Lesser & Cleaner Code. You can learn Knockout very easily by following the link http://learn.knockoutjs.com/.

Step-5: With this we are all done with the code, now it’s time to build & deploy the App and see it in action. In order to test the Custom Search App, first verify the Data Source which contains the data related to our search term. In our case it is “Customers” List with a Column “ContactTitle” containing the Search Term “Owner” as shown below

7

Now specify the search term in textbox provided as a part of App UI and click “Execute Search using REST API” as shown below:

8

Before going to the search result in the “Result Panel” (which is a div on App UI), we should spend a few minutes to analyze the response object to determine what exactly is returning from the server against our request. If we view the response object in Fiddler as shown below, we will see a complete JSON Object containing the data in terms of Key/Value Pairs. This information is very important as this will help us to navigate the result set that we are interested in. Please Note that we utilize this information under Step-4 Point 5, where we are saving the result set from response object to local variable.

9

10

Finally if we see to the Result Panel we should be able to get the results based on the search term “Owner”.

11

This simple demo helps you to grasp the notion of how can we employ REST API while working with Custom Search Apps/Solutions.

Hope this will help someone in need…