Sunday, August 24, 2014

Using the HttpClient Class with SharePoint 2013 REST API

Technorati Tags: ,

The System.Net.Http.HttpClient class is new in .Net framework 4.5 and was introduced under the ASP.Net Web API. The class and has many methods that support asynchronous programming and  is the best choice for writing client apps that make HTTP requests. Compared to the traditional System.Net.HttpWebRequest class not only does the HttpClient class have more options it also has extension methods such as PostAsJsonAsync and PostAsXmlAsync  methods which are available in System.Net.Http.Formatting assembly shipped with ASP.Net MVC 4 framework. In this post I am going to give you a tip on how to successfully post to the SharePoint REST API using the HttpClient class. There are many examples on how to post to the SharePoint REST API using the HttpWebRequest class but hardly any using the HttpClient class. If you are not careful you would think that the HttpClient class did not work with the SharePoint REST API.

I Keep Getting a 400 Bad Request Message when Posting

Below is a small example on how to use the HttpClient to create a folder in SharePoint 2013. The key to success is setting the Content-Type header correctly when posting. If you have used the REST API you know you must set the Content-Type header to “application/json; odata=verbose”. If you don’t you will get a “400 Bad Request” error. You can use the HttpClient.DefaultRequestHeaders collection to add headers but when trying to add the “Content-Type” header the collection will throw an “InvalidOperationException” with this message “{"Misused header name. Make sure request headers are used with HttpRequestMessage, response headers with HttpResponseMessage, and content headers with HttpContent objects."}”. So ok I must not be setting the content-type correctly on the HttpContent object. The StringContent class is what you are supposed to use as an argument when calling the HttpClient.PostAsync method. Looking at the StringContent class your first inclination is to use the constructor and give it the json that you want to post. The constructor takes the json, encoding type and media type as arguments. The media type corresponds to the content-type.

StringContent strContent = new StringContent(json, System.Text.Encoding.UTF8, "application/json;odata=verbose");

Unfortunately sending “application/json;odata=verbose” as the media type argument causes a “FormatException” with the message {"The format of value 'application/json;odata=verbose' is invalid."}. If you just use “application/json” you will receive a “400 bad request” error because the “odata=verbose” is missing.  So how do you get around this. First of all you must create the StringContent object with the json as the only argument to the constructor and then set the StringContent.Headers.ContentType property to “application/json;odata=verbose” using the MediaTypeHeaderValue.Parse method.

StringContent strContent = new StringContent(json);               
strContent.Headers.ContentType = MediaTypeHeaderValue.Parse("application/json;odata=verbose");

Mystery solved.

private void CreateFolder(HttpClient client, string digest)
{
client.DefaultRequestHeaders.Clear();
client.DefaultRequestHeaders.Add("Accept", "application/json;odata=verbose");
client.DefaultRequestHeaders.Add("X-RequestDigest", digest);
client.DefaultRequestHeaders.Add("X-HTTP-Method", "POST");

string json = "{'__metadata': { 'type': 'SP.Folder' }, 'ServerRelativeUrl': '/shared documents/folderhttp1'}";
client.DefaultRequestHeaders.Add("ContentLength", json.Length.ToString());
try
{
StringContent strContent = new StringContent(json);
strContent.Headers.ContentType = MediaTypeHeaderValue.Parse("application/json;odata=verbose");
HttpResponseMessage response = client.PostAsync("_api/web/folders", strContent).Result;

response.EnsureSuccessStatusCode();
if (response.IsSuccessStatusCode)
{
var content = response.Content.ReadAsStringAsync();
}
else
{
var content = response.Content.ReadAsStringAsync();
}
}
catch (Exception ex)
{
System.Diagnostics.Debug.WriteLine(ex.Message);
}

}

HttpClient is Here


HttpClient is a modern HTTP client for .NET. It provides a flexible and extensible API for accessing all things exposed through HTTP. You should use it instead of the HttpWebRequest. You can read more about it here System.Net.Http. Another great source of information when using the HttpClient with SharePoint REST is Dennis RedField’s blog Cloud 2013 or Bust. This blog has an in depth 4 part series on how to use the HttpClient with SharePoint REST. Changing our habits as developers can be a slow process. However, some new APIs can be confusing especially when used against SharePoint 2013. SharePoint 2013 is not fully OData compliant yet and has some quirks, namely content-type checking.  I hope this tip can save you some time.

Thursday, July 31, 2014

Uploading Large Documents into SharePoint Online with REST,CSOM, and RPC using C#

Technorati Tags: ,,

There are many articles that give great examples on how to upload documents to SharePoint Online using jQuery and REST. These are useful to get around the message size limitation of use CSOM/JSOM when uploading documents. This message size limitation is not configurable in SharePoint Online. There are few examples on how to upload large documents using C#. In this blog post I will show you how to use C# and the SharePoint REST, Managed CSOM and RPC to upload large documents (up to 2GB) to SharePoint Online. There are a few things you need to take care of to get all these  to work with SharePoint Online.

Credentials and Cookie Containers

In the code examples below both REST and RPC use the HttpWebRequest class to communicate with SharePoint. When using this class from C# you must set the Credentials and the CookieContainer properties of the HttpWebRequest object. The following helper methods creates the Microsoft.SharePoint.Client.SharePointOnlineCredentials and gets the System.Net.CookieContainer for the SharePointOnlineCredentials.

public static class Utils
{

public static CookieContainer GetO365CookieContainer(SharePointOnlineCredentials credentials, string targetSiteUrl)
{

Uri targetSite = new Uri(targetSiteUrl);
string cookieString = credentials.GetAuthenticationCookie(targetSite);
CookieContainer container = new CookieContainer();
string trimmedCookie = cookieString.TrimStart("SPOIDCRL=".ToCharArray());
container.Add(new Cookie("FedAuth", trimmedCookie, string.Empty, targetSite.Authority));
return container;


}

public static SharePointOnlineCredentials GetO365Credentials(string userName, string passWord)
{
SecureString securePassWord = new SecureString();
foreach (char c in passWord.ToCharArray()) securePassWord.AppendChar(c);
SharePointOnlineCredentials credentials = new SharePointOnlineCredentials(userName, securePassWord);
return credentials;
}



}

Uploading Large Documents With REST


The following code takes the site URL, document library title, and a file path to a local file and adds the file to the root folder collection of the site. If you want to use folders you can modify this code to handle it. The REST call requires a form digest value to be set so I have included the code that makes a REST call to the contextinfo to get it. Please make sure to set the time out on the HttpWebRequest to about 10 minutes because large files will exceed the default time out of 100 seconds. 10 minutes should be adequate to cover the unpredictable upload speeds of ISP’s and SharePoint Online.

public static void UploadRest(string siteUrl, string libraryName, string filePath)
{
byte[] binary = IO.File.ReadAllBytes(filePath); ;
string fname = IO.Path.GetFileName(filePath);
string result = string.Empty;
string resourceUrl = siteUrl + "/_api/web/lists/getbytitle('" + libraryName + "')/rootfolder/files/add(url='" + fname + "',overwrite=true)";

HttpWebRequest wreq = HttpWebRequest.Create(resourceUrl) as HttpWebRequest;
wreq.UseDefaultCredentials = false;
SharePointOnlineCredentials credentials = Utils.GetO365Credentials("your login", "your password");
wreq.Credentials = credentials;
wreq.CookieContainer = Utils.GetO365CookieContainer(credentials, siteUrl);

string formDigest = GetFormDigest(siteUrl, credentials, wreq.CookieContainer);
wreq.Headers.Add("X-RequestDigest", formDigest);
wreq.Method = "POST";
wreq.Timeout = 1000000;
wreq.Accept = "application/json; odata=verbose";
wreq.ContentLength = binary.Length;


using (IO.Stream requestStream = wreq.GetRequestStream())
{
requestStream.Write(binary, 0, binary.Length);
}

WebResponse wresp = wreq.GetResponse();
using (IO.StreamReader sr = new IO.StreamReader(wresp.GetResponseStream()))
{
result = sr.ReadToEnd();
}


}
public static string GetFormDigest(string siteUrl, ICredentials credentials, CookieContainer cc)
{
string formDigest = null;

string resourceUrl = siteUrl +"/_api/contextinfo";
HttpWebRequest wreq = HttpWebRequest.Create(resourceUrl) as HttpWebRequest;

wreq.Credentials = credentials;
wreq.CookieContainer = cc;
wreq.Method = "POST";
wreq.Accept = "application/json;odata=verbose";
wreq.ContentLength = 0;
wreq.ContentType = "application/json";
string result;
WebResponse wresp = wreq.GetResponse();

using (IO.StreamReader sr = new IO.StreamReader(wresp.GetResponseStream()))
{
result = sr.ReadToEnd();
}

var jss = new JavaScriptSerializer();
var val = jss.Deserialize>(result);
var d = val["d"] as Dictionary;
var wi = d["GetContextWebInformation"] as Dictionary;
formDigest = wi["FormDigestValue"].ToString();

return formDigest;

}

Uploading Large Documents with CSOM


At one time I thought you could not do this with CSOM, however fellow MVP Joris Poelmans brought to my attention that the AMS sample Core.LargeFileUpload was able to upload over 3 mb files O365 Development Patterns and Practices. This can only be done if you are setting the FileCreationInfo ContentStream property with an open stream to the file. This gets around the message size limit of CSOM because the ContentStream is using the MTOM optimizations and sending the raw binary rather than a base64 encoded binary. This is much more efficient and is faster that the other methods. This appears to be a later change in CSOM and optimized for SharePoint Online. The CSOM code does not need a cookie container. I also tried using File.SaveBinaryDirect method but I received “Cannot Invoke HTTP Dav Request” since this is not supported in SharePoint Online.

 public static void UploadDocumentContentStream(string siteUrl, string libraryName, string filePath)
{
ClientContext ctx = new ClientContext(siteUrl);
ctx.RequestTimeout = 1000000;
ctx.Credentials = Utils.GetO365Credentials("your login", "your password");
Web web = ctx.Web;

using (IO.FileStream fs = new IO.FileStream(filePath, IO.FileMode.Open))
{
FileCreationInformation flciNewFile = new FileCreationInformation();

// This is the key difference for the first case - using ContentStream property
flciNewFile.ContentStream = fs;
flciNewFile.Url = IO.Path.GetFileName(filePath);
flciNewFile.Overwrite = true;


List docs = web.Lists.GetByTitle(libraryName);
Microsoft.SharePoint.Client.File uploadFile = docs.RootFolder.Files.Add(flciNewFile);

ctx.Load(uploadFile);
ctx.ExecuteQuery();
}
}

Uploading Large Documents with RPC


RPC still lives and is supported in SharePoint Online. The code below is simplified. RPC can be hard to understand because the syntax for the different parameters is from years ago. RPC is basically an HTTP POST to C++ dll. It can be fast but it was not faster than CSOM.  The parameters and binary must be combined and separated by a line feed into one common byte array before posting. The libraryName parameter cannot be the title of document library but the actual URL for it. Instead of Documents you must use Shared Documents. You will note many of the parameters are URL Encoded because RPC is very particular about characters in the URL. Finally, note that the code feeds the byte array to the request stream in chunks. This helps prevent triggering of SharePoint Online throttling limits.

 public static void UploadDocumentRPC(string siteUrl, string libraryName, string filePath)
{
string method = HttpUtility.UrlEncode("put document:14.0.2.5420");
string serviceName = HttpUtility.UrlEncode(siteUrl);
string document = HttpUtility.UrlEncode(libraryName + "/" + IO.Path.GetFileName(filePath));
string metaInfo = string.Empty;
string putOption = "overwrite";
string keepCheckedOutOption = "false";
string putComment = string.Empty;
string result = string.Empty;

string fpRPCCallStr = "method={0}&service_name={1}&document=[document_name={2};meta_info=[{3}]]&put_option={4}&comment={5}&keep_checked_out={6}";
fpRPCCallStr = String.Format(fpRPCCallStr, method, serviceName, document, metaInfo, putOption, putComment, keepCheckedOutOption);

byte[] fpRPCCall = System.Text.Encoding.UTF8.GetBytes(fpRPCCallStr + "\n");
byte[] postData = IO.File.ReadAllBytes(filePath);
byte[] data;

if (postData != null && postData.Length > 0)
{
data = new byte[fpRPCCall.Length + postData.Length];
fpRPCCall.CopyTo(data, 0);
postData.CopyTo(data, fpRPCCall.Length);
}
else
{
data = new byte[fpRPCCall.Length];
fpRPCCall.CopyTo(data, 0);
}

HttpWebRequest wReq = WebRequest.Create(siteUrl + "/_vti_bin/_vti_aut/author.dll" ) as HttpWebRequest;
SharePointOnlineCredentials credentials = Utils.GetO365Credentials("your login", "your password");
wReq.Credentials = credentials;
wReq.CookieContainer = Utils.GetO365CookieContainer(credentials, siteUrl);
wReq.Method="POST";
wReq.Timeout = 1000000;
wReq.ContentType="application/x-vermeer-urlencoded";
wReq.Headers.Add("X-Vermeer-Content-Type", "application/x-vermeer-urlencoded");
wReq.ContentLength=data.Length;

using (IO.Stream requestStream = wReq.GetRequestStream())
{
int chunkSize = 2097152;
int tailSize;
int chunkNum = Math.DivRem(data.Length, chunkSize, out tailSize);

for (int i = 0; i < chunkNum; i++)
{
requestStream.Write(data, chunkSize * i, chunkSize);
}

if (tailSize > 0)
requestStream.Write(data, chunkSize * chunkNum, tailSize);

}

WebResponse wresp = wReq.GetResponse();
using (IO.StreamReader sr = new IO.StreamReader(wresp.GetResponseStream()))
{
result = sr.ReadToEnd();
}

}

Three Ways of Uploading Large Documents to SharePoint Online


All of the above code examples are good ways to upload large documents to SharePoint Online. All of them utilize the Client Object Model to create the credentials and cookie that is required for SharePoint Online. Getting the cookie is rather complicated without using the Client Object Model. All three methods require that you set the request timeout to a large value because uploading to SharePoint Online is much slower than SharePoint On-Premises. Experiment with the code samples. I tested these with 200mb files and the CSOM was the fastest but your results may vary. I like variety and having multiple ways of accomplishing a task.