Monday, November 30, 2009

How To: Page search results in SharePoint 2007

Technorati Tags: ,,,

Sometimes a little knowledge can go a long way in implementing a scalable search solution. Like in the case of not returning all the results from a SharePoint Microsoft.Office.Server.Search.Query.FullTextSQLQuery. If your search returns hundreds or even thousands of results you should consider paging the results. It is fairly easy to page search results in SharePoint. The key is to return the total number of pages available in the results when you first issue the query. Then based on that number you can pass in your page number and calculate the starting row for the search and set the StartRow property. The following code shows some simple code to get up and running on implementing paging search results. This example can also use the KeywordQuery object also.

This example uses Microsoft.Office.Server.Search and System.Data namespaces.

 

public static DataTable GetFTSQueryPagedSearch(Int32 pageNumber)
{

           DataTable results = new DataTable();
           int pageSize = 50;
           int startRow = (pageNumber -1) * 50;

           using (SPSite site = new SPSite("http://basesmcdev2/sites/tester1"))
           {
               FullTextSqlQuery fts = new FullTextSqlQuery(site);              
               fts.QueryText = "SELECT Title,FileExtension,ContentType,Created,
                                            LastModifiedTime,Path FROM SCOPE()
                                            WHERE (CONTAINS(Path,'\"/sites/tester1/Lists/* \"'))";

               fts.ResultTypes = ResultType.RelevantResults;
               fts.RowLimit = pageSize;
               fts.StartRow = startRow;

               ResultTableCollection rtc = fts.Execute();
               if (rtc.Count > 0)
               {

                   using (ResultTable relevantResults = rtc[ResultType.RelevantResults])
                   {
                       results.Load(relevantResults, LoadOption.OverwriteChanges);

                       int totalNumberOfPages = (int)Math.Round((double)relevantResults.TotalRows / pageSize);
                   }

               }

               return results;

           }

}

Tuesday, November 24, 2009

How To: Copy list items across site collections in SharePoint

Technorati Tags: ,,,

The other day I demonstrated some code on how to copy files from one document library to another in another site collection. However, the code does not work with lists and list items. Dealing with folders and items within folders is completely different with SharePoint lists. In fact, it seems needlessly complicated. The sample code below takes a source site collection url, destination site collection url, list name (must exist in both source and destination site collection, the string of the UniqueId guid of the starting folder, and the folder’s url.

The difficult part was getting the folder url to work when creating a new folder in a SharePoint list. You must use the SPListItemCollection.Add method passing in an empty string, SPFileSystemObjectTyp.Folder, and the folder url.  In order to make this work with sub folders you must construct a folder url in the form of “grandparent.folder.name +/ + parent.folder.name … + / + folder.name”. There is nothing on the SPFolder object that can give you this “container” path. So I had to do some substring(ing).

The other problem was getting at the child items of a folder in a SharePoint list. When using a document library the files collection of the SPFolder works great, unfortunately, this does not exist with SharePoint lists. So you have to make use of the SPQuery object to bring back the child items within the folder. Finally, you can use linq to filter out the list items from the folders. All in all it works great and will create an identical copy of the folder structure from one list to the other across site collections.

 

public static void CopyListFolderToAnotherSiteCollection(string sourceUrl, string destinationUrl,
          string listName, string uniqueFolderID, string newFolderUrl)
      {
          SPListItem newItem = null;

          using (SPSite sourceSite = new SPSite(sourceUrl))
          {
              using (SPWeb sourceWeb = sourceSite.OpenWeb())
              {

                  SPList sourceList = sourceWeb.Lists[listName];
                  SPFolder sourceFolder = sourceWeb.GetFolder(new Guid(uniqueFolderID));
                  SPQuery q = new SPQuery();
                  q.Folder = sourceFolder;
                  SPListItemCollection sourceItems = sourceList.GetItems(q);
                  using (SPSite destSite = new SPSite(destinationUrl))
                  {
                      using (SPWeb destWeb = destSite.OpenWeb())
                      {
                          SPList destList = destWeb.Lists[listName];

                          SPListItem destItem = destList.Items.
                              Add(string.Empty, SPFileSystemObjectType.Folder, newFolderUrl);

                          destItem.Update();
                          SPFolder destFolder = destWeb.GetFolder(destItem.UniqueId);

                          List<SPListItem> listItems = (from l in sourceItems.OfType<SPListItem>()
                                         where l[SPBuiltInFieldId.ContentType].ToString() != "Folder" select l).ToList();

                          List<SPListItem> folderItems = (from l in sourceItems.OfType<SPListItem>()
                                                        where l[SPBuiltInFieldId.ContentType].ToString() == "Folder"
                                                        select l).ToList();

                          foreach (SPListItem sourceItem in listItems)
                          {
                              newItem = destList.Items.Add(destFolder.ServerRelativeUrl,
                                                                                  SPFileSystemObjectType.File,null);
                              newItem["Title"] = sourceItem["Title"];
                              newItem[SPBuiltInFieldId.ContentType] = sourceItem[SPBuiltInFieldId.ContentType];
                              newItem.Update();
                          }

                          foreach (SPListItem f in folderItems)
                          {
                              SPFolder folder = f.Folder;
                              int index = folder.ServerRelativeUrl.IndexOf(sourceList.RootFolder.Url);
                              string folderUrl = folder.ServerRelativeUrl.Substring(index + sourceList.RootFolder.Url.Length + 1);
                              CopyListFolderToAnotherSiteCollection(sourceUrl, destinationUrl,
                                                           listName, folder.UniqueId.ToString(), folderUrl);
                          }

                      }
                  }
              }
          }
      }

Monday, November 23, 2009

How To: Copy files across site collections in SharePoint

Technorati Tags: ,,,

SharePoint has great built in methods to copy or move documents across sub-sites or lists within the same site collection. You can use SPFile.MoveTo or SPFilem.CopyTo. You can even move whole folders and their contents with the SPFolder.MoveTo method. Very nice. Unfortunately, these do not work across site collections. However you can use the object model and a little bit of recursion to copy folders across site collections. The following code takes the url string to the source site collection, url string to the destination site collection, the name of the document library, the url string for the folder and the folder name.

public static void CopyFolderToAnotherSiteCollection(string sourceUrl, string destinationUrl,
          string listName, string folderUrl, string folderName)
      {
          string listRelatvieFolderUrl = folderUrl.Substring(folderUrl.IndexOf("/")+1);
          using (SPSite sourceSite = new SPSite(sourceUrl))
          {
              using (SPWeb sourceWeb = sourceSite.OpenWeb())
              {
                  SPFolder sourceFolder = sourceWeb.GetFolder(folderUrl);                            
                  using(SPSite destSite = new SPSite(destinationUrl))
                  {
                      using(SPWeb destWeb = destSite.OpenWeb())
                      {
                          SPListItem destItem = destWeb.Lists[listName].Items.
                              Add(string.Empty, SPFileSystemObjectType.Folder, listRelatvieFolderUrl);

                          destItem.Update();

                          SPFolder destFolder = destWeb.GetFolder(folderUrl);

                          foreach (SPFile file in sourceFolder.Files)
                          {  
                              destFolder.Files.Add(file.Url, file.OpenBinary());                            
                          }

                          foreach (SPFolder folder in sourceFolder.SubFolders)
                          {
                              CopyFolderToAnotherSiteCollection(sourceUrl, destinationUrl, listName,
                                  folder.Url, folder.Name);
                          }

                      }
                  }
              }
          }
      }

The SPFolder.Files.Add has many overloaded methods. You can decrease the amount of memory by using a stream object instead of the byte array returned by the SPFile.OpenBinary. You can also add metadata by populating a hashtable and sending that.

Friday, November 20, 2009

How To: Make an ECB menu item for a folder

Technorati Tags:

There seems to be a lot of information on how to create a “Edit Control Block” for a list item in SharePoint. An ECB is a menu item that shows up when you click on an item in a SharePoint list or document library. You can generally make these by defining a feature and a custom action xml file. In the custom action xml file you set certain attributes of the “CustomAction” element. The most important of which is the UrlAction attribute which can be a url which the user will be redirected to when clicking on the item.

http://msdn.microsoft.com/en-us/library/ms460194.aspx

The location attribute determines where the menu item will be displayed. Setting the location to “Edit Control Block” tells SharePoint to display the menu item in the context menu for the list item. In a document library you can further refine when the menu item is displayed by setting the RegistrationType to “FileType” and the RegistrationId to a file extension like “tif”. Then the menu item will only show up for tif files.

<CustomAction RegistrationType="FileType" RegistrationId="tif" Location="EditControlBlock" Sequence="106" Title="View" ImageUrl="/_layouts/images/view.gif">
  <UrlAction Url="_layouts/settings.aspx"/>
</CustomAction>


So how do I make this menu item just for folders in a list or library. Well you use the RegistrationType set to “ContentType” and the RegistrationId set to “0x0120” which is the contenttype id for a folder in SharePoint.



<CustomAction RegistrationType="ContentType" RegistrationId="0x0120" Location="EditControlBlock" Sequence="106" Title="View" ImageUrl="/_layouts/images/view.gif">
  <UrlAction Url="_layouts/settings.aspx"/>

</CustomAction>



Finally, you can actually call javascript from your UrlAction Url attribute. The following shows how to construct a querystring detecting whether to use https or http for the querystring.


<CustomAction RegistrationType="FileType" RegistrationId="tif" Location="EditControlBlock" Sequence="106" Title="View" ImageUrl="/_layouts/images/view.gif">
  <UrlAction Url="javascript: var isHttps = window.location.href.indexOf('https') == 0; var i = (isHttps) ? 8 : 7; var itemUrlToken = '{ItemUrl}'; var siteUrlwoHttp = window.location.href.substring(i) ; var index = siteUrlwoHttp.indexOf('/'); var itemUrl = ((isHttps) ? 'https://' : 'http://') + siteUrlwoHttp.substring(0, index) + itemUrlToken; window.open('/_layouts/custompage.aspx?doc=' + itemUrl);" "/>
</CustomAction>


 

Wednesday, October 28, 2009

The state of Search (SharePoint 2010)

Technorati Tags:

I recently attended the SharePoint conference in Las Vegas and the big emphasis was on Search and "findability". Microsoft has made search a key piece to SharePoint's enterprise content management capabilities. Getting content into SharePoint has been easy, but now customers are demanding better solutions on how to find and leverage the content once it is in SharePoint. Customers want to be able to browse, filter and make decisions on search results. SharePoint 2007 search lacked any useful way to refine and browse results. The only thing you had was the ability to page through results looking at the highlighted properties, title and url of the document. SharePoint 2010 search has made some progress versus 2007.  Below I will discuss the changes made in search results that enhance “findability”, improvements for content acquisition for search, and much needed bug fixes that should have been there years ago. All this is based on the July 2009 technical preview and material from the SharePoint Conference 2009.

 

Improvements in standard search center

The standard search center in 2010 has added a “Preferences” link for user preferences when searching.  There are two preferences. First there is enabling “Search Suggestions” very similar to other search engines when type in a keyword it will auto-populate suggestions of terms other users have used. This has to be enabled by the site collection administrator.

The second preference is which languages you want to search in. This will give you the ability to select which languages you want given higher relevance and how the search engine will interpret the keywords.

 

Improvements in advanced search

The advanced search web part has not changed at all in 2010. You still have the ability to search on keywords and use property restrictions. The improvements come with the plethora of new search results web parts including “Top Federated Results” , “Related Queries” and “ Search Refinements” that are included on the results.aspx page. The “Top Federated Results” web part allows you to add multiple search results from other sources such as bing, google, people, or even an external FAST search server.  You can set up your own federated search locations via the “Manage Federated Locations” page in Central Administration\Search Service Application page.

 

The “Related Queries” web part seems to be used for connecting your own federated search page, so users can take SharePoint results and view them together with a results from custom source.

Finally, there is the “Search Refinements” web part which allows users to navigate results by clicking on links that resubmit a search query using criteria to narrow the results down.  Out of the box the web part includes refinements for “Result Type” which is the file extension of the document that is returned. Author and “Modified Date” are also included. These refinements can be added to and modified via the “Filter Category Definition” builder.    The “Filter Category Definition” is another xml document that is similar to the xml that you define for the “Property Restrictions” in the Advanced Search web part. Here you can define your own refinement names and use the either values or ranges of values. For instance, if you have a date and time managed property you could define your own periods like “First Quarter” and  “Second Quarter”…. In the xml there is a way to tell the web part to use relative values to compute the ranges. I will explain more about that in a later post. The refinements can also use the new  “Managed Metadata Columns” which take advantage of the new taxonomy service for custom terms.

This web part increases the “findability” of SharePoint 2010 search. The FAST SharePoint Search will also include counts of the refinements, thumbnail previews of the documents and custom relevancy sorting making the SharePoint search even more usable.

 

 

Search Optimizations

SharePoint 2010 offers many new search result web parts and they all can be put on the same web part page. This could be a performance problem rendering the page when executing multiple queries to multiple sources. Fortunately, SharePoint 2010 has optimize page rendering by making all of these web parts Ajax enabled providing fast rendering by allowing the queries to be executed asynchronously. You will notice each of the results web parts there is an “Ajax Options” section in the tool pane. You can enable “Asynchronous Update” which by default it is not. You can also have your queries refresh them selves based on a interval or display a manual refresh button to keep the network traffic down.

Other optimizations involve crawling. SharePoint 2007 had problems crawling large amounts of data. It could take weeks to do a full crawl of a million documents.  With SharePoint 2010 new optimizations have been included.  First there is “security only” crawling. Periodically SharePoint 2010 will do a security only crawl which involves only crawling the acl (permission changes) without bringing down the content to the index servers. Secondly, the crawler can crawl any “Business Data Connector” source and will inline cache the data for better performance on searching.  Unfortunately, there is still no ability to inject a document into the index. This would be useful if you are doing workflow and add a document. The users will still have to wait until the next incremental crawl. If you need this capability you will have to purchase FAST SharePoint Search. The ability to use multiple index servers and

Finally, there is a new option to decrease the amount of database space that is used to store managed property values. This option is available within the Managed Property edit page in the Search Service Application. It allows for storing text based managed property values as binary hashes.

 

Query Language Improvements

The search box now supports more robust keyword searching. In SharePoint 2007 if you entered two or more keywords into the search text box they were implicitly OR together. In SharePoint  2010 you can now tell search what you want to  do with the keywords. For example, this would find all document containing both the term CAT and DOG:

cat AND dog

This would find all documents containing either cat or dog:

cat OR dog

In the July technical preview the and/or must be all caps for it to work.

SharePoint 2010 also now supports wild card searching from the search box. So if you wanted to find content that contained both computer or computation, you can enter comp*.

Bug Fixes

One of the biggest bugs in SharePoint 2007 FullTextSQL queries was the inability to OR together two different managed properties and return correct results.

Custom cross list search pitfalls.

The crux of the bug is that if there is not a value for both OR managed properties the document will not come back in the results. So if you have managed properties like col1 and col2 and the document only has a value for either col1 or col2 then it will not be returned. SharePoint 2010 has fixed this to a certain extent.  The July technical preview shows it only working with text based managed properties. If you try to OR a text managed property with a decimal managed property is still will not work. Hopefully beta 2.

Another issue was the “Query malformed error” you would receive when the FullTextSQL would OR together more than 10 values on the same managed property. There is a hard limit of 10 in SharePoint 2007. This is fixed in 2010.

Summary

In summary, the state of search in SharePoint 2010 is great. Things are looking up. With the new search web parts, optimizations, query language enhancements, extended configuration and bug fixes, it looks like SharePoint 2010 will become a key component of a enterprise caliber ECM solution. I will post soon how to set up custom search refinements in the “Search Refinements” web part. So stay tuned.

Tuesday, September 8, 2009

SharePoint Server MVP

Technorati Tags: ,,

I found out yesterday I was awarded MVP for SharePoint Server. Its is great to be rewarded for something you love to do. You cannot keep what you cannot give away. Giving back to the community is an easy thing to do, especially in the MSDN forums. SharePoint is such a product that if you know how something works then you must share it with others, it makes SharePoint better. I work for KnowledgeLake http://www.knowledgelake.com  and we offer a great ECM document imaging product. I want to thank KnowledgeLake for giving me the opportunity to develop my knowledge of SharePoint over the last 5 years.

Downloading Content From SharePoint (Let me count the ways)

Technorati Tags: ,,,

This is a follow up post to the “Uploading Content To SharePoint”. In that post I evaluated the performance and complexity of uploading content to SharePoint with three different methods. In this post I will be doing comparisons for downloading content from SharePoint. Once again there are many ways to get content from SharePoint. When I say content I am focusing on files and metadata. There is an additional way to get a list item using the Lists web service and the GetListItems method; however, this only returns metadata.

Method Complexity Scalability Metadata Versions
Copy.asmx 2 4 yes no
WebDav 5 5 yes* yes**
Rpc 10 10 yes yes

*must be used in conjunction with Lists web service

**must be used in conjunction with Versions web service

Copy Web Service

The copy web service is the easiest to use to get content from SharePoint because it is familiar web service programming and it allows the content and the metadata to be retrieved in one method call. The scalability is not as good as the other methods because it uses the more verbose soap protocol. Finally, one of the biggest disadvantages using the copy web service is the fact that it does not support returning versions of a document. The copy web service was made for migrating content between document libraries and sites within the same site collection not necessarily versions. I did try using the copy web service to retrieve an older version of a document by setting the url argument to the GetItem method using the older version’s url.

The version url looks like this:

http://servername/site/_vti_history/512/documentlibrary/filename.tiff

Unfortunately, when sending in a version url the GetItem method executes successfully but both the metadata and binary are empty.

public static void DownloadDocument()
{

copyservice.Copy c = new copyservice.Copy();
c.Url = "http://basesmcdev2/sites/tester1/_vti_bin/copy.asmx%22;
c.UseDefaultCredentials = true;

byte[] myBinary = new byte[] { };
copyservice.FieldInformation information = new copyservice.FieldInformation();
copyservice.FieldInformation[] info = { information };

string docUrl = "http://basesmcdev2/sites/tester1/Shared%20Documents/+aaa/mama.bmp%22;

uint result = c.GetItem(docUrl, out info, out myBinary);
if(File.Exists("c:\\newfile.bmp")) File.Delete("c:\\newfile.bmp");
using (Stream s = File.Create("c:\\newfile.bmp"))
{
s.Write(myBinary, 0, myBinary.Length);
s.Flush();
}
}

WebDav

Using webdav to download a document is fairly simple if you have the full url of the document. However, if you want to include metadata you must call the GetListItem of the list web service. This becomes tricky when the only piece of information you have is the url. You can use the list web service but you will need to know the name of the list and the id of the item in order to retrieve the metadata. In the end, if you are going to call a SharePoint web service then just call the copy web service and get both the metadata and file in one call.

public static void GetFileWithMetaData(string url)
{

WebClient wc = new WebClient();
wc.UseDefaultCredentials = true;

byte[] response = wc.DownloadData(url);

string returnStr = Encoding.UTF8.GetString(response);

if (File.Exists("c:\\default.txt")) File.Delete("c:\\default.txt");

using (Stream s = File.Create("c:\\detault.txt"))
{
s.Write(response, 0, response.Length);
s.Flush();
}

}

So how does WebDav support versions? You can give the DownLoadData method the url of the version mentioned previously and it will return the binary for that version. So, now the question is how to you obtain the url’s of previous versions? Below is a code snippet that uses the Versions web service to retrieve a particular version’s url for a document for a given file url and version number.

public static string GetVersionUrl(string url, double version)
{
double versionNumeric = 0;
string versionNumber = string.Empty;
string versionUrl = string.Empty;

versionservice.Versions vs = new versionservice.Versions();
vs.Url = "http://basesmcdev2/sites/tester1/_vti_bin/versions.asmx%22;
vs.UseDefaultCredentials = true;

XmlNode versionsNode = vs.GetVersions(url);

if (versionsNode != null)
{
using (StringReader sr = new StringReader(versionsNode.OuterXml))
{
using (XmlTextReader xtr = new XmlTextReader(sr))
{
XElement versionInfo = XElement.Load(xtr);

var versionResults = from r in versionInfo.Elements() where r.Name.LocalName == "result" select r;

foreach (XElement versionElement in versionResults)
{

var versionAttr = (from a in versionElement.Attributes() where a.Name == "version" select a);
versionNumber = versionAttr.First().Value;
versionUrl = versionElement.Attributes("url").First().Value.ToString();

//current version is in the form of "@1.0" and will not pass TryParse
if (versionAttr.Count() > 0 && double.TryParse(versionNumber, out versionNumeric))
if (version == versionNumeric) return versionUrl;
else
if (version == Convert.ToDouble(versionNumber.Substring(1))) return versionUrl;

}

}

}
}

return string.Empty;
}

FrontPage RPC (Remote Procedure Calls)

Most developers are not familiar with frontpage remote procedure calls. However, they are the most efficient and yet the most complex to code against. You could create your own wrapper classes to simplify coding. You must understand the command structure and be able to parse the return html correctly. In the case of downloading a document it is especially complex. You must come up with a way to parse out the returned metadata within the html. Below is an example of a successful return of a “get document” rpc call. Anything below the closing </html> tag is the file.

<html><head><title>vermeer RPC packet</title></head>
<body>
<p>method=get document:12.0.0.4518
<p>message=successfully retrieved document 'tester4/Code128.tif' from 'tester4/Code128.tif'
<p>document=
<ul>
<li>document_name=tester4/Code128.tif
<li>meta_info=
<ul>
<li>vti_rtag
<li>SWrt:FED298A3-6030-40DA-A984-D0A04A673741@00000000013
<li>vti_etag
<li>SW&#34;&#123;FED298A3-6030-40DA-A984-D0A04A673741&#125;,13&#34;
<li>vti_parserversion
<li>SR12.0.0.6318
<li>vti_modifiedby
<li>SRBASESMCDEV2&#92;steve.curran
<li>vti_filesize
<li>IR3387
<li>vti_timecreated
<li>TR22 Oct 2008 19:45:25 -0000
<li>ContentType
<li>SWDocument
<li>ContentTypeId
<li>SW0x01010087C42E0D80709D4CB61D6558C94571E4
<li>vti_title
<li>SW
<li>statechoice
<li>SWstateone
<li>vti_lastheight
<li>IX2200
<li>vti_timelastmodified
<li>TR13 Apr 2009 22:04:31 -0000
<li>vti_nexttolasttimemodified
<li>TR13 Apr 2009 22:02:20 -0000
<li>vti_candeleteversion
<li>BRtrue
<li>vti_canmaybeedit
<li>BXtrue
<li>vti_backlinkinfo
<li>VXLists/threeStateTasks/3_.000 Lists/threeStateTasks/4_.000
<li>myreq
<li>SWnbnbvnbv
<li>vti_author
<li>SRBASESMCDEV2&#92;steve.curran
<li>vti_lastwidth
<li>IX1696
<li>vti_sourcecontrolversion
<li>SRV1.0
<li>vti_sourcecontrolcookie
<li>SRfp_internal
<li>vti_level
<li>IR1
</ul>
</ul>
</body>
</html>








Finally, below is an example on how to call the “get document” rpc method. You must work with the returned byte array and copy the section which represents the file to another byte array. This method supports retrieving older versions. Just pass in the version number, zero would represent the current version. The fileUrl argument represents document library name plus the file name. For example, “Shared Documents/FileName.tiff”.










public static void DownloadDocumentRPC(string fileUrl, int version)
{

string method = "get document: 12.0.0.4518";
string serviceName = "http://basesmcdev2/sites/tester1/_vti_bin/_vti_aut/author.dll";
string verstr = version > 0 ? "V" + version.ToString() : string.Empty;
string document = fileUrl;
byte[] data;
string returnStr = string.Empty;
byte[] fileBytes = null;

string fpRPCCallStr = "method={0}&service_name={1}&document_name={2}&doc_version={3}&get_option={4}&timeout=0";

method = HttpUtility.UrlEncode(method);
fpRPCCallStr = String.Format(fpRPCCallStr, method, serviceName, document, verstr, "none");

try
{
//add line feed character to delimit end of command
byte[] fpRPCCall = System.Text.Encoding.UTF8.GetBytes(fpRPCCallStr + "\n");

data = new byte[fpRPCCall.Length];
fpRPCCall.CopyTo(data, 0);

HttpWebRequest wReq = WebRequest.Create(serviceName) as HttpWebRequest;

wReq.Credentials = System.Net.CredentialCache.DefaultCredentials;
wReq.Method = "POST";
wReq.ContentType = "application/x-vermeer-urlencoded";
wReq.Headers.Add("X-Vermeer-Content-Type", "application/x-vermeer-urlencoded");
wReq.ContentLength = fpRPCCall.Length;

using (Stream requestStream = wReq.GetRequestStream())
{

requestStream.Write(fpRPCCall, 0, fpRPCCall.Length);

int chunkSize = 2097152;


//Now get the response from the server
WebResponse response = wReq.GetResponse();
int lastBytesRead, totalBytesRead;
long contentLength = response.ContentLength;
bool noLength = false;

if (contentLength == -1)
{
noLength = true;
contentLength = chunkSize;
}

byte[] returnBuffer = new byte[(int)contentLength];
using (Stream responseStream = response.GetResponseStream())
{
totalBytesRead = 0;

do
{
lastBytesRead =
responseStream.Read(returnBuffer, totalBytesRead, ((int)contentLength) - totalBytesRead);
totalBytesRead += lastBytesRead;
if (noLength && (totalBytesRead == contentLength))
{
contentLength += chunkSize;
byte[] buffer2 = new byte[(int)contentLength];
Buffer.BlockCopy(returnBuffer, 0, buffer2, 0, totalBytesRead);
returnBuffer = buffer2;
}
}
while (lastBytesRead != 0);

}

if (noLength)
{
byte[] buffer3 = new byte[totalBytesRead];
Buffer.BlockCopy(returnBuffer, 0, buffer3, 0, totalBytesRead);
returnBuffer = buffer3;
}

returnStr = Encoding.UTF8.GetString(returnBuffer);

//get begining of file bytes
int startpos = returnStr.IndexOf("</html>") + 8;
using (MemoryStream ms =
new MemoryStream(returnBuffer, startpos, returnBuffer.Length - startpos))
fileBytes = ms.ToArray();


if (File.Exists("c:\\newfile.bmp")) File.Delete("c:\\newfile.bmp");
using (Stream s = File.Create("c:\\newfile.bmp"))
{
s.Write(fileBytes, 0, fileBytes.Length);
s.Flush();

}



}

}
catch (Exception ex)
{
//error handling

}

}




Once again I hope this comparison of the different methods of downloading content from SharePoint will help you plan your next application’s SharePoint integration. Knowing the different methods will prevent you from having write your own web service. Ultimately, taking advantage of SharePoint “out of the box” tools will make your applications easier to install, configure and maintain.

Uploading Content Into SharePoint (Let me count the ways)

Many developers new to SharePoint want to know how to get content into SharePoint from another application. This application is usually a remote application either running in a asp.net application or a desktop application. The developers may be familiar with the SharePoint object model and how to use it to put content into SharePoint. However, when it comes to doing this remotely there seems to be a lot of confusion. This confusion is brought on by the fact that there are numerous out of the box ways of doing this. You can put content into SharePoint by using web services, WebDav or frontpage remote procedure calls. The problem arises when developers chose a method and find out that certain methods don’t support certain functions that you would normally see either from the SharePoint UI or from the SharePoint object model. This article will give you a brief description of the methods available for remotely putting content into SharePoint and compare the methods based on certain factors you should be aware of as a SharePoint developer. These factors include complexity, folder creation, scalability and indexing. Complexity and scalability are rated on a scale of 1 through 10.

 

 

Method Complexity Scalability Indexing Folder Creation
Copy.asmx 5 4 yes yes*
WebDav 2 5 yes* yes*
Rpc 10 10 yes yes

* must be used in conjunction with Lists.asmx web service

 

 

Copy Web Service

To create content remotely the copy web service is probably your best bet. The copy web service enables you to create new documents and send the metadata for indexing in one call. This makes the web service more scalable. Many times users want to create a new folder to store their documents. Unfortunately, the copy web service does not have a method for creating a folder. The following is a code snippet for creating new content in SharePoint via the copy webs service:

public static void CreateNewDocumentWithCopyService(string fileName)
        {
            copyservice.Copy c = new copyservice.Copy();
            c.Url = "http://servername/sitename/_vti_bin/copy.asmx";
            c.UseDefaultCredentials = true;

            byte[] myBinary = File.ReadAllBytes(fileName);
            string destination = http://servername/sitename/doclibrary/ + Path.GetFileName(fileName);
            string[] destinationUrl = { destination };

            copyservice.FieldInformation info1 = new copyservice.FieldInformation();
            info1.DisplayName = "Title";
            info1.InternalName = "Title";
            info1.Type = copyservice.FieldType.Text;
            info1.Value = "new title";

            copyservice.FieldInformation info2 = new copyservice.FieldInformation();
            info2.DisplayName = "Modified By";
            info2.InternalName = "Editor";
            info2.Type = copyservice.FieldType.User;
            info2.Value = "-1;#servername\\testmoss";

            copyservice.FieldInformation[] info = { info1, info2 };
            copyservice.CopyResult resultTest = new copyservice.CopyResult();
            copyservice.CopyResult[] result = { resultTest };

            try
            {
                //When creating new content use the same URL in the SourceURI as in the Destination URL argument

                c.CopyIntoItems(destination, destinationUrl, info, myBinary, out result);
            }
            catch (Exception ex)
            {

            }

        }

 

The benefits of using the copy web service is that it is simple to code against. Most developers are familiar with the web service programming. Also the copy web service supports creating content and metadata in one call thus not creating multiple versions. Unfortunately, this method suffers from one problem and that is the use of byte arrays. If you plan on uploading large files lets say in excess of 2mb, then chances are you will receive sporadic “out of memory” errors. It may not happen on your development server but may happen on your production server. This is because the windows OS needs to allocate byte arrays with contiguous memory. If the server’s memory is fragmented (has a lot of available memory but not much contiguous memory) then you will receive this error. Thus, the copy web service is not very scalable. Finally, web services tend to be verbose given their soap protocol and the marshalling from string to native types makes them slower than other methods.

 

 

WebDav

Most developers are familiar with WebDav because it is used to display document libraries in Windows Explorer. Here the familiar dragging and dropping of files into SharePoint can be accomplished. You can accomplish the same thing by using the System.Net.WebClient class as follows:

 

public static void UploadFile(string fileName, string destination)
{
       WebClient wc = new WebClient();
       wc.UseDefaultCredentials = true;
       byte[] response = wc.UploadFile(destination + Path.GetFileName(fileName) , "PUT", fileName);
}

 

Ok this seems simple enough. As you can see it is not as complex as using the Copy web service. However, it does not support sending any metadata long with the file content. This can be a major problem if the document library has multiple content types, so the new file will be put into the document library with the default content type. Another big issue is if the default content type has required fields. The file will remain checked out until the fields are populated. This prevents other users from seeing the document or from being returned in any searches.  It is a great solution if you are just bulk migrating data from an external data store to SharePoint. You more than likely will have to do extra work afterwards. Adding metadata after uploading will also cause the creation of extra versions of the document being created unnecessarily. The fact that it does not use the soap protocol but straight http makes it more scalable than the copy web service. Unfortunately, it still suffers from the fact that it uses a byte array to upload the file. So sooner or later you will run into “out of memory “ exceptions.  So how can I create a folder before using WebDav?  You can use the lists web service to accomplish this:

 

public static XmlNode UpdateListItemCreateFolder(string docLibraryName, string folderName)
{
       listservice.Lists listProxy = new listservice.Lists();

       string xmlFolder = "<Batch OnError='Continue'><Method ID='1' Cmd='New'><Field Name='ID'>New</Field><Field Name='FSObjType'>1</Field><Field Name='BaseName'>” + folderName + “</Field></Method></Batch>";

       XmlDocument doc = new XmlDocument();
       doc.LoadXml(xmlFolder);

       XmlNode batchNode = doc.SelectSingleNode("//Batch");

       listProxy.Url = "http://servername/sitename/_vti_bin/lists.asmx";
       listProxy.UseDefaultCredentials = true;

      XmlNode resultNode = listProxy.UpdateListItems(docLibraryName, batchNode);

      return resultNode;

}

 

 

FrontPage RPC (Remote Procedure Calls)

Most developers are not familiar with RPC and what it can do. The complexity of coding RPC is high due to the fact that construction of commands and the interpreting of responses can be tedious and error prone. However this method proves to be the most scalable and the fastest. It also supports sending both the content and the metadata in one call. RPC has numerous commands including one for creating folders and it supports the use of streams rather than a byte array. Below is sample code to create a new document in SharePoint using RPC with a stream.

public static void CreateDocumentRPC(string name, string docLib, string title, bool overWrite, Stream fileBinary)
{

            string method = "put document: 12.0.0.4518";
            string serviceName = "http://servername/sitename/_vti_bin/_vti_aut/author.dll";
            string document = docLib + "/" + name;
            string metaInfo = string.Empty;
            string putOption = overWrite ? "overwrite" : "edit";
            string keepCheckedOutOption = "false";
            string comment = string.Empty;
            string returnStr = string.Empty;
            byte[] data;
            string returnError = string.Empty;

            string fpRPCCallStr = "method={0}&service_name={1}&document=[document_name={2};meta_info=[{3}]]&put_option={4}&comment={5}&keep_checked_out={6}";

            method = HttpUtility.UrlEncode(method);
            putOption = HttpUtility.UrlEncode(putOption);
            metaInfo = "vti_title;SW|" + title;
            fpRPCCallStr = String.Format(fpRPCCallStr, method, serviceName, document, metaInfo, putOption, comment, keepCheckedOutOption);


            try
            {
                //add line feed character to delimit end of command
                byte[] fpRPCCall = System.Text.Encoding.UTF8.GetBytes(fpRPCCallStr + "\n");

                data = new byte[fpRPCCall.Length];
                fpRPCCall.CopyTo(data, 0);

                HttpWebRequest wReq = WebRequest.Create(serviceName) as HttpWebRequest;

                wReq.Credentials = System.Net.CredentialCache.DefaultCredentials;
                wReq.Method = "POST";
                wReq.ContentType = "application/x-vermeer-urlencoded";
                wReq.Headers.Add("X-Vermeer-Content-Type", "application/x-vermeer-urlencoded");
                wReq.ContentLength = fpRPCCall.Length + fileBinary.Length;

                using (Stream requestStream = wReq.GetRequestStream())
                {

                    requestStream.Write(fpRPCCall, 0, fpRPCCall.Length);

                    byte[] tmpData = null;
                    int bytesRead = 0;
                    int chunkSize = 2097152;
                    int tailSize;
                    int chunkNum = Math.DivRem((int)fileBinary.Length, chunkSize, out tailSize);

                    //chunk the binary directly from the stream to buffer.
                    for (int i = 0; i < chunkNum; i++)
                    {
                        data = new byte[chunkSize];
                        bytesRead = fileBinary.Read(tmpData, 0, chunkSize);
                        requestStream.Write(data, 0, chunkSize);
                    }

                    //send the remainde if any.
                    if (tailSize > 0)
                    {
                        data = new byte[tailSize];
                        bytesRead = fileBinary.Read(data, 0, tailSize);
                        requestStream.Write(data, 0, tailSize);

                    }

                    //Now get the response from the server
                    WebResponse response = wReq.GetResponse();
                    int num2,num3;
                    long contentLength = response.ContentLength;
                    bool noLength = false;

                    if (contentLength == -1)
                    {
                        noLength = true;
                        contentLength = chunkSize;
                    }

                    byte[] returnBuffer = new byte[(int) contentLength];
                    using (Stream responseStream = response.GetResponseStream())
                    {
                         num3 = 0;

                        do
                        {
                            num2 = responseStream.Read(returnBuffer, num3, ((int) contentLength) - num3);
                            num3 += num2;
                            if (noLength && (num3 == contentLength))
                            {
                                contentLength += chunkSize;
                                byte[] buffer2 = new byte[(int) contentLength];
                                Buffer.BlockCopy(returnBuffer, 0, buffer2, 0, num3);
                                returnBuffer = buffer2;
                            }
                        }
                        while (num2 != 0);
                    }
                    if (noLength)
                    {
                        byte[] buffer3 = new byte[num3];
                        Buffer.BlockCopy(returnBuffer, 0, buffer3, 0, num3);
                        returnBuffer = buffer3;
                    }

                     returnStr = Encoding.UTF8.GetString(returnBuffer);

                }           

            }
            catch (Exception ex)
            {
              //error handling

            }

}

 

As you can see the complexity of coding against rpc can be daunting. You can refactor this code into something much more reusable. Parsing of the return response can be a bit strange also. Below is an example of a successful document creation response from the SharePoint server:

<html><head><title>vermeer RPC packet</title></head>
<body>
<p>method=put document:12.0.0.4518
<p>message=successfully put document 'tester2/crpc.png' as 'tester2/crpc.png'
<p>document=
<ul>
<li>document_name=tester2/crpc.png
<li>meta_info=
<ul>
<li>vti_rtag
<li>SW|rt:61935CFA-736B-4311-97AA-E745777CC94A@00000000001
<li>vti_etag
<li>SW|&#34;&#123;61935CFA-736B-4311-97AA-E745777CC94A&#125;,1&#34;
<li>vti_filesize
<li>IR|1295
<li>vti_parserversion
<li>SR|12.0.0.6318
<li>vti_modifiedby
<li>SR|BASESMCDEV2&#92;test.moss
<li>vti_timecreated
<li>TR|19 May 2009 17:28:35 -0000
<li>vti_title
<li>SW|wackout
<li>vti_lastheight
<li>IX|78
<li>ContentTypeId
<li>SW|0x010100B1C4E676904AB94BA76515774B23E02D
<li>vti_timelastmodified
<li>TR|19 May 2009 17:28:35 -0000
<li>vti_lastwidth
<li>IX|411
<li>vti_author
<li>SR|BASESMCDEV2&#92;test.moss
<li>vti_sourcecontrolversion
<li>SR|V1.0
<li>vti_sourcecontrolcookie
<li>SR|fp_internal
</ul>
</ul>
</body>
</html>

 

If there is not substring of “osstatus” or “Windows SharePoint Services Error” then it is a success.

 

I hope this helps in your evaluation of different ways you can upload data to SharePoint. There are many things to consider when selecting which method to use. High on your priority list should be speed, scalability and the ability to send metadata. Sometimes the method that looks the hardest can be the best choice and it is up to you as a developer to abstract away the complexity.

Unable to delete unmapped crawled properties in SharePoint Search

 

Technorati Tags:

One of the most convenient features of SharePoint Search is its ability to automatically discover crawled properties and create new managed properties. This is good because as a SharePoint administrator I don't want to have to take requests from users to set up new managed properties to search on. So basically whenever an incremental crawl occurs and the crawler discovers a new SharePoint column it will create a new crawled property and automatically create a new managed property to search against. The only unfortunate problem is that the crawler creates the new managed property with a name predicated with "ows". This is not very friendly but it is standard way for users to use the property in a keyword search, for example owsDocumentName:salaries.

This automatic managed property creation can be set in Shared Services Administration:SharedServices> Search Settings > Crawled Property Categories > Edit Category.

 

 

The "Automatically generate a new managed property for each crawled property discovered in this category" option is not checked by default. So many times clients set up their metadata and upload documents without turning this on. Many clients do want to go back and have to set up all the managed properties. So one option is to check the "Delete all unmapped crawled properties" option and click OK. This is supposed to delete all the crawled properties that are not mapped to any managed properties. After this you can do another full crawl and the crawler will rediscover the crawled properties that were deleted and generate new managed properties. Very nice. Unfortunately, this does not work for all unmapped crawled properties. I noticed there were a lot of unmapped crawled properties that were not being deleted. For instance, ows_baseName(Text). According to Microsoft only unmapped crawled properties will be deleted. You can verify this by examining the dbo.proc_MSS_DeleteCrawledPropertiesUnmappedForCategory stored procedure located in the SharedServices database.

So what is the deal here. Examining the managed properties stored in the SharedServices database in the MSSManagedProperties table, I can see a lot of managed properties that have very long PID values such as 214741802 that have their Hidden column set to true. The ows_baseName crawled property is mapped to one of these hidden managed properties called "TempTitle" and thus it could not be deleted. The SharePoint UI will not show these hidden managed properties. Why? All I know is that when I want to delete all unmapped crawled properties it should do so. The UI should show these. Apparently, Microsoft does not want users changing the mappings.

Below is a list of these hidden managed properties. Some of them look interesting. For instance, CrawlUrl or SiteContainer. I tried using some of these in a fulltextSql query but the search engine threw a "Property does not exist error". Really?

 

2147418028 AnchorLink

2147418033 BaseHref

2147418035 ChangeID

2147418036 CrawlObjectID

2147418090 CrawlUrl

2147418034 CRC

2147450879 DefaultProperties

2147418016 DirLink

2147418021 DirLinkSecurityUpdate

2147418020 DirLinkWithTime

2147418026 EndAnchor

2147418022 FollowAll

2147418023 FollowNone

2147418037 IndexRare

2147418039 LinkHRef

2147418040 LinkOfficeChild

2147418041 LinkOfficeChildList

2147418024 NormalizedForwardURL

2147418025 NormalizedURL

2147418042 PluggableSecurityBlob

2147418089 PluggableSecurityTrimmerId

2147418019 RedirectedURL

2147418018 Robots

2147418080 Scope

2147418027 SecurityBlob

2147418031 SecurityProvider

2147418038 SiteContainer

2147418091 Summary Description

2147418032 TempTitle

2147418029 URLAnchor