Getting Started With SharePoint Web Services

SharePoint supports interoperability and remote operations through a set of web services, and by SharePoint I am referring to WSS 3.0 (Windows SharePoint Services) and MOSS 2007 (Microsoft Office SharePoint Server). There are number of approaches for programmatic access to SharePoint, such as the object model, web services, RPC over HTTP, as well as WebDAV, all of which have their benefits and their place depending on the needs of the application. The SharePoint RPC protocols can be useful and powerful and it certainly has its strengths, especially when it comes to adding content. The object model is much more robust and feature rich than the web services, but it does not offer support for remote operations, so in order to leverage the object model your code must be running on the SharePoint server, and you must be writing .NET code in order to leverage it. There are a number of different reasons for working with SharePoint data remotely, be it administrative scripts or the need to work with SharePoint content such as tasks in an application running on a client machine, or non-SharePoint applications in the enterprise. The SharePoint web services are built on top of the object model and expose a subset of features available in the object model, and allow for remote operations and the use of any language and platform that supports the consumption of web services. For most tasks the web services that ship in SharePoint are adequate, but you may find needed operations in the object model that have not been surfaced in the web services, which will require extending the web services with a custom web service implementation. In this article we will take a look at the web services available in WSS3 and MOSS 2007 out of the box, and how to consume them from Java and .NET.

Web Services Architecture Overview

I will not go in to the detailed differences between the WSS and MOSS, as there are a number of articles that do a very good job explaining this, only as needed and where related to the web services. Keep in mind that MOSS 2007 is built on top of WSS 3.0, which is also built on top of ASP.NET 2.0 and runs in IIS (Internet Information Server). A solid understanding of IIS and ASP.NET is important, and go long way in working with and understanding SharePoint. A majority of the SharePoint web services we will discuss are included in WSS 3.0, but there are a few additional ones that are included in MOSS 2007 to leverage the additional features that ship with it, such as the more advanced enterprise search features.

The SharePoint web services are implemented in ASP.NET Web Services (ASMX), and you will find the physical files for most of these web services in the "Microsoft Shared" directory under "web server extensions\12\ISAPI" typically located at "C:\Program Files\Common Files\Microsoft Shared\web server extensions\12\ISAPI". The admin web service for the Central Administration tool is located in the ADMISAPI folder for which the central administration console contains a virtual directory named "_vti_adm". When a SharePoint site is created it will contain a virtual directory named "_vti_bin" that points to this location. IIS does not contain any application or virtual directories for subwebs, they do however contain mapping to the _vti_bin virtual directory through SharePoint metadata and HttpModules.

For each service you will find a *.wsdl.aspx file that generates the service WSDL (Web Services Description Language), a *.disco.aspx providing the service discovery implementation, and the actual *.asmx endpoint files, most of which simply contain a page directive referencing the SharePoint assembly and type containing the actual implementation for the service.

Service File Type	Description
*.wsdl.aspx (search.wsdl.aspx)	File that generates the WSDL (Web Services Description Language) in SharePoint
*.disco.aspx (seach.disco.aspx)	File that provides the discovery implementation
*.asmx (search.asmx)	Web services endpoint, most of which contain a page directive referencing the SharePoint assembly and type containing the actual implementation for the service

Out of the Box Services

There are a number of web services implemented OOTB (out of the box) in SharePoint that will address most of the common and basic tasks, from administrative tasks to search and working with list data, and much more. Below is a listing of the SharePoint web services with an overview of the functionality exposed for your reference. A simple expanded list like this helps me in working with the web services giving me a quick view of the overall services and methods available.

Service	Administration
(_vti_adm/Admin.asmx)	Administrative methods for creating deleting sites and retrieving languages used in the deployment CreateSite DeleteSite GetLanguage RefreshConfigCache
Alerts (Alerts.asmx)	Methods for working with SharePoint list item alerts DeleteAlerts GetAlerts
Authentication (Authentication.asmx)	Client proxy that provides user authentication for sites that use forms-based authentication Login (Used to authenticate) Mode (Returns the authentication mode of the current site)
Copy (Copy.asmx)	Methods to copy files between or within sites CopyIntoItems (Copy document as byte array to location on server) CopyIntoItemsLocal (Copy document from one location on the same server to another) GetItem (Creates a byte array of a document that can be passed to theCopyIntoItems method)
Document Workspace (Dws.asmx)	Methods for managing Document Workspace sites and data CanCreateDwsUrl CreateDws CreateFolder FindDwsDoc GetDwsData GetDwsMetaData RemoveDwsUser RenameDws UpdateDwsData
Forms (Forms.asmx)	Methods for returning forms that are used in the user interface when working with the contents of a list GetForm GetFormCollection
Imaging (Imaging.asmx)	Methods to create and manager picture libraries CheckSubwebAndList CreateNewFolder Delete Download Edit GetItemsByIds GetItemsXMLData GetListItems ListPictureLibrary Rename Upload
List Data Retrieval (DspSts.asmx)	Perform queries against sites and list in SharePoint Query (Performs queries against SharePoint lists and sites)
Lists (Lists.asmx)	Methods for working with Lists and List Data AddAttachment AddDiscussionBoardItem AddList AddListFromFeature ApplyContentTypeToList DeleteAttachment DeleteContentType DeleteContentTypeXmlDocument DeleteList GetAttachmentCollection GetList GetListAndView GetListCollection GetListContentType GetListContentTypes GetListItemChanges GetListItemChangesSinceToken GetListItems GetVersionCollection UndoCheckout UpdateContentType UpdateContentTypesXmlDocument UpdateContentTypeXmlDocument UpdateList UpdateListItems
Meetings(Meetings.asmx)	Create and manage Meeting Workspace Sites AddMeeting AddMeetingFromICal CreateWorkspace DeleteWorkspace GetMeetingInformation GetMeetingWorkspaces RemoveMeeting RestoreMeeting SetAttendeeResponse SetWorkspaceTitle UpdateMeeting UpdateMeetingFromICal
People(People.asmx)	Resolve and find Principals ResolvePrincipals
Permissions (Permissions.asmx)	Methods for working with permissions for a site or list AddPermission AddPermissionCollection GetPermissionCollection RemovePermission RemovePermissionCollection UpdatePermission
Directory Management(sharepointemailws.asmx)	Methods for managing Active Directory e-mail distribution groups and their memberships ChangeContactsMembershipInDistributionGroup ChangeUsersmembershipInDistributionGroup CreateContact CreateDistributionGroup DeleteContact DeleteDistributionGroup GetJobStatus ModifyContact ModifyDistributionGroup RenameDistributionGroup
Site Data (SiteData.asmx)	Methods that return metadata or list data from sites or lists EnumerateFolder GetAttachments GetChanges GetContent GetList GetListCollection GetListItems GetSite GetSiteAndWeb GetSiteUrl GetURLSegments GetWeb
Sites(Sites.asmx)	Methods for returning information about the collection or site template ExportWeb GetSiteTemplates GetUpdatedFormDigest ImportWeb
Search(spsearch.asmx)	Methods for searching via search services Query QueryEx Registration Status
Users & Groups(usergroup.asmx)	Methods for working with users role definitions and groups AddGroup AddGroupToRole AddRole AddRoleDef AddUserCollectionToGroup AddUserCollectionToRole AddUserToGroup AddUserToRole GetAllUserCollectionFromWeb GetGroupCollection GetList GetListAndView GetListCollection GetGroupCollectionFromRole GetGroupCollectionFromSite GetGroupCollectionFromUser GetGroupCollectionFromWeb GetGroupInfo GetRoleCollection GetRoleCollectionFromGroup GetRoleCollectionFromUser GetRoleCollectionFromWeb GetRoleInfo GetRolesAndPermissionsForCurrentUser GetRolesAndPermissionsForSite GetUserCollection GetUserCollectionFromGroup GetUserCollectionFromRole GetUserCollectionFromSite GetUserCollectionFromWeb GetUserInfo GetUserLoginFromEmail RemoveGroup RemoveGroupFromRole RemoveRole RemoveUserCollectionFromGroup RemoveUserCollectionFromRole RemoveUserCollectionFromSite RemoveUserFromGroup RemoveUserFromRole RemoveUserFromSite RemoveUserFromWeb UpdateGroupInfo UpdateRoleDefInfo UpdateRoleInfo UpdateUserInfo
Versions (Versions.asmx)	Methods for working with file versions DeleteAllVersions DeleteVersion GetVersions RestoreVersion
Views(Views.asmx)	Methods for working with list views AddView DeleteView GetViewCollection GetViewHtml UpdateView UpdateViewHtml UpdateViewHtml2
Web Part Pages(WebPartPages.asmx)	Methods for working with Web Part Pages AddWebPart AddWebPartToZone AssociateWorkflowMarkup ConvertWebPartFormat DeleteWebPart ExecuteProxyUpdates FetchLegalWorkflowActions GetAssemblyMetaData GetBindingResourceData GetCustomControlList GetDataFromDataSourceControl GetFormCapabilityFromDataSourceControl GetSafeAssemblyInfo GetWebPart GetWebPart2 GetWebPartCrossPageCompatibility GetWebPartPage GetWebPartPageConnectionInfo GetWebPartPageDocument GetWebPartProperties GetWebPartProperties2 RemoveWorkflowAssociation RenderWebPartForEdit SaveWebPart SaveWebPart2 ValidateWorkflowMarkupAndCreateSupportObjects
Webs(Webs.asmx)	Methods for working with sites and subsites CreateContentType CustomizeCss DeleteContentType GetActivatedFeatures GetAllSubWebCollection GetColumns GetContentType GetContentTypes GetCustomizedPageStatus GetListTemplates GetWeb GetWebCollection RemoveContentTypeXmlDocument RevertAllFileContentStreams RevertCss RevertFileContentStream UpdateColumns UpdateContentType UpdateContentTypeXmlDocument WebUrlFromPageUrl
MOSS Search (Search.asmx)	Methods for searching via MOSS (Microsoft Office SharePoint Server) Search services, which also includes a method to retrieve the managed search properties GetSearchMetadata (Search managed properties) Query QueryEx Registration Status

Consuming the Query Services

Now that we have an understanding of the SharePoint web services, there is nothing like kicking the tires and taking a ride after reading through the owner's manual. We don't have time here to dive deep in to all of the web services, and will save the others for the next article. For some reason a number of the web services in SharePoint either take or return an XML Node or an XML encoded string, and unfortunately without a typed schema the tools are unable to infer what this should be and generate those nice proxy objects for us. I have seen a number of implementations using string formatting with the web services to address the lack of a typed schema, and have searched long and far for a schema that describes the XML passed and returned from these services. I ended up creating queryresponse.xsd which was inferred from a combination of the documentation and the xml passed in and out of these services, allowing me to generate the class representation using xsd.exe for .NET users and JAXB for java users.

Note: There are two different search web services, as discussed previously, search.asmx for MOSS and spsearch.asmx for WSS. These services are very similar in their schema, but attempting to call spsearch.asmx on a machine running MOSS will throw an exception due to the fact that the WSS indexing and query services have been disabled in favor of the more advanced search features in MOSS.

Example using .NET

Consuming the SharePoint web services from .NET is rather simple and straight forward, you simply need to add a web reference and use the generated proxy. We will add a step to leverage the XSD we created to generate a set of classes representing a query and response that we will use to serialize the XML sent and deserilize the XML returned.

Configure Visual Studio 2005 to add XSD.exe to the menu

Visual studio includes a tool called XSD.exe that can be used to generate .NET types from an XSD. This is however a command line tool and I like to add a menu option to visual studio to do this from the IDE. If you have already done this or have some other add-in to do something similar you can skip this step.

1. Open the "External Tools" Dialog in the Visual Studio "Tools" menu
2. Click "Add"
3. Set the necessary properties and click "Ok"

Property	Value
Command	XSD >> CSharp
Title	C:\Program Files\Microsoft Visual Studio 8\SDK\v2.0\Bin\xsd.exe
Arguments	$(ItemPath) /c
Initial Directory	$(ItemDir)
Use Output Window	Checked

Download the XSD files

Download the XSD files from here which will be used to generate the proxy classes used to serialize and de-serialize the search requests and responses. Unpack the downloaded file to a working directory; this file contains the XSD along with the completed project.

Create a Project and Generate a .NET type from the XSD Now that we have the XSD we will create a Windows Forms project and generate a .NET type form the downloaded XSD. After opening up Microsft.Search.Query.xsd, go to the tools menu and select the XSD menu item that we created in the previous step to generate the Microsoft_Search_Query.cs file.

Add a web reference

We are going to start by adding a web reference to the search web service, by right-clicking on the "references" node for our project in the "solution explorer" window. There we will select "Add Web Reference", enter "http://<server_name>/_vti_bin/spsearch.asmx" in the URL textbox and then click "Go". You may be prompted to login to the SharePoint server and you should now have a list of Methods supported by this service. Add "SPSearch" for the "Web Reference Name" and click "Add Reference".

Subclass the generate proxy

To simplify things even further, we will subclass the generated proxy with our own so that we can wrap the QueryService to hide the validation, serialization, and de-serialization of the XML passed and returned in the Query method using our class we generated from the XSD.

Create the application to call the service

Now we simply instantiate a query class and QueryService class, and then pass the query object to the Query method on the QueryService class.

SPSearch.QueryService search = new SPSearch.QueryService();
Microsoft_Query_Request request = new Microsoft_Query_Request();
search.Query(request);

Example using Java

Consuming the SharePoint web services from Java is not much different, the fundamental steps are the same, generate our class from the provided XSD, generate a web services proxy, and build the application. Most of the challenges you will face in consuming SharePoint web services from java are with authentication and the use of DataSet's in the SharePoint web services. There are a number of Java IDEs available to simplify working with web services, but since I primarily work with .NET these days I will show an example using the SDK and in this case leave the choosing of an IDE to you.

Note: This example was built and tested using J2SE 6.0 Update 3

Java SharePoint Sample Project Directory and Root Files

The full project and generated files are included along with couple of batch files to build and run the sample. You will need to modify "xjc-build.bat" and "build-run.bat" with the home directory for your JDK bin folder to run this, and edit wsspsample/Main.java with your server settings (endpoint, username, and password) before building and running.

Import the Service WSDL

I start by retrieving the WSDL from my SharePoint server to my development machine for a couple reasons. We will need to modify any WSDL that uses ADO.NET Data Sets, so as to work properly with the tools. Also, access to the WSDL on the SharePoint server typically requires authentication and the Java generated proxy requires the WSDL in the constructor, making authentication a bit of problem for me.

Retrieve the Search WSDL

For Windows SharePoint Services 3.0 search service open up Internet Explorer and navigate to Error! Hyperlink reference not valid. then from File|Save As..., save the page to the working directory as spsearch.wsdl. If you are working with MOSS 2007 then you will use Error! Hyperlink reference not valid.

Modify the Search WSDL

The QueryEx method as well as the MOSS 2007 GetSearchMetaData method returns ADO.NET Data Sets. An ADO.NET Data Set is dynamically bound and is represented in WSDL with a cyclic reference to the WSDL schema attribute, which causes some problems for us with the JAXB wsimport tool. I found some workarounds that worked well with previous versions of the Java JDK, but these did not work with the version I was using. Instead I chose to modify the WSDL as follows by opening it up in notepad and searching for instances of "s:schema" to modify where it was used as a ref in a schema "element" element, and remove the cyclic reference and leaving the schema "any" element.

<s:sequence>
     <s:element ref="s:schema"/>
     <s:any/>
</s:sequence>

Was change to the following:

<s:sequence>
     <s:any minOccurs="0" maxOccurs="unbounded"/>
</s:sequence>

Import the WSDL

Now that we have our WSDL locally and it has been modified, we can use JAXB wsimport.exe tool to generate the proxy classes.

wsimport -p wsspsample.webref.spsearch -keep spsearch.wsdl
wsimport -p wsspsample.webref.search -keep search.wsdl

We use the -keep flag with the import to keep the java code so that we can tweak the generated code for our application. The tools generate code with an absolute path, so if you are planning on moving the application to a different directory and storing the WSDL with the application we will need to modify "webref\search\QueryService.java" url and build it ourselves. To do this we simply remove the absolute path to the WSDL and use a relative path instead.

url = new URL("file:search.wsdl");

Generate Request Response Classes

To generate the Request and Response classes you are going to need to use the four "Microsoft.Search" XSDs that I have created and the JAXB xjc.exe tool. In the following we simply suppress generation of package level annotation and specify a target package, and the tool creates our classes for us.

xjc -npa -p wsspsample.xom.query -d . Microsoft.Search.Query.xsd
xjc -npa -p wsspsample.xom.response -d . Microsoft.Search.Response.xsd

Use the Generated Classes

We have an class representation of the Query and Response packets passed to and returned form the QueryService Query method, as well as a Query service proxy. Now it's time to put that to use in an application. The first thing we need to do is create a QueryService instance, the constructor for this object will go and retrieve the local copy of our WSDL and load settings. From the QueryService instance we retrieve a QueryServiceSoap instance via the getQueryServiceSoap method. Note that you will need to hang on to and use the QueryServiceSoap reference as we do with the qsp variable, since the getQueryServiceSoap method seems to return a new instance. We can then modify the endpoint address here to something other than what is contained in the WSDL to make our application more dynamic allowing it to connect to other SharePoint sites and SubSites other than that set in the WSDL.

QueryService qs = new QueryService();
QueryServiceSoap qsp = qs.getQueryServiceSoap();

BindingProvider bp = (BindingProvider)qsp;
//bp.getRequestContext().put(BindingProvider.USERNAME_PROPERTY, "Administrator");
//bp.getRequestContext().put(BindingProvider.PASSWORD_PROPERTY, "pass@word1");
bp.getRequestContext().put(BindingProvider.ENDPOINT_ADDRESS_PROPERTY,
"http://barbie/_vti_bin/spsearch.asmx");

Now we create methods to serialize the query and deserlialize the response packets.

public static String SerializeQuery(QueryPacket qp)
{
     try
     {
         JAXBContext jc =JAXBContext.newInstance("wsspsample.xom.query");
         Marshaller ma = jc.createMarshaller();
         ma.setProperty(Marshaller.JAXB_FORMATTED_OUTPUT,Boolean.TRUE);
         ByteArrayOutputStream os = new ByteArrayOutputStream();
         ma.marshal(qp, os);
         return os.toString();     
     }
     catch(JAXBException ex)
     {
         return "";
     }
 }
 public static ResponsePacket DeserializeResponse(String s)
 {
     try
     {
         JAXBContext jc = JAXBContext.newInstance("wsspsample.xom.response");
          Unmarshaller um = jc.createUnmarshaller();
         StreamSource source = new StreamSource(new StringReader(s));
          return (ResponsePacket)um.unmarshal(source);
     }
     catch (JAXBException ex)
     {
         return new ResponsePacket();
     }
 }

From here we simply create a QueryPacket and necessary classes, set our values and call the service.

// Create a Search Query Packet Object
QueryPacket qp = new QueryPacket();
QueryType qt = new QueryType();
ContextType ct = new ContextType();
QueryTextType ctt = new QueryTextType();

ct.setQueryText(ctt);
qt.setContext(ct);
qp.setQuery(qt);

// Set search values
ctt.setValue("sharepoint");
ctt.setType("STRING");      //This is the default - not necessary
ctt.setLanguage("en-us");   //This is the default - not necessary

// Call the web service query
String sResponse = qsp.query(SerializeQuery(qp));

// Deserialize the response
ResponsePacket resp = DeserializeResponse(sResponse);

Build and Run the Application

We have our generated web services proxy and object model for query request and response, as well as the application that leverages them, now we simply need to cross our fingers, build, run, and test this. You will need a SharePoint server setup and will want to set the correct endpoint address used in Main.java to your test SharePoint server as well as the credentials if necessary.

References and Links

Windows SharePoint Services 3.0 Web Services Reference.