Just stumbled upon a bug in SharePoint 2010. Imagine this scenario: You have two lists and add them to a Site as DataForm WebParts. You then configure the Web Parts so that one of them sends Filter values to the other one and uncheck the option for "Send first row to connected Web Parts when page loads". You refresh the Page and notice that the list that receives the Filter (the Consumer) does not show any items because the list that provides the Filter (the Provider) does not have any items selected.

In SharePoint 2007, not selecting an item in the Provider List would cause the consumer List to display all items instead, which was (at least for us) highly desirable. In SharePoint 2010, you have to select an Item in the Provider to provide a filter and there is no way to have the Consumer List display all items anymore.

The reason why I think this is a bug is the result of an extended Debugging session that eventually led me to the class Microsoft.SharePoint.WebPartPages.DataFormWebPart and specifically it's method ApplyReceivedFilterData. I'm not going to paste the source of this class, but the culprit is this line: if (dictionary2[str] != null)

This is supposed to only add Filters that have a value and skip the ones that are null. The problem is that dictionary2[str] is not null but DBNull, which then causes SharePoint 2010 to process the value anyway and generate a nasty <Query><Where><IsNull><FieldRef Name="FieldToReceiveFilter" /></IsNull></Where></Query> in the SelectCommand of the underlying DataSource

Unfortunately, DataFormWebPart.ApplyReceivedFilterData is not virtual, so sub-classing it does not seem to be an option. Not sure if there is a good workaround for this. I verified this behavior in both RTM and December 2010 Hotfixed versions of SharePoint 2010.

Edit: Just checked the SharePoint 2007 DataFormWebPart.ApplyReceivedFilterData() in Reflector and lo and behold, they did check for DBNull in 2007:

        Label_0046:
            if (current[str] is DBNull)
            {
                goto Label_006D;
                // Note by myself: Label_006D is the call to MoveNext, aka. continue
            }

However, Path.GetExtension does return the extension with a dot (or string.Empty in case the file does not have an extension).

Additionally, there may be upper/lowercase conflicts.

SharePoint 2010 added a ton of new Features, including a separation into Service Applications (replacing the SSP) and many new Social Features like tagging and commenting. Also, the Development tools radically improved. However, I think that the Object Model didn't scale well over the years. I think it's downright broken in SharePoint 2010 and that the next version of SharePoint needs a completely new Object Model with properly separated APIs/Modules.

Why? Here are some of the big and small issues I've encountered while I wanted to do some really simple things.

SPMetal doesn't generate all field Types
Create a new List in SharePoint, then enable the Managed Metadata and Enterprise Keywords option on the List or add a new Field to the List which is a Managed Metadata column. Run SPMetal against the list. Look how the Proxy doesn't have the Keyword field.

It's bad enough that the generated proxy is unnecessarily fragile, but not supporting all of the *built-in* field types makes it unusable for all but the most simple queries. Granted, Managed Keywords is a separate Feature of Standard/Enterprise, but it's still an official Microsoft out of the box feature.

Querying Managed Keywords through CAML is only possible by Name, not by ID
Let's say you want to query the List and get all items that have two different keywords. As Keywords can have the same name (if they are in different paths of the Term Set), I thought it would be best to query with the Guid of the Term.

Well, turns out you can't. You can query by their WssId though (which is an int that seems to be assigned uniquely on each Site Collection) by adding the LookupId="TRUE" attribute. However, try to chain two queries together with an <And> or by adding the value twice. What happens? You get all items that have any Keyword. It's an Or, not And.

The only way is to Query by the Title and hope you never have duplicate titles or that you can enable the Full Path option on the Field. Now, for the standard Keywords this isn't that much of a problem as they don't have a hierarchy and therefore no duplicates, but you aren't always that lucky.

This is a deeper issue though, it's a problem with the LookupMulti field from which the Multi-Taxonomy Field inherits.

No way to query the User Profile store effectively
This is something that is easy in pretty much every CMS on the market: Give me a distinct list of departments in the company. If the user profile store has a field that holds the department, it's literally a SELECT DISTINCT(Department) from USERS ORDER BY Department.

In SharePoint, there is no way to do that. You could query the User Information list on the main site collection, but that may not contain all users. You can query Active Directory directly, but what's the point of the User Profile store in SharePoint then?

If you want to get a list of all departments, you get a UserProfileManager and loop through all profiles, then fill a List<string> or HashSet. This is slow and resource intensive.

No way to get some tag statistics efficiently
Another really simple scenario: I have a list that contains a lot of items. Users can use the Tags & Notes feature of SharePoint to tag items. We want to get statistics: Give me the top 10 tags that start with 'su' and how often they were used on this list.

The first problem is obviously that SharePoint stores social tags against an exact URL. And when I mean exact URL, then I mean:

• Alternate Access Mappings are not supported. If one person uses http://internalportal and another person uses https://portal.internal.example.com, they won't see each other's tags and notes
• If you have a Ribbon open and your URL has the "InitialTabID=...." QueryString, then your Tags and Notes will not be visible to other people who don't have it

So that's bad enough. But what about getting the statistics? Using SQL, such a feature is developed in 5 minutes since all you need is this query:

SELECT TOP 10 InputTermLabel, COUNT(InputTermLabel) AS Count
FROM dbo.SocialTags
WHERE InputTermLabel LIKE 'su%' AND UrlID IN (
SELECT UrlID
FROM dbo.Urls
WHERE Url like 'http://myportal/Lists/MyList/DispForm.aspx%')
GROUP BY InputTermLabel
ORDER BY Count DESC

Using the Object Model, it is impossible to do this effectively. The closest I got is this:

var result = new Dictionary<string, int>();
var baseUrl = "http://myportal/Lists/MyList/DispForm.aspx";
var stm = new SocialTagManager(SPServiceContext.GetContext(site));
var terms = stm.GetAllTerms(new Uri(baseUrl),0);
foreach (var term in terms)
{
    var name = term.Term.Name;
    if (!name.StartsWith("su", StringComparison.InvariantCultureIgnoreCase)) continue;
    var tc = stm.GetUrls(term.Term);
    int usageCount = tc.Count(url => url.AbsoluteUri.StartsWith(baseUrl));
    result[name] = usageCount;
}
result = result.OrderByDescending(kvp => kvp.Value)
             .Take(10).ToDictionary(kvp => kvp.Key, kvp => kvp.Value);

As you may see, this is incredibly inefficient. The SQL Server sends me ALL the terms (at least I can specify a base URL!) and only on the client I can filter the ones starting with 'su'. Then I have to ask for every single Term to get the URLs. If you have a couple hundred tags, this is an incredible resource hog (tons of SQL Queries, a lot of network traffic, high memory usage). That would be the moment to wrap it into a Timer Job, cache it and not have those statistics in Real time.

The last one made me break one of the golden rules with SharePoint
The golden rule is: Never ever directly talk to the SQL Database, always go through to the SharePoint Object Model.

Today, I broke that rule, and I don't feel dirty or guilty. That last point was the final straw. The Object Model simply doesn't cut it anymore, it's weaker than what I can do with some low end PHP CMS Systems, weaker than stuff like XOOPS or PHP-Nuke. The whole social functionality is poorly integrated.

After trying to get Tag-Statistics for 6 hours, I spent 30 Minutes to write a small wrapper: Iterate through all the SPDatabases in the farm, find the SocialDB that belongs to the current Web Application, use Reflection to get the SqlSession property and use Reflection again to call ExecuteReader on it. As a result, I get a nice SqlDataReader back.

No fighting with the Object Model, just plain "I want this, give it to me without killing all my resources".

Obviously, I now need to be careful with Service Packs and Hotfixes. I've set up a little tool that can compare SQL Schemas. That way, whenever a Hotfix/Service Pack gets released I run it to see if any of the Databases/Tables I use changed and adapt my code accordingly. Even if it takes me 15 Minutes to make the changes, Microsoft would have to release 24 Schema-Changing Updates to break even with the 6 hours I wasted today on this.

Granted, I use it strictly for SELECTs and only as a last resort and I wouldn't release code containing it, but as some point I have to get pragmatic about it. I can't spend a month developing something that should only take a week at maximum just because SharePoint doesn't like me.

Sorry for ranting so much, but SharePoint 2010 simply feels like a 10 year old application in places. There is so much stuff that was bad when it was still SharePoint Portal Server 2001 aka. Tahoe (ONET.xml...) and that's worse now that we have nicer technologies.

Why are pretty much all of the collections non-generic? Why is SPListItemCollection still a non-generic Collection of Object and not a List<SPListItem>?
Why are there no standalone SharePoint MSBuild tasks, thus requiring Visual Studio 2010 to be installed on a build server?
Why does creating Content Types through XML Files require me to specify fields twice? Why does creating a List for that Content Type through XML require me to specify the fields a third time?
Why does a product than can cost hundreds of thousands of dollars in licensing still require me to depend on Reflector and looking at the Stored Procedures just to understand simple things?
Why is there no easy way to check for the existence of certain items in some collections without iterating through it completely? Usually, there is only an indexer that throws an Exception if an item isn't found.
Why are so many useful functions and/or classes internal or sometimes even internal sealed? Many collections have internal "GetItem" functions that return null if the item wasn't found, which is great but as said, they are internal. The external functions usually wrap the GetItem call and throw an Exception if null - ARGH!
Why is there no MS Connect site to report issues and ask someone from MS to resolve them in a patch? Oh wait, there is a SharePoint Community Connect Site. It even has some bugs and the occasional Microsoft poster - except that all the real issues are ignored there.
Why are the simple things so hard and the hard things impossible to do?

The weird thing is, I'm not even that unhappy developing for it. As said, I still believe that SharePoint is a good product for the users to use and does so many things right there.
But I also believe that the Object Model needs a huge refresh.

I assume Microsoft wants to make upgrading as painless as possible, seeing how SharePoint is targeted at Enterprises. So very little changes, dragging the old stuff along.
I just hope that the next SharePoint version will be based on the .net 4 CLR and that the breaking changes there (unlike 3.0 and 3.5, .net 4.0 introduced a new CLR) would allow some people at Microsoft to show some bravery by revamping the whole OM and turning SharePoint from an ancient but modern looking product into a modern product.

</rant>

With SharePoint 2010 you get the tools to convert the time according to the users time zone. There are two types of Regional Settings: Each Site (SPWeb) has RegionalSettings that specify the Time Zone (and Locale, Calendar etc.) for that site. This is useful if you have sites that are predominately used by people in one time zone. The second type of Regional Settings are the one the user (SPUser) can set (My Settings - My Regional Settings). Those are the same settings as the ones on SPWeb, but each user can specify their own setting.

When storing Dates in code, you have two options:

• Store the time in local time of the Web and use DatesInUtc = true on a SPQuery to get it back as Utc
• Store the time in Utc and to not use DatesInUtc on SPQuery

What does that mean? As said, each SPWeb has it's own Regional Settings. Let's assume you have a date of 2010-06-14 15:00:00.

If the TimeZone of the SPWeb is Pacific Time (GMT-8) and you query the List using SPQuery, you get back this date. If you however set DatesInUtc = true on the SPQuery, you get back 2010-06-14 22:00:00. SharePoint doesn't know if 15:00:00 was already UTC, so using DatesInUtc may translate a date twice.

The caveat here is that when storing dates, you would normalize them either to UTC or to the Local Time of the Web. What would you do if some employee from Texas (which runs on Central Time, GMT-6) enters 2010-06-14 15:00:00? You would need to store it either as GMT-8 (so the time becomes 13:00:00) or as UTC (22:00:00).

Needless to say, I prefer to store all dates as UTC if the list isn't visible to the user directly. Then when querying the list through Code, I can just convert the time to whatever the user's timezone is:

var user = SPContext.Current.Web.CurrentUser;
// Always perform a Null-Check on SPUser.RegionalSettings
if (user.RegionalSettings != null)
{
    return user.RegionalSettings.TimeZone.UTCToLocalTime(listDateUtc);
}
else
{
    // User didn't set a time zone, so use the one from the Web
    return SPContext.Current.Web.RegionalSettings.TimeZone.UTCToLocalTime(listDateUtc);
}

Overall, the option for people to set their own timezones independently from the SPWeb is a fantastic and long needed addition. On the other hand, it does make dealing with times a bit more complex.

If the list is visible to the user, you may need to normalize the times differently (for example, use user.RegionalSettings.TimeZone.LocalTimeToUTC to convert a user time to UTC and then SPWeb.RegionalSettings.TimeZone.UTCToLocalTime to convert the time to the Web-Time).

If you do build custom pages that make use of the Microsoft.SharePoint.WebControls.DateTimeControl then you can just use UseTimeZoneAdjustment="true" on it to have it automatically convert to UTC and back (SelectedDate will be UTC when accessed through code, but the User's/Web's time when rendered).

As this is backed by a .net class, I had some issues getting changes to it done. I've added a new field, updated the .bdcm file, redeployed and started a Full Crawl, only to get error messages in the Crawl saying that the Property that I've added cannot be found in the backing class.

Solution is as simple as logical: Reset the SharePoint Server Search 14 service (OSearch14). As this holds the Assembly in memory and as the Sharepoint deployment tools do only recycle the IIS process, there will be a mismatch between the .bdcm file and the (outdated, but still loaded) assembly.

Also, you might have to re-create the Content Source. And don't forget to increase the version number if you edit the .bdcm file manually.

Here is the source code of the UpdateProgressEx, which includes a new Property, DisplayInline. If this is set to true, then we render a span with display: inline. If this is set to false or not set at all, we have the default behavior of a div/block. If DisplayInline is set to true, then we force DynamicLayout to be false. Why? Because if DynamicLayout is true, then it seems like the MS AJAX JavaScript control the display, and they render it as a block. As I did not want to modify it any further, I just force DynamicLayout to false because for me, that works good enough. Also, I haven't tested if it works in visual design mode and if the property panel shows the new DisplayInline Property, as I work only in code view. YMMV, but it's Open Source after all, so feel free to properly fix it

Here is the code:

// UpdateProgressEx
//
// Author:
//  Michael Stum <website@stum.de>
//  http://www.stum.de/2010/01/28/an-asp-net-ajax-updateprogress-that-can-render-inline
//
// Original Author:
//   Igor Zelmanovich <igorz@mainsoft.com>
//   (C) 2007 Mainsoft, Inc.  http://www.mainsoft.com
//
//
// Permission is hereby granted, free of charge, to any person obtaining
// a copy of this software and associated documentation files (the
// "Software"), to deal in the Software without restriction, including
// without limitation the rights to use, copy, modify, merge, publish,
// distribute, sublicense, and/or sell copies of the Software, and to
// permit persons to whom the Software is furnished to do so, subject to
// the following conditions:
//
// The above copyright notice and this permission notice shall be
// included in all copies or substantial portions of the Software.
//
// THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
// EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
// MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
// NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE
// LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
// OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
// WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
using System;
using System.Collections.Generic;
using System.ComponentModel;
using System.Web.UI;

namespace StumDE.WebControls
{
    [PersistChildren(false)]
    [ParseChildren(true)]
    [DefaultProperty("AssociatedUpdatePanelID")]
    [Designer("System.Web.UI.Design.UpdateProgressDesigner, System.Web.Extensions.Design, Version=1.0.61025.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35")]
    public class UpdateProgressEx : Control, IScriptControl
    {
        ITemplate _progressTemplate;
        ScriptManager _scriptManager;

        [Category("Behavior")]
        [DefaultValue("")]
        [IDReferenceProperty(typeof(UpdatePanel))]
        public string AssociatedUpdatePanelID
        {
            get
            {
                return (string)ViewState["AssociatedUpdatePanelID"] ?? String.Empty;
            }
            set
            {
                ViewState["AssociatedUpdatePanelID"] = value;
            }
        }

        [Category("Behavior")]
        [DefaultValue(false)]
        public bool DisplayInline
        {
            get
            {
                object o = ViewState["DisplayInline"];
                if (o == null)
                    return false;
                return (bool)o;
            }
            set
            {
                ViewState["DisplayInline"] = value;
            }
        }

        [Category("Behavior")]
        [DefaultValue(500)]
        public int DisplayAfter
        {
            get
            {
                object o = ViewState["DisplayAfter"];
                if (o == null)
                    return 500;
                return (int)o;
            }
            set
            {
                ViewState["DisplayAfter"] = value;
            }
        }

        [Category("Behavior")]
        [DefaultValue(true)]
        public bool DynamicLayout
        {
            get
            {
                // If DisplayInline is set, force dynamic layout to be false
                if(DisplayInline) return false;
                object o = ViewState["DynamicLayout"];
                if (o == null)
                    return true;
                return (bool)o;
            }
            set
            {
                ViewState["DynamicLayout"] = value;
            }
        }

        [PersistenceMode(PersistenceMode.InnerProperty)]
        [Browsable(false)]
        public ITemplate ProgressTemplate
        {
            get
            {
                return _progressTemplate;
            }
            set
            {
                _progressTemplate = value;
            }
        }

        ScriptManager ScriptManager
        {
            get
            {
                if (_scriptManager == null)
                {
                    _scriptManager = ScriptManager.GetCurrent(Page);
                    if (_scriptManager == null)
                        throw new InvalidOperationException(String.Format("The control with ID '{0}' requires a ScriptManager on the page. The ScriptManager must appear before any controls that need it.", ID));
                }
                return _scriptManager;
            }
        }

        protected virtual IEnumerable<ScriptDescriptor> GetScriptDescriptors()
        {
            string updatePanelClientId;
            if (String.IsNullOrEmpty(AssociatedUpdatePanelID))
                updatePanelClientId = null;
            else
            {
                var updatePanel = FindControl(AssociatedUpdatePanelID) as UpdatePanel;
                if (updatePanel == null)
                    throw new InvalidOperationException("No UpdatePanel found for AssociatedUpdatePanelID '" + AssociatedUpdatePanelID + "'.");
                updatePanelClientId = updatePanel.ClientID;
            }
            var descriptor = new ScriptControlDescriptor("Sys.UI._UpdateProgress", this.ClientID);
            descriptor.AddProperty("associatedUpdatePanelId", updatePanelClientId);
            descriptor.AddProperty("displayAfter", DisplayAfter);
            descriptor.AddProperty("dynamicLayout", DynamicLayout);
            descriptor.AddProperty("displayInline", DisplayInline);
            yield return descriptor;
        }

        protected virtual IEnumerable<ScriptReference> GetScriptReferences()
        {
            yield break;
        }

        protected override void OnPreRender(EventArgs e)
        {
            base.OnPreRender(e);
            ScriptManager.RegisterScriptControl(this);

            if (_progressTemplate == null)
                throw new InvalidOperationException(String.Format("A ProgressTemplate must be specified on UpdateProgress control with ID '{0}'.", ID));

            var container = new Control();
            _progressTemplate.InstantiateIn(container);
            Controls.Add(container);
        }

        protected override void Render(HtmlTextWriter writer)
        {
            if (DynamicLayout)
                writer.AddStyleAttribute(HtmlTextWriterStyle.Display, "none");
            else
            {

                writer.AddStyleAttribute(HtmlTextWriterStyle.Display, DisplayInline ? "inline" : "block");
                writer.AddStyleAttribute(HtmlTextWriterStyle.Visibility, "hidden");
            }
            writer.AddAttribute(HtmlTextWriterAttribute.Id, ClientID);
            if (DisplayInline)
            {
                writer.RenderBeginTag(HtmlTextWriterTag.Span);
            }
            else
            {
                writer.RenderBeginTag(HtmlTextWriterTag.Div);
            }
            base.Render(writer);
            writer.RenderEndTag();

            ScriptManager.RegisterScriptDescriptors(this);
        }

        #region IScriptControl Members

        IEnumerable<ScriptDescriptor> IScriptControl.GetScriptDescriptors()
        {
            return GetScriptDescriptors();
        }

        IEnumerable<ScriptReference> IScriptControl.GetScriptReferences()
        {
            return GetScriptReferences();
        }

        #endregion
    }
}

Well, turns out that SharePoint 2007/WSSv3 explicitly does not support nested master pages. To quote an Article in MSDN directly:

Although the underlying technology enables the creation of nested master pages, nested master pages are not supported in Windows SharePoint Services. Using nested master pages can cause unexpected behavior in some scenarios, such as preventing content from rendering.

Indeed, unexpected behavior, such as content not rendering is the exact problem I have. Now I remember why I love SharePoint so much: It's taking the essential functions of ASP.net and then adds some traps that really hurt you unexpectedly.

Basically, SharePoint Designer Workflows cannot start themselves anymore. So if you have a Workflow that is started when an item is changed and you change the item from within the workflow, previously SharePoint would start a second instance of your workflow. This was both blessing and curse. Curse because if you were not careful, you would start an infinite loop. Not fun if your boss asks you why he received 4500 mails during the night... But blessing because you could use it to build State-Machine Workflows from SharePoint Designer.

I guess it's the Infinite Loop part that triggered the change, and I think that change made sense. But please dear SharePoint team: Document a change like this, because it IS breaking, and you know as well as me that SP Designer Workflows cannot be debugged, so it's not only a pain in the ass, but it's also a way to really make people lose trust in your application.

Anyway, you can read more about this change at the SharePoint Designer Team Blog. The error message you will see in your SharePoint Logfile looks like this by the way:

08/26/2009 15:11:59.50 w3wp.exe (0x272C) 0x25E0 Windows SharePoint Services Workflow Infrastructure 72fv Unexpected AutoStart Workflow: Microsoft.SharePoint.SPException: Recursive workflows are not permitted. at Microsoft.SharePoint.Workflow.SPWorkflowAutostartEventReceiver.AutoStartWorkflow(SPItemEventProperties properties, Boolean bCreate, Boolean bChange, AssocType atyp)

Looks like the more versions they make, the more stuff they drop out of the name. First it was "SharePoint Portal Server", then "Office SharePoint Server", and now just "SharePoint Server". Makes sense to be fair, SharePoint is really a brand on it's own, so why blur the lines too much. But the actual question will be: Is 2010 going to end this SPWeb.Dispose madness, that not only causes all sorts of weird issues, but is also inconsistent.

Really, SPWeb.Dispose is a big gun that is made for people to shoot themselves in the foot, and I'd love to see that improved more than anything else (Well, apart from CAML, that abomination of SQL).

Now, there is a truly hidden Gem: Expression Builders. Using <%$ MyExpressionBuilder.Property %>, I can directly do this type of DataBinding in the .aspx file without requiring code behind. I have not yet tested if it works if you have not set AllowServerSideScript="true" in the PageParserPaths tag of SharePoint's web.config (something that I would always avoid).

The full article about Custom Expression Builders can be found over at Phil Haack's blog, along with an explanation how this is different from <%= %>.

If you want to create a Group that merely serves as a Container for Users, it will not show up in SPWeb.Groups, and calls to Groups.GetById() will fail. If you do not want to add some dummy permissions, use SPWeb.SiteGroups instead, which contains all groups in the Site Collection.

Calling SPGroupCollection.GetByID will throw an SPException if the Group was not found.

I have written a small application that will do this automatically for you: .disco/.wsdl in, disco.aspx and wsdl.aspx out. It requires .net Framework 3.0 (just as Sharepoint) and is available in both binary and source form on CodePlex: http://www.codeplex.com/discoverter

On a related note, this was also a test to see how good CodePlex is. I will write up a little CodePlex vs. SourceForge post later on as I have experience with both now, and both have advantages and disadvantages.

DDF Files are not only unlike any other standard file format, they are also somewhat complicated when it comes to subfolders. In order to ease the work of creating .ddf files a bit, I made a small command line tool called "MakeDDF" which allows you to specify a directory and it will create the .ddf file that contains all the files in that directory. The resulting file can then be passed to makecab /f to create your .wsp file a bit more painless than useful.

It's freeware and requires the .net Framework 2.0, get it from here.