Can we build a better console.log?

For a lot of Front End JavaScript work, our browser has become the de-facto IDE thanks to powerful built-in tools in Firefox, Chrome or Edge. However, one area that has seen little improvement over the years is the console.log function.

Nowadays, I might have 8 or 9 different modules in my webpage that all output debug logs when running non-minified versions to help debugging. Additionally, it is also my REPL for some ad-hoc coding. This results in an avalanche of messages that are hard to differentiate. There are a few ways to make things stand out. console.info, console.warn and console.error are highlighted in different ways:
cnsolelevel

Additionaly, there’s console.group/.groupEnd but that requires you to wrap all calls inside calls to .group(name) and .groupEnd()
onsolegroup

The problem is that there is no concept of scope inherent to the logger. For example, it would be useful to create either a scoped logger or pass in a scope name:

console.logScoped("Foo", "My Log Message");
console.warnScoped("Foo", "A Warning, oh noes!");

var scoped = console.createScope("Foo");
scoped.log("My Log Message");
scoped.warn("A Warning, oh noes!");

This would allow Browsers to create different sinks for their logs, e.g., I could have different tabs:
scopes
From there, we can even think about stuff like “Write all log messages with this scope to a file” and then I could browse the site, test out functionality and then check the logfile afterwards.

Of course, there are a few issues to solve (Do we support child scopes? Would we expose sinks/output redirection via some sort of API?), but I think that it’s time to have a look at how to turn the console.log mechanism from printf-style debugging into something a lot closer to a dev logging facility.

Thoughts on ORMs, 2016 Edition

This is a bit of a follow up to my 3 year old post about LINQ 2 SQL and my even older 2011 Thoughts on ORM post. I’ve changed my position several times over the last three years as I learned about issues with different approaches. TL;DR is that for reads, I’d prefer handwritten SQL and Dapper for .Net as a mapper to avoid having to deal with DataReaders. For Inserts and Updates, I’m a bit torn.

I still think that L2S is a vastly better ORM than Entity Framework if you’re OK with solely targeting MS SQL Server, but once you go into even slightly more complex scenarios, you’ll run into issues like the dreaded SELECT N+1 far too easily. This is especially true if you pass around entities through layers of code, because now virtually any part of your code (incl. your View Models or serialization infrastructure) might make a ton of additional SQL calls.

The main problem here isn’t so much detecting the issue (Tools like MiniProfiler or L2S Prof make that easy) – it’s that fixing the issue can result in a massive code refactor. You’d have to break up your EntitySets and potentially create new business objects, which then require a bunch of further refactorings.

My strategy has been this for the past two years:

  1. All Database Code lives in a Repository Class
  2. The Repository exposes Business-objects
  3. After the Repository returns a value, no further DB Queries happen without the repository being called again

All Database Code lives in a Repository Class

I have one or more classes that end in Repository and it’s these classes that implement the database logic. If I need to call the database from my MVC Controller, I need to call a repository. That way, any database queries live in one place, and I can optimize the heck out of calls as long as the inputs and outputs stay the same. I can also add a caching layer right there.

The Repository exposes Business-objects

If I have a specialized business type (say AccountWithBalances), then that’s what the repository exposes. I can write a complex SQL Query that joins a bunch of tables (to get the account balances) and optimize it as much as I want. There are scenarios where I might have multiple repositories (e.g., AccountRepository and TransactionsRepository), in which case I need to make a judgement call: Should I add a “Cross-Entity” method in one of the repositories, or should I go one level higher into a service-layer (which could be the MVC Controller) to orchestrate?

After the Repository returns a value, no further DB Queries happen without the repository being called again

But regardless how I decide on where the AccountWithBalances Getter-method should live, the one thing that I’m not going to do is exposing a hot database object. Sure, it sounds convenient to get an Account and then just do var balance = acc.Transactions.Sum(t => t.Amount); but that will lead to 1am debugging sessions because your application broke down once more than two people hit it at the same time.

At the end, long term maintainability suffers greatly otherwise. It seems that it’s more productive (and it is for simple apps), but once you get a grip on T-SQL you’re writing your SQL Queries anyway, and now you don’t have to worry about inefficient SQL because you can look at the Query Execution Plan and tweak. You’re also not blocked from using optimized features like MERGE, hierarchyid or Windowing Functions. It’s like going from MS SQL Server Lite to the real thing.

So raw ADO.net? Nah. The problem with that is that after you wrote an awesome query, you now have to deal with SqlDataReaders which is not pleasant. Now, if you were using LINQ 2 SQL, it would map to business objects for you, but it’s slow. And I mean prohibitively so. I’ve had an app that queried the database in 200ms, but then took 18 seconds to map that to .net objects. That’s why I started using Dapper (Disclaimer: I work for Stack Overflow, but I used Dapper before I did). It doesn’t generate SQL, but it handles parameters for me and it does the mapping, pretty fast actually.

If you know T-SQL, this is the way I’d recommend going because the long-term maintainability is worth it. And if you don’t know T-SQL, I recommend taking a week or so to learn the basics, because long-term it’s in your best interest to know SQL.

But what about Insert, Update and Delete?

Another thing that requires you to use raw SQL is Deletes. E.g., “delete all accounts who haven’t visited the site in 30 days” can be expressed in SQL, but with an ORM you can fall into the trap of first fetching all those rows and then deleting them one by one, which is just nasty.

But where it gets really complicated is when it comes to foreign key relationships. This is actually a discussion we had internally at Stack Overflow, with good points on either side. Let’s say you have a Bank Account, and you add a new Transaction that also has a new Payee (the person charging you money, e.g., your Mortgage company). The schema would be designed to have a Payees table (Id, Name) and a Transactions table (Id, Date, PayeeId, Amount) with a Foreign Key between Transactions.PayeeId and Payees.Id. In our example, we would have to insert the new Payee first, get their Id and then create a Transaction with that Id. In SQL, I would probably write a Sproc for this so that I can use a Merge statement:

CREATE PROCEDURE dbo.InsertTransaction
	@PayeeName nvarchar(100),
	@Amount decimal(19,4)
AS
BEGIN
    SET NOCOUNT ON;

    DECLARE @payeeIdTable TABLE (Id INTEGER)
    DECLARE @payeeId INT = NULL

    MERGE dbo.Payees AS TGT
        USING (VALUES (@PayeeName)) AS SRC (Name)
        ON (TGT.Name = SRC.Name)
    WHEN NOT MATCHED BY TARGET THEN
        INSERT(Name) VALUES (SRC.Name)
    WHEN MATCHED THEN
        UPDATE SET @payeeId = TGT.Id
    OUTPUT Inserted.Id INTO @payeeIdTable
    ;

    IF @payeeId IS NULL
        SET @payeeId = (SELECT TOP 1 Id FROM @payeeIdTable)

    INSERT INTO dbo.Transactions (Date, PayeeId, Amount)
    VALUES (GETDATE(), @payeeId, @Amount)
END
GO

The problem is that once you have several Foreign Key relationships, possibly even nested, this can quickly become really complex and hard to get right. In this case, there might be a good reason to use an ORM because built-in object tracking makes this a lot simpler. But this is the perspective of someone who never had performance problems doing INSERT or UPDATE, but plenty of problems doing SELECTs. If your application is INSERT-heavy, hand rolled SQL is the way to go – if only just because all these dynamic SQL queries generated by an ORM don’t play well with SQL Server’s Query Plan Cache (which is another reason to consider stored procedures (sprocs) for Write-heavy applications)

Concluding this, there are good reasons to use ORMs for developer productivity (especially if you don’t know T-SQL well) but for me, I’ll never touch an ORM for SELECTs again if I can avoid it.

When nslookup works but you can’t ping it, NetBIOS may be missing

I have a custom DNS Server (running Linux) and I also have a server running Linux (“MyServer”). The DNS Server has an entry in /etc/hosts for MyServer.

On my Windows machines, I can nslookup MyServer and get the IP back, but when I try to access the machine through ping or any of the services it offers, the name doesn’t resolve. Access via the IP Address works fine though.

What’s interesting is that if I add a dot at the end (ping MyServer.) then it suddenly works. What’s happening?!

What’s happening is that Windows doesn’t use DNS but NetBIOS for simple name resolution. nslookup talks to the DNS Server, but anything else doesn’t use DNS.

The trick was to install Samba on MyServer, because it includes a NetBIOS Server (nmbd). On Ubuntu 16.04, just running sudo apt-get install Samba installs and auto-starts the service, and from that moment on my Windows machines could access it without issue.

There are ways to not use NetBIOS, but I didn’t want to make changes on every Windows client (since I’m using a Domain), so this was the simplest solution I could find. I still needed entries in my DNS Server so that Mac OS X can resolve it.

Why Feature Toggles enable (more) frequent deployments

As Nick Craver explained in his blog posting about our deployment process, we deploy Stack Overflow to production 5-10 times a day. Apart from the surrounding tooling (automated builds, one-click deploys, etc.) one of the reasons that is possible is because the master-branch rarely ever stays stale – we don’t feature branch a lot. That makes for few merge-nightmares or scenarios where suddenly a huge feature gets dropped into the codebase all at once.

The thing that made the whole “commit early, commit often” principle click for me was how easy it is to add new feature toggles to Stack Overlow. Feature Toggles (or Feature Flags), as described by Martin Fowler make the application [use] these toggles in order to decide whether or not to show the new feature.

The Stack Overflow code base contains a Site Settings class with (as of right now) 1302 individual settings. Some of these are slight behavior changes for different sites (all 150+ Q&A sites run off the same code base), but a lot of them are feature toggles. When the new IME Editor was built, I added another feature toggle to make it only active on a few sites. That way, any huge issue would’ve been localized to a few sites rather than breaking all Stack Exchange sites.

Feature toggles allow for a half-finished feature to live in master and to be deployed to production – in fact, I can intentionally do that if I want to test it with a limited group of users or have our community team try it before the feature gets released network-wide. (This is how the “cancel misclicked flags” feature was rolled out). But most importantly, it allows for changes to constantly go live. If there is any unintended side-effects, we notice it faster and have an easier time locating it as the relative changeset is small. Compare that to some massive merge that might introduce a whole bunch of issues all at once.

For feature toggles to work, it must be easy to add new ones. When you start out with a new project and want to add your first feature toggle, it may be tempting to just add that one new toggle, but if the code base grows bigger, having an easy mechanism really pays off. Let me show you how I add a new feature toggle to Stack Overflow:

[SiteSettings]
public partial class SiteSettings
{
    // ... other properties ...

    [DefaultValue(false)]
    [Description("Enable to allow users to retract their flags on Questions, Answers and Teams.")]
    [AvailableInJavascript]
    public bool AllowRetractingFlags { get; private set; }
}

When I recompile and run the application, I can go to a developer page and view/edit the setting:

RetractFlags

Anywhere in my code, I can gate code behind an if (SiteSettings.AllowRetractingFlags) check, and I can even use that in JavaScript if I decorate it with the [AvailableInJavascript] attribute (Néstor Soriano added that feature recently, and I don’t want to miss it anymore).

Note what I did not have to do: I did not need to create any sort of admin UI, I did not need to write some CRUD logic to persist the setting in the database, I did not need to update some Javascript somewhere. All I had to do was to add a new property with some attributes to a class and recompile. What’s even better is that I can use other datatypes than bool – our code supports at least strings and ints as well, and it is possible to add custom logic to serialize/deserialize complex objects into a string. For example, my site setting can be a semi-colon separated list of ints that is entered as 1;4;63;543 on the site, but comes back as an int-array of [1,4,63,543] in both C# and JavaScript.

I wasn’t around when that feature was built and don’t know how much effort it took, but it was totally worth building it. If I don’t want a feature to be available, I just put it behind a setting without having to dedicate a bunch of time to wire up the management of the new setting.

Feature Toggles. Use them liberally, by making it easy to add new ones.

Dell XPS 15 9550 (2016), 6 weeks in

One of the perks of working for Stack Overflow is that you get to choose your own work computer. I decided to go with a 2016 Dell XPS 15 (9550) (not to be confused with the earlier XPS 15, which had the model number 9530) and settled on this configuration:

Dell XPS 15 9550

One of the big selling features of the XPS 15 is a gorgeous 4K (3840×2160) display – why didn’t I get that one? Simple: At 15″, I’d have to use Windows display scaling to make stuff not too tiny. Display scaling is a lot better in Windows 10 than it was in 7 or 8, but it’s still not great. I have used display scaling on Mac OS X, and on Windows it’s still just a giant crutch. So I decided on the normal 1080p display, and I like it a lot. It’s bright, it’s IPS and thus doesn’t suffer from colors being all weird when viewed from the side, and it doesn’t kill the battery nearly as much as the 4K screen.

At home, I’m also running two 2560×1440 displays (One Dell U2715H connected via a USB-C-to-DisplayPort adapter, and one Dell U2515H connected via HDMI).

The external display situation is a bit weird at the moment (Late July 2016). The XPS 15 has a Thunderbolt port, so supporting 2x 4K monitors at 60Hz each should be possible using the TB15 thunderbolt dock. The problem is that the Thunderbolt dock doesn’t work properly and is currently not sold (Dell might have a fix sometime in August). There have been 3 or 4 Thunderbolt BIOS updates over the last few weeks, but as it stands right now, unless you have a Thunderbolt display the port doesn’t do much.

It can be used as a normal USB-C port and drive a single 4K screen at 60 Hz with the DisplayPort adapter cable, and that works fine. Dell does have a USB-C dock (Dell WD15) which has HDMI and Display Port, but unlike the built-in HDMI port, it cannot drive a 2560×1440 screen over HDMI.

That basically means that unless you’re only using 1080p screens, there isn’t a good dock out there and you’re better off connecting external screens directly to the laptop. I have tried daisy chaining the two 2560×1440 displays (2nd display into 1st using display port cable, 1st into USB-C port using DP-USBC cable) and that worked fine

The keyboard is surprisingly good. It’s still a laptop keyboard, but it’s normal sized keys with enough travel to not feel strange. Because it’s a 15″ laptop and the keyboard is towards the screen though, my arm is resting on the bottom edge of the laptop which isn’t the most comfortable position. It’s one of the sacrifices to be made. The covering is some rubbery material that feels good, but finger-stains are readily visible.

The touchpad is pretty good, as close to a Macbook touchpad as I’ve encountered so far, although it doesn’t have the glass cover that makes the Macbook feel frictionless. It was definitely one of the reasons I wanted a Dell XPS laptop, because the touchpad is one of the main reasons to buy an Apple laptop, and I feel that there’s no need to regret not getting one.

The battery runtime is pretty good. There are two Battery choices: a 56 WHr and an 84 WHr. The battery situation is a bit interesting: Basically the lower-end models come with a 2.5″ S-ATA hard drive and a 32 GB M.2 SSD:

urRSfVS

The higher-end models come with only a M.2 SSD and either a 56 or 84 WHr battery. In case of the 84 WHr battery, it takes up the space that the 2.5″ hard drive would take, and so it’s not possible to add an additional 2.5″ hard drive to an XPS 15 with a 84 WHr battery (in case you were thinking of adding a second hard drive for data).

In theory, it is possible to add a 2.5″ drive to a model with the 56 WHr battery, but no mounting hardware is included and it seems Dell doesn’t sell it individually. So if you’re really thinking of putting in two hard drives (say, a dream configuration of a 1 TB Samsung SM961 and a 4 TB Samsung 850 EVO), you’d have to buy the XPS 15 in a configuration that includes a hard drive and swap them out.

Battery runtime with the 84 WHr battery is pretty good – Dell makes some lofty claims of 17 hour runtime that of course aren’t reached in real world use, but I get at least 6 hours out if my normal use (WiFi enabled, nothing connected to USB, display to about 60% brightness, Visual Studio, SQL Server, IIS, no video streaming). The i7 uses a bit more power than the Core i5-6300HQ that is also offered, but not much more since they are both Quad-Cores, the i7 basically just adding Hyper-Threading. (There is also a model with a Core i3-6100H CPU but honestly, I’d get at least the i5). The 4K Infinity Display apparently really drains the battery from what others have said

Overall, after using the laptop for about 5 weeks both as a stationary computer (external displays, keyboard and mouse) and as a portable, I’m highly satisfied with it. Thunderbolt woes aside, it’s insides are up-to-date with a Skylake-CPU, a PCI Express NVMe SSD, a really good IPS display, pretty much the best Wintel touchpad out there and USB-C. Literally the only other Windows laptop I would look at is the XPS 13 in for something a bit smaller. With laptops like these, comparing it to the Macbook Pro is always a hot topic, despite the (as of July 31) MBP’s really outdated hardware. For me, it boiled down to the question if I needed to run Mac OS X, and since I have two Macs already (Late 2010 Mac Pro, 2015 Retina Macbook) the answer was “no”, and thus the Dell XPS 15 won out. So far, I do not regret that decision.

CoreI7
RAM

(Note that BitLocker is enabled, which may skew results downward a bit)
(Note that BitLocker is enabled, which may skew results downward a bit)

cinebench

Handling IME events in JavaScript

Stack Overflow has been expanding past the English-speaking community for a while, and with the launch of both a Japanese version of Stack Overflow and a Japanese Language Stack Exchange (for English speakers interested in learning Japanese) we now have people using IME input regularly.

For those unfamiliar with IME (like I was a week ago), it’s an input help where you compose words with the help of the operating system:
IME
In this clip, I’m using the cursor keys to go up/down through the suggestions list, and I can use the Enter key to select a suggestion.

The problem here is that doing this actually sends keyup and keydown events, and so does pressing Enter. Interestingly enough, IME does not send keypress events. Since Enter also submits Comments on Stack Overflow, the issue was that selecting an IME suggestion also submits the comment, which was hugely disruptive when writing Japanese.

Browsers these days emit Events for IME composition, which allowed us to handle this properly now. There are three events: compositionstart, compositionupdate and compositionend.

Of course, different browsers handle these events slightly differently (especially compositionupdate), and also behave differently in how they treat keyboard events.

  • Internet Explorer 11, Firefox and Safari emit a keyup event after compositionend
  • Chrome and Edge do not emit a keyup event after compositionend
  • Safari additionally emits a keydown event (event.which is 229)

So the fix is relatively simple: When you’re composing a Word, we should not have Enter submit the form. The tricky part was really just to find out when you’re done composing, which requires swallowing the keyup event that follows compositionend on browsers that emit it, without requiring people on browsers that do not emit the event to press Enter an additional time.

The code that I ended up writing uses two boolean variables to keep track if we’re currently composing, and if composition just ended. In the latter case, we swallow the next keyup event unless there’s a keydown event first, and only if that keydown event is not Safari’s 229. That’s a lot of if’s, but so far it seems to work as expected.

submitFormOnEnterPress: function ($form) {
    var $txt = $form.find('textarea');
    var isComposing = false; // IME Composing going on
    var hasCompositionJustEnded = false; // Used to swallow keyup event related to compositionend

    $txt.keyup(function(event) {
        if (isComposing || hasCompositionJustEnded) {
            // IME composing fires keydown/keyup events
            hasCompositionJustEnded = false;
            return;
        }

        if (event.which === 13) {
            $form.submit();
        }
    });

    $txt.on("compositionstart",
            function(event) {
                isComposing = true;
            })
        .on("compositionend",
            function(event) {
                isComposing = false;
                // some browsers (IE, Firefox, Safari) send a keyup event after
                //  compositionend, some (Chrome, Edge) don't. This is to swallow
                // the next keyup event, unless a keydown event happens first
                hasCompositionJustEnded = true;
            })
        .on("keydown",
            function(event) {
                // Safari on OS X may send a keydown of 229 after compositionend
                if (event.which !== 229) {
                    hasCompositionJustEnded = false;
                }
            });
},

Here’s a jsfiddle to see the keyboard events that are emitted.

.net Framework 4.6.2 adds support to sign XML Documents using RSA-SHA256

One of the hidden useful gems in the .net Framework is the System.Security.Cryptography.Xml.SignedXml class, which allows to sign XML documents, and validate the signature of signed XML documents.

In the process of implementing both a SAML 2.0 Service Provider library and an Identity Provider, I found that RSA-SHA256 signatures are common, but not straight forward. Validating them is relatively easy, add a reference to System.Deployment and run this on app startup:

CryptoConfig.AddAlgorithm(
    typeof(RSAPKCS1SHA256SignatureDescription),
    "http://www.w3.org/2001/04/xmldsig-more#rsa-sha256");

However, signing documents with a RSA-SHA256 private key yields a NotSupportedException when calling SignedXml.ComputeSignature(). Turns out that only .net Framework 4.6.2 will add support for the SHA2-family:

X509 Certificates Now Support FIPS 186-3 DSA

The .NET Framework 4.6.2 adds support for DSA (Digital Signature Algorithm) X509 certificates whose keys exceed the FIPS 186-2 limit of 1024-bit.

In addition to supporting the larger key sizes of FIPS 186-3, the .NET Framework 4.6.2 allows computing signatures with the SHA-2 family of hash algorithms (SHA256, SHA384, and SHA512). The FIPS 186-3 support is provided by the new DSACng class.

Keeping in line with recent changes to RSA (.NET Framework 4.6) and ECDsa (.NET Framework 4.6.1), the DSA abstract base class has additional methods to allow callers to make use of this functionality without casting.

After updating my system to the 4.6.2 preview, signing XML documents works flawlessly:

// exported is a byte[] that contains an exported cert incl. private key
var myCert = new X509Certificate2(exported);
var certPrivateKey = myCert.GetRSAPrivateKey();

var doc = new XmlDocument();
doc.LoadXml("<root><test1>Foo</test1><test2><bar baz=\"boom\">Real?</bar></test2></root>");

var signedXml = new SignedXml(doc);
signedXml.SigningKey = certPrivateKey;

Reference reference = new Reference();
reference.Uri = "";
XmlDsigEnvelopedSignatureTransform env = new XmlDsigEnvelopedSignatureTransform();
reference.AddTransform(env);
signedXml.AddReference(reference);

signedXml.ComputeSignature();
XmlElement xmlDigitalSignature = signedXml.GetXml();
doc.DocumentElement.AppendChild(doc.ImportNode(xmlDigitalSignature, true));

// doc is now a Signed XML document

Late 2015 PC Build

I’ve been saying for a while that the Mid 2010 Mac Pro was the best computer I’ve ever owned. The internal layout was just so great, swapping out hard drives or RAM was easy, no cables in the way, airflow well thought out, just awesome.

Unfortunately, being a Mac limits upgrades. While there are ways to upgrade the CPU, upgrading the video card requires some firmware flashing and trial and error. I had some luck buying a card to drive my 4K Monitor off MacVidCards, but the pricing for anything decent just didn’t work for me.

After 5 years, it was time to relegate my Mac Pro to be dedicated to Logic Pro X and Final Cut Pro X and go back to building a PC. I still had some leftover parts from my previous PC and ended up with this parts list for a bit less than $1000:

The pieces

(Disclaimer: Intel, nVidia and Gigabyte have been sponsors of my employer and are advertising with a game made by said employer. My choice these components was independent of that and mostly driven by price, availability and benchmarks from sites like Tom’s Hardware or Anandtech or whatever Google came up with.)

For the case, I had to concede that PC cases just can’t compete with the Mac Pro. The Corsair case looks nice from the front, has no obnoxious window in the side, is mostly toolless and has enough space. Check the YouTube review linked above. I went with a 650W power supply which is more than plenty to support the 65W CPU, 60W Graphics Card and all the other stuff. 80Plus Platinum for just under $100 is neat. It’s not modular, but modular PSUs usually result in me losing the extra cables anyway.

On the CPU side, Intel is a no-brainer these days for games. Skylake Core-CPUs just came out, are priced really well and are fast as heck. Since I don’t care about overclocking, the i5-6500 won out over the 6600 or 6600K simply because it’s a lot cheaper.

Going with that is a Z170 chipset board. The Z170XP-SLI isn’t very expensive, has USB 3.1 Type-C port, supports DDR4 and has a M.2 SSD slot – more on that later.

I have a collection of RAM sticks at home, mainly because whenever I upgrade RAM, the old doesn’t fit in any machine. This machine comes with yet another type of RAM, DDR4. How much you need is always up for discussion, I wouldn’t go below 8 GB these days, and I didn’t see a reason to get more than 16 GB. YMMV, but RAM is cheap enough to err on the side of more. Make sure to get a pair – the CPU uses dual channel memory controllers, which means that you should use two or four modules and if you use two, make sure to populate the two slots of the same color. I got DDR4-2133 memory which is technically the slowest, but the only speed the board supports without overclocking. A lot of marketing talks about support for faster DDR4 speeds, but in parentheses you usually see (O.C.). I’m not interested in overclocking, so I went with the speed that’s supported.

The OS drive is pure luxury overkill. Getting an SSD is a must these days, and normally I’d have gone with a Samsung 850 Pro – StorageReview can tell you why. This is still a S-ATA based SSD though. S-ATA was aimed at mechanical hard drives and is limited to 6 Gigabit/s. A 256 GB drive runs at about $130. On the other hand, the 950 Pro uses the M.2 slot and supports NVM Express (NVMe). It’s shaped like a stick of gum and sits directly on the motherboard – that’s why I went with this board. It’s ridiculously fast (2.5 GB/s compared to 550 MB/s on the 850 Pro), although IOPS are roughly the same. I’d not consider anything but Samsung these days as they make all components – flash chips, controller, finished product – and aren’t priced much differently than the competition.

The graphics card is the one component where I had to compromise. My number one choice would’ve been a Geforce GTX-970. However, these cards (and other 2nd generation Maxwell cards) suffer from coil whine, which is a high pitched noise when playing games. I didn’t want to take the risk, so I went with a first generation Maxwell-based GTX 750 Ti which are cheaper and will work well as a stopgap until coil whine is solved.

Whether or not you need a sound card is debatable these days. I had the X-Fi Titanium HD and love the cinch outputs to go to my amplifier.

I love my K70 keyboard because of the media keys (complete with a volume wheel) and because mechanical keyboards are just a must-have.

NVM Express and Windows

The thing with brand new hardware standards is that older operating systems don’t support it well. In my case, I was faced with the preference to run Windows 7 and the necessity to support a NVMe-based boot drive. There is a hotfix to add NVMe support to Windows 7, and Intel has a set of instructions. But at the end of the day, it was time to go to a newer version of Windows.

Windows 10 supports a NVMe boot drive natively, so no issues there.

Blurry Fonts in OS X when scaling external 4K Monitor

I have a 4K Monitor connected to my Mac Pro. However, since 4K on 28″ is a bit too small for my taste, I prefer to make use of OS X’s scaling options and run a 2560×1440 resolution.

As OS X is vector based internally, it is capable of pretending to run in that resolution but still output a native 4K image to my monitor – in other words, the output should be razor sharp and crisp.

However, when I looked at the options (System Preferences > Displays > Scaled), I noticed that all the non-native options were marked as (low resolution) and when selected, text looked blurry.

The trick is to hold down Option (Or “Alt” on non-Mac keyboards) and click scaled – this will unlock additional resolutions, including a 2560×1440 without the (low resolution) suffix that looks as it should be – crisp.

I don’t know why these don’t show up by default – I have a third-party graphics card (Geforce GT 640 from MacVidCards instead of the original, non-4K supporting Radeon 5770) and a third party monitor (Acer B286HK).

Building a NAS with OpenBSD

Over a recent long weekend, I’ve decided to build a small NAS for home use, mainly to have some of my data backed up and to have an archive of old stuff I don’t need all the time. Both of my Laptops have 256 GB SSDs, and while that’s usually enough, it’s good to have some extra headroom sitting around.

The idea was to:

  • Have a place to backup my stuff
  • Have a machine that can do BitTorrent downloads on its own
  • Have a machine that allows my to access big files from multiple other PCs
  • Have a machine that works as a local git server

The Hardware

I bought the motherboard and case a few years ago for something else, so I think better options are available now.

The desired setup:

  • Use the 128 GB SSD as the boot drive – because it’s mSATA it fits directly on the motherboard, and doesn’t take up space for mounting drives
  • Use the two 2.5″ 1 TB drives as a RAID 1 – that way, I’m protected against hard drive failure. Do note that RAID 1 is more an availability than a safety thing because viruses or accidential deletion of files isn’t something a RAID can help with
  • Use the one 3.5″ 3 TB drive as a big store for non-critical stuff, like backups of my Steam games or temporary BitTorrent files

The case doesn’t have much space for drives, even though the motherboard has plenty of S-ATA ports.

For the operating system, I went with OpenBSD 5.7 x64. I prefer OpenBSDs very minimalistic approach of offering a tiny base system, and then allowing me to add exactly the pieces of software that I need. I’m not going to give a full rundown of how OpenBSD works, because if you’re really interested you should definitely read Absolute OpenBSD.

Basic System Setup

Do setup a user during setup – in my case, I called him User.

My 128 GB SSD is partitioned as follows:

#                size           offset  fstype [fsize bsize  cpg]
  a:             2.0G               64  4.2BSD   2048 16384    1 # /
  b:             8.2G          4209024    swap                   # none
  c:           119.2G                0  unused                   
  d:             4.0G         21398592  4.2BSD   2048 16384    1 # /tmp
  e:            15.0G         29800544  4.2BSD   2048 16384    1 # /var
  f:             8.0G         61255840  4.2BSD   2048 16384    1 # /usr
  g:             2.0G         78027680  4.2BSD   2048 16384    1 # /usr/X11R6
  h:            15.0G         82220640  4.2BSD   2048 16384    1 # /usr/local
  i:             3.0G        113675936  4.2BSD   2048 16384    1 # /usr/src
  j:             3.0G        119957344  4.2BSD   2048 16384    1 # /usr/obj
  k:            59.0G        126238752  4.2BSD   2048 16384    1 # /home

The best setup varies on preference of course, in my case I stuck mostly to the OpenBSD defaults and only gave /usr/src and /usr/obj some extra space.

After the system boots up for the first time, add powerdown=YES to /etc/rc.shutdown. This turns off the machine when shutdown -h now is called. Do note that halt doesn’t seem to respect that, and needs to be invoked with halt -p. To my delight, pushing the power button on the case turns off the machine properly – hooray for working ACPI support!

The first thing before installing any software should be to follow -stable, recompiling the kernel, userland, and xenocara.

# cd /usr
# export CVSROOT=anoncvs@anoncvs.usa.openbsd.org:/cvs
# cvs -d$CVSROOT checkout -rOPENBSD_5_7 -P src ports xenocara

# cd /usr/src/sys/arch/amd64/conf
# config GENERIC.MP
# cd ../compile/GENERIC.MP
# make clean && make
# make install
# reboot

# rm -rf /usr/obj/*
# cd /usr/src
# make obj
# cd /usr/src/etc && env DESTDIR=/ make distrib-dirs
# cd /usr/src
# make build
# cd /usr/xenocara
# rm -rf /usr/xobj/*
# make bootstrap
# make obj
# make build
# reboot

This takes a long time, over an hour on this machine. After that, it’s time to do package setup

Add FETCH_PACKAGES=yes to /etc/mk.conf, and export PKG_PATH=ftp://ftp5.usa.openbsd.org/pub/OpenBSD/5.7/packages/amd64/to begin installing packages.

The OpenBSD packages and ports system is a bit interesting, because it seems that packages are built only once when a new OpenBSD version is released, and then never updated. You have to manually compile newer versions of software. That’s not that big of a deal, because with FETCH_PACKAGES enabled, the system will fetch packages if they are still the correct version and only build ports where needed.

Setting up a data drives, incl. RAID 1

I decided that my data drives should live under /var/netshared, so I created this and two subdirectories – data and glacier. I will set permissions later.

I have 2x 1 TB hard drives, from which I want to build a RAID 1. First, setup disklabels for both drives (disklabel -E sd0, then sd1), making sure that the partition type is RAID instead of the default 4.2BSD.

OpenBSD area: 0-1953525168; size: 931.5G; free: 0.0G
#                size           offset  fstype [fsize bsize  cpg]
  a:           931.5G                0    RAID                   
  c:           931.5G                0  unused

Then, run bioctl -c 1 -l sd0a,sd1a softraid0 to create the RAID. The -c 1 flag sets the RAID level (RAID 1 = mirroring), and -l (lowercase L) is a list of partitions that form the raid. The softraid0 at the end is an internal identifier – it must start with softraid. bioctl will then create a new device that will appear like a hard drive and can be used as such.

The actual device will be something like /dev/sd4. You need to run disklabel on the new device to create a partition, this time of the usual 4.2BSD type. In order to add it to /etc/fstab, you need to get the duid, which you can get by running disklabel sd4:

# /dev/rsd4c:
type: SCSI
disk: SCSI disk
label: SR RAID 1
duid: cc029b4fe2ac54dd

(I do note that using duids in fstab is optional, but I highly recommend it as it makes you independent of device name changes as long as the actual drive is the same)

Remember to run newfs /dev/sd4a to create a file system. OpenBSD will pick FFS for drives smaller than 1 TB, and FFS2 for drives bigger than 1 TB. Check man newfs for options.

Here’s how my fstab looks:

e8bd5e30aba4f036.b none swap sw
e8bd5e30aba4f036.a / ffs rw 1 1
e8bd5e30aba4f036.k /home ffs rw,nodev,nosuid 1 2
e8bd5e30aba4f036.d /tmp ffs rw,nodev,nosuid 1 2
e8bd5e30aba4f036.f /usr ffs rw,nodev 1 2
e8bd5e30aba4f036.g /usr/X11R6 ffs rw,nodev 1 2
e8bd5e30aba4f036.h /usr/local ffs rw,nodev 1 2
e8bd5e30aba4f036.j /usr/obj ffs rw,nodev,nosuid 1 2
e8bd5e30aba4f036.i /usr/src ffs rw,nodev,nosuid 1 2
e8bd5e30aba4f036.e /var ffs rw,nodev,nosuid 1 2
cc029b4fe2ac54dd.a /var/netshared/data ffs rw,nodev,nosuid,noexec,noatime 1 2
f4540651dabd448d.a /var/netshared/glacier ffs rw,nodev,nosuid,noexec,noatime 1 2

Notice the nosuid,noexec,noatime,nodev flags on the two data drives. This is just some precaution against malicious files, and noatime is just to reduce disk wear by a tiny fraction. Check the manpage of mount for more information.

Setting up a user

During OpenBSD Setup, a user should’ve been setup. If you decided not to, use useradd to create one now.

Create a group for access to the shared directories: groupadd netshared

Add the user to that group: user mod -G netshared User

Change owner and permissions:

chown -R User:netshared /var/netshared/* 
chmod -R 0770 /var/netshared/*

Note that the execution-bit is required to traverse directories, so chmod 0660 wouldn’t work as a permission mask. Since the file system is mounted noexec, it doesn’t matter anyways.

Installing Samba

Start by installing the samba port:

# cd /usr/ports/net/samba
# make install

Then, configure samba (thanks Pierre-Philipp Braun for the tip with sed):

cd /etc/samba/
mv smb.conf smb.conf.dist
sed '/^#/d; /^;/d; /^$/d;' smb.conf.dist > smb.conf
vi smb.conf

Here’s my smb.conf:

[global]
   workgroup = WORKGROUP
   server string = Samba Server
   security = user
   load printers = no
   log file = /var/log/samba/smbd.%m
   max log size = 50
   dns proxy = no
   printing = BSD
   unix extensions = no
   allow insecure wide links = no
[data]
   path = /var/netshared/data
   valid users = User
   writable = yes
   printable = no
[glacier]
   path = /var/netshared/glacier
   valid users = User
   writable = yes
   printable = no

If you want to give access to groups instead of individual users, prefix with an @-sign: valid users = @netshared

The manpage – man smb.conf – is very extensive. If you want to finetune permissions, take the time to browse through it.

To start samba on system startup, add this to /etc/rc.conf.local:

pkg_scripts="samba"
samba_flags=""

This should be it – start samba through /etc/rc.d/samba start and try accessing your new file shares!

Using the server as a git server

This isn’t really a NAS-specific, but git specific. If you want to install git on the server, cd /usr/ports/devel/git and make install.

Create or clone a bare repository on the NAS:

cd /var/netshared/data
mkdir myrepo.git
cd myrepo.git
git init --bare

Or clone an existing repository as a bare clone:

cd /var/netshared/data
git clone --bare https://github.com/mstum/faml.git

Then, on your machines, clone from that repository:
git clone \\nas\data\faml.git

This will automatically set up an origin remote on your local clone, so any changes you make on your laptop can be pushed to the server through git push.

Setting up a BitTorrent client

Install the port of transmission:

cd /usr/ports/net/transmission
make install

This will automatically create a _transmission user – add it to the netshared group:
user mod -G netshared _transmission

Create folders for BitTorrent:

mkdir /var/netshared/glacier/BitTorrent
mkdir /var/netshared/glacier/BitTorrent/incomplete
mkdir /var/netshared/glacier/BitTorrent/complete
mkdir /var/netshared/glacier/BitTorrent/watch
chown -R User:netshared /var/netshared/glacier/BitTorrent

Edit the /var/transmission/.config/transmission-daemon/settings.json file (if it doesn’t exist, run /etc/rc.d/transmission-daemon start and then stop it – changes to the file will be lost if you edit it while the daemon is running)
Important settings/changes:

"download-dir": "/var/netshared/glacier/BitTorrent/complete",
"incomplete-dir": "/var/netshared/glacier/BitTorrent/incomplete",
"incomplete-dir-enabled": true,
"rpc-whitelist": "127.0.0.1,192.168.1.*",
"rpc-whitelist-enabled": true,
"watch-dir": "/var/netshared/glacier/BitTorrent/watch",
"watch-dir-enabled": true

These settings make it so that any .torrent you drop into the watch directory immediately gets added and started. Downloads go into the incomplete directory while they are downloading, and are then moved to the complete directory afterwards.

rpc-whitelist is a comma-separated list of IPs that can remotely control transmission, so this should be limited to your local network. You can access the web UI on http://nas:9091/transmission/web which is pretty neat.

To auto-start transmission, edit your /etc/rc.conf.local and add transmission_daemon to the pkg_scripts. I recommend starting it before samba, so that samba gets shutdown before transmission. (OpenBSD stops services in the reverse order of startup).

Keeping up to date

Keeping OpenBSD up to date is described in the following -stable link above. Basically, CVS update all of src, ports, xenocara if needed, then recompile and reboot.

To check if your ports are up to date, you can run /usr/ports/infrastructure/bin/out-of-date, then cd into any ports and run make update.
Note that if you’ve installed a package, it’s safe to update it through make update in the ports directory – packages are really just precompiled ports, no “magic”.

Closing Remarks

This was really just a howto of how setup my NAS currently, aimed at people that already know OpenBSD. If you’re curious about a *NIX server and don’t mind spending some time to learn the system. I’m highly pleased with OpenBSD. The system is minimalist – there are not many moving parts by default – and really invites to understand stuff properly.

If you have a more sophisticated NAS setup, you may want to look at FreeNAS as well. Do note that the 8 GB minimum RAM requirement is not a joke – FreeNAS will install and seemingly run on 4 or even 2 GB, but random data loss is almost guaranteed to occur.