Moving Forward

Recently there has been a huge surge in event-driven servers. With the introduction and wide-spread adoption of Node.js as a Javascript based application server, and nginx, a HTTP proxying server one has to wonder what it is about event-driven architecture that works so well. These servers are touted as literal silver bullets for devops, promising massive gains in performance and concurrency with no changes in hardware, and amazingly enough for a lot of workloads they do indeed deliver.

Let’s take a closer look at event-driven architecture, how it’s different than traditional concurrency models, and what the future of these servers might look like.

The Old Way

Traditionally servers like Apache have used the single child per connection model. When a user connects to the server a child process is spawned, and handles the connection. Each connection gets a separate thread and child, and as the request is processed data is returned. As the request blocks on things like database reads and web service requests the child process waits.

This works pretty well for small workloads, but it really doesn’t scale well. The going gets tough when the number of requests gets too large. Apache will quickly hit the maximum number of child processes, and everything gets slow. Each request has its own thread, and when using PHP the amount of memory required by each process can be quite large. The typical PHP runtime can take up to 64MB.

There’s also a number of reliability problems associated with this model. With a misconfigured server it’s super easy to launch a denial of service attack against Apache. A large number of simultaneous requests can quickly exhaust the available resources, and typical workloads can face really serious bottlenecks.

The fact of the matter is that operating systems aren’t designed to handle workloads associated with web traffic. The traditional threading model assumes that a small number of intensive operations will be required for an application to run. Linux was designed with the idea of a handful of users executing multithreaded programs that juggled simple operations, such as background file writing and UI presentation. What it wasn’t designed to do was handle thousands of simultaneous connections, where the constraints don’t come from the system itself, but from waiting for database queries to return and remote procedure calls to execute.

Forking and threading are pretty heavy processes, creating threads creates overhead, and necessitates the allocation of an entirely new stack and execution thread for each child. Additionally the context swaps are pretty brutal, and there’s an open question of whether the CPU scheduling model fits well with what a typical web-server workload looks like.

All of this basically sums to the fact that operating systems don’t provide, out of the box, an abstraction that makes sense for handling highly concurrent workloads. We realized this pretty quickly once the internet started taking off, and started looking for a solution.

The New Way

Recognizing that the observed workloads are dramatically different than what was expected, a solution was proposed. The observation was made that the web workload consists of a lot of waiting. Apache, despite spawning a bunch of child processes, consuming gobs of memory, typically just sits around waiting for other tasks to complete. This observation led to a lot of head scratching, and someone had an interesting idea.

Since so much of the web workload consisted of waiting, it was proposed that we abandon the idea of child processes all together. Instead of spawning a thread for each request, all the requests would be managed by a single thread, and this thread would be called the event loop. This event loop would gracefully pop between all the active connections, and fire off asynchronous requests to storage and database servers, and when these requests return additional events would be popped onto the stack to be handled.

This is actually a really cool idea, we solve a lot of problems. All of the sudden the number of concurrent requests handled isn’t bounded as tightly. Sure, there’s some overhead in maintaining a list of open TCP connections, but memory requirements aren’t ballooning out of control anymore because so much of the runtime is now shared.

Node.js and Nginx both use this approach to build applications that scale to a super large number of connections. Everything happens inside an event loop, and multiple connections are handled gracefully.

Limitations

Event loops are not all roses and butterflies however. Looking specifically at Node.js there are some thrones as well. The most glaring omission from Node.js is a multithreaded implementation. It seems like event loop techniques are uniquely suited to be made multithreaded. The intuitive thought is that since events are pretty independent of each other, it shouldn’t be difficult to parallelize.

Theoretically that’s true, but there’s some technical reasons Node hasn’t become multithreaded, as well as a interesting argument I’ll explore in a second. Node is based off of V8, developed by Google. V8 is a high-performance Javascript engine, and it works remarkably well, but it was not designed to be multithreaded. Javascript executes on a single thread and this makes a lot of sense in the Chrome browser. Adding multithreading would be pretty tough, the architecture just wasn’t designed with it in mind.

What does the future look like?

So there’s also a really interesting argument against adding threading to Node. With the evolution of nginx as a reverse proxy server for HTTP, allowing for distribution of loads across many separate running instances, the authors of Node would tell you that the best way to multithread Node is to fork into separate processes.

At first glance this might seem like an argument constructed to explain away an implementation defect, but I think there’s a much more interesting story here. They really are advocating that a logical server should be a program that runs best on single core machine, with a small memory footprint, and well-defined limitations that allow for predictable performance. In contrast, Apache originally tried to manage concurrency and threading within the process, gobbling up all available resources. In Node, we don’t even bother, eschewing complexity in return for a really elegant, scalable implementation.

Event-driven architectures are a large step away from threads being the basic unit of concurrency, to a model where a CPU itself serves as the basic unit. Predictions of CPUs with hundreds of cores becoming commonplace keep rolling in, and tossing a bunch of cores onto a die is really one of the best ways to keep Moore’s law alive into the coming decade.

Interestingly enough cloud-based computing platforms sell units of computation that match these requirements. It’s obvious that a single cloud instance is well suited to running a single Node.Js server, and that scaling horizontally should involve spinning up identical servers with separate Node instances, and load balancing.

Web workloads are really different than that of typical user applications. They require systems to support a high number of concurrent users with relatively simple CPU and memory requirements. They also are highly compartmentalized. We’re lucky that the web workload requirements fit so nicely with the capabilities of large distributed server farms. They are mostly composed of high numbers of completely orthogonal requests, with very loose concurrency models.

A New Operating System?

I think that the event-driven model is here to stay. It’s part of a larger trend to address the impedance mismatch between existing servers and the demand of the web workload, and to break apart large monolithic servers (Apache) into small scalable pieces (nginx, Node.js, and fastphp servers). The next big shift I think will be a recombination of these pieces. By breaking them apart we allow for an exploration of what works, by combining them together, along different boundaries, we can build high-performance servers that shed some of the overhead introduced by ill-suited legacy systems.

Diverging from the topic of event-driven servers, lets take a look at what the future of servers that handle web workloads looks like. If we settle on a virtual CPU as the basic unit of concurrency, with a single Node.js instance running on each virtual server, I think the next obvious place for optimization is the operating system itself. It’s obvious that the threading and concurrency abstractions presented by modern operations systems are mismatched to the demands of the web workload, and tighter integration of the server with the kernel could yield even better performance by removing impedance caused by inefficiencies in the system call interfaces.

One could imagine a streamlined operating system that eliminates much of the overhead, and contains additional tweaks to file and network drivers to achieve even better performance. With paravirtualization we’re starting to see the emergence of a common hardware interface for the virtual kernels of the future that allows for much of the complexity of the operating system to be removed. This paves the way for development of microkernels, highly tuned to run a Node-like server.

Good news, I’ve moved everything to a LEMP based stack!

Previously this blog, as well as a number of small personal projects had been hosted using the downright terrible shared hosting service provided by HostGator. I’ve never been a fan of HostGator, they treat their customers poorly and I’ve had numerous support incidents while hosting my site with them. It’s been clear for a long time that I needed a change.

With everyone moving to new technology like Node.js and Nginx it only makes sense that I should move to a virtualized server. My next project more than most likely will involve utilizing some of these new technologies, and hosting non web-based services. I took some time and evaluated a lot of the VPS solutions out there currently and the word on the street is that Linode offers some of the best service, features, and reliability out there. Cheaper solutions exist, but for $20 you essentially get a complete self-managed server.

The Scary World of Unmanaged Hosting

The downside of a Linode instance is that the management of your site is suddenly in your own hands, and you are responsible for the security and maintenance of your server. This really isn’t a problem in modern-day systems, Linux is pretty secure out of the box, and servers are designed with a minimal footprint.

Why Linode?

Linode is not the cheapest VPS out there. You can find better deals, but what you won’t be able to find is a better community and collection of technical information. They have done a wonderful job providing ample documentation to get up and running, and cover all the common questions that one has when starting to host their own server in the real world.

The people behind Linode know their technology. Linode is built on top of Xen, the industry leading virtualization technology you’ll find powering many popular cloud services (EC2, Rackspace, etc) with a really nice web-based control panel for managing instances.

Additionally for development you’ll find a number of prebuilt StackScripts. These are really handy for quickly trying out a new technology. Allowing for user-submitted StackScripts makes it easy to try out new technologies if you have a spare node. I didn’t use a StackScript, only because installing the tools by hand generally gives you a pretty good feel for where they are located and how they are configured.

Let’s Get Started!

So let’s get started. I really didn’t do anything novel here, so I’ll be concise:

• Order a Linode – I ordered the $20 a month Linode, it should work for now. I set it up with Ubuntu 11.10 out of the box. It was set up instantly and with a few quick clicks I had a server up and running with SSH access.
• Install the Stack – This wonderful tutorial documents the process really well. I installed nginx, PHP, and mySQL from apt-get, unless you’re planning on doing devel work on these tools I’m not sure what advantages there are to compiling from source.
• Backup and Move WordPress – Surprisingly moving a large application like WordPress is surprisingly simple. I took the easy approach and exported a database copy in the form of an SQL script to recreate the database, and did a file-by-file copy of the WordPress site. I uploaded everything using SFTP (FTP is kind of old school- I wouldn’t recommend it) and quickly was up and running.
• Update the DNS Records – Linode offers a complete DNS solution, and provides nameservers for hosting domains. Surprisingly, by far the most time intensive part of this process was waiting for GoDaddy’s domain configuration page to load. I’m planning on moving my domain away from GoDaddy, and this has added some fuel to the fire.

And the results…

I had always suspected that HostGator had oversold its service to a unacceptable degree, I often experienced phantom delays and timeouts, with their support team being unable to turn up any problems. After switching I’ve noticed that the response time for my site has improved significantly, and the variance has also fallen. This means that visitors will get a much better experience, which makes me happy.

I’ve already started looking at new projects now that I have a nice permanent piece of computation hooked up with a large pipe to the internet at large. I would recommend something like this to any developer who needs more capabilities than shared hosting can provide. While there’s a slight overhead required to manage and update services, I think the flexibility and performance you gain are totally worth the effort.

Mounting Linux directories remotely using SFTP is incredibly handy. It allows you to easily move files back and forth between systems, and easily test and develop websites and other hosted services. It used to be super-easy to setup, MacFuse was the de facto way to install things, but has recently had some growing pains when it comes to being supported in Mac OS Lion. It is no longer actively maintained by Google and just doesn’t work very well.

There’s been a number of groups that have picked up the gauntlet and continued support for MacFuse, but instructions for installation by the end user aren’t really clear. MacFuse is designed to be a module to build file systems on top of so it’s generally not documented too well.

After a little bit of fumbling, I’ve found a procedure that works.

• Install everything with MacPorts – You’ll need MacPorts installed for this to work, the open-source package management system for Mac OS.

  We’ll be installing Fuse4X, which is a continued effort to develop on the MacFuse codebase.

  sudo port install fuse4x sshfs
  

• Make a mount point and test it out – Go ahead and create a directory in your home directory, and try to create a sshfs mount.

  mkdir ~/remote
  sshfs andrew@someserver.com:/home/andrew/ ~/remote -oauto_cache,reconnect,defer_permissions,negative_vncache
  cd ~/remote; ls
  

  If all goes well this should simply complete, substituting your username and server hostname in above. I recommend having SSH public key authentication turned on, else you’ll have to enter a password every time you’d like to mount this volume.

  You’ll be able to browse the directory in Finder just like a normal directory, however it appears as a volume within a directory, and Finder masks the file name. No worries though, it’ll still work for applications that reference the path.

• Make it happen every time your login – Finally we can use a login hook to make this happen every time. Toss the mount command in a shell script:

  #!/bin/bash
  sshfs andrew@someserver.com:/home/andrew/ ~/remote -oauto_cache,reconnect,defer_permissions,negative_vncache
  

  Chmod the file 777 and run the following command to add the script to your login hook:

  sudo defaults write com.apple.loginwindow LoginHook /path/to/script
  

Now every time you login, as long as you’re connected to the internet, you’ll have access to your files!

Recently while trying to check out an SVN repo via unsigned security certificate over https, hosted using the Windows-based VisualSVN Server I discovered an annoyance with the newer versions of Ubuntu.

svn: Commit failed (details follow):
svn: OPTIONS of 'https://HOSTNAME/svn/repo': SSL handshake failed: SSL error: Key usage violation in certificate has been detected. (https://HOSTNAME)

The bug only appeared after upgrading to Ubuntu 11.10. After doing some investigation I determined that the bug was the result of switching from using OpenSSL in previous versions of neon, the tool used for HTTP support in Subversion, built for Ubuntu, to the newer GnuTLS SSL libraries due to a more liberal LGPL license.

GnuTLS is considered to be less mature, and has a much stricter policy when it comes to key usage flags being set correctly in certificates.

The fix for this bug is luckily fairly straightforward. A version of libneon exists with OpenSSL support in Ubuntu, by simply removing the GnuTLS version and creating a symbolic link everything will begin to work again.

sudo mv /usr/lib/libneon-gnutls.so.27 /usr/lib/libneon-gnutls.so.27.old
sudo ln -s /usr/lib/libneon.so.27 /usr/lib/libneon-gnutls.so.27