InfoQ

News

Interview: Dan Farino About MySpace’s Architecture

Posted by Abel Avram on Nov 19, 2008 03:57 AM

Community
.NET,
Architecture
Topics
.NET Framework ,
Enterprise Architecture ,
Performance & Scalability ,
Configuration Management
Tags
PowerShell ,
IIS ,
MySpace

In this interview taken by InfoQ’s Ryan Slobojan, Dan Farino, Chief Systems Architect at MySpace, talks about the system architecture and the challenges faced when building a very large online community. Because MySpace is built almost entirely on the .NET Framework, Dan explains how a .NET product scales on hundreds of servers.

Watch: Dan Farino About MySpace’s Architecture (28 min.)

In the beginning of this interview, Dan speaks about general challenges encountered and the solutions used to make a very large web site run smoothly. He enters into details of dealing with performance bottlenecks, database or system failures, the need for debugging and logging applications. He mentions having problems with database bottlenecks which were addressed by implementing a custom cache.

Dan also talks about the .NET platform used to support the MySpace’s web site. He says that .NET has scaled well for them serving millions of users from hundreds of servers. One of the problems mentioned is the garbage collector which was intermittently introducing a significant delay in web site’s response time. Administering hundreds of servers is a difficult and time consuming task. Dan says they are using Microsoft’s PowerShell from the time when the technology was still in research, code name Monad, and he is very happy with it.

Weird... by Branden Root Posted Nov 19, 2008 12:01 PM
Re: Weird... by Abel Avram Posted Nov 20, 2008 9:00 AM
Re: Weird... by Jason Gleason Posted Dec 10, 2008 11:35 AM
How do you test it? by Pavel Veller Posted Nov 21, 2008 2:36 PM
Re: How do you test it? by Andre Sobotovych Posted Nov 27, 2008 7:34 AM
  1. Back to top

    Weird...

    Nov 19, 2008 12:01 PM by Branden Root

    I could have sworn the site was written in Coldfusion, given that all the URLs are .cfm, and they even have fuseactions in them. I guess there's a .net middle layer. MySpace is comparable to Twitter in my mind - the site's really slow, doesn't really do much complicated actions, and has a tendency to break a lot. So, listen to this and learn what not to do ;)

  2. Back to top

    Re: Weird...

    Nov 20, 2008 9:00 AM by Abel Avram

    As Dan says in the interview, the site started on ColdFusion/Win2k but then introduced IIS and .NET.

  3. Back to top

    How do you test it?

    Nov 21, 2008 2:36 PM by Pavel Veller

    Dan, listening to you (or similarly Dan Pritchet talking about eBay architecture) I have same questions in my mind: how do you test new features before you roll them out? how do you test them scale? you don't have test lab with same level of scale as your production farm, do you? that makes me think you can't guarantee or knowingly predict exact level of performance until real users hit the new feature in real time. how do you deal with that and what did you have to build into your system to support new features deployment, as well as rolling features back quickly if apparently they did not work out as you had expected? and another question is what did you have to do to support existence of "practical" development environments that behave as your production system but do not require each develop to work on dozens of servers, partitioned databases, and cache instances. How did this change your system's architecture?

  4. Back to top

    Re: How do you test it?

    Nov 27, 2008 7:34 AM by Andre Sobotovych

    Pavel, you might be surprised how many companies don't have proper development/QA setups that replicate production environment well. I witnessed number of scenarios when code passed developer and QA testing with flying colors but utterly failed in production environment. The failure was caused (as you already guessed it) by difference in development/QA vs production environment setups. Developers were writing code for one environment while being completely oblivious (no fault of their own) to the production environment pains and hazards. Now, this might not be the case with MySpace. However, I certainly agree with you on the fact that Dan didn't say a word about testing their 'hydra'. But on the other hand, he wasn't asked that question. NOTE to Ryan: It certainly would be great to hear from MySpace on how they test the code.

  5. Back to top

    Re: Weird...

    Dec 10, 2008 11:35 AM by Jason Gleason

    Myspace uses a product called "Blue Dragon" that switches out the cf server to use a .net middle tier. It's a very cool option to migrate from cf to .net. (the .cfm pages you see are actually being handled by .net).

Educational Content

Bindings, Platforms, and Innovation

This presentation focuses on the Internet and separating myth from fact, history from the future, and the mundane from the imaginative. Bob Frankston presents a vision of what could and should be.

Orchestrating Long Running Activities with JBoss / JBPM

This article explores the use of JBoss and jBPM to implement design solutions that effectively address the issue of orchestrating long running activities.

Neo4j - The Benefits of Graph Databases

This presentation covers the use of graph databases as an optimal solution for data that is difficult to fit in static tables, rapidly evolving data or data that has a lot of optional attributes.

Realistic about Risk: Software development with Real Options

This session introduces Real Options and shows how it can help in running your project. Real Options is a decision-making process that can be used to manage risk.

Communication Flexibility Using Bindings

This article discusses the use of bindings on services and references (including the instance of non-configured bindings) as the means to implement SCA communications in a Web and SOA environment.

Writing DSLs in Groovy

After a short introduction to DSLs, Scott Davis plays with the keyboard showing how to approach the creation of a DSL by typing working snippets of Groovy code that get executed.

Scaling Agile with C/ALM (Collaborative Application Lifecycle Management)

IBM Rational and InfoQ present, Scaling Agile with C/ALM, an eBook showing organizations how to become “finely tuned software delivery machines” by enabling team integration and scaling.

Concurrent Programming with Microsoft F#

Amanda Laucher presents a real life enterprise application written in F#. She shows actual code snippets, explaining design decisions and suggesting how to use some of the F# constructs.