Richard Minerich on Functional Programming, F#, Type Providers and Dynamic Languages

1. I’m Barry Burd here at QCon New York, I’m speaking with Rick Minerich, Rick tell us who you are and what you’ve been up to lately.

Well, Barry, I am a researcher, I do research and development at a company called Bayard Rock and we do anti money laundering, a lot of the largest banks in the US use our software which is largely based on F#, the functional programming language, I help run the Haskell user group here in the City and I run the F# user group.

2. And what attracted your company to the F# functional programming language?

Well, I got head hunted over Twitter as my boss really felt like F# would be a great language to do math in and he is a primarily MATLAB user, but when he saw F# he just felt it would be a great fit and we talked for a while and it originally was a contract arrangement for two days, I was going to come in and help them play with F# and learn about it and on the last day they offered me a job and I took it. I’ve been very happy ever since.

3. So, does F# in general have advantages as the functional language that you chose or is it mostly because it’s a good language to do math in?

Well, there are a lot of things that go into this kind of decision and in this case the company already has a large investment in .NET, you know SQLServer, they’ve got lots of C# software and F# just slots in nicely with that, of course there are lots of functional programming languages and F# is a derivative of OCaml, so there are lots of choices to pick from if you just look purely from the language standpoint but very rarely are any decisions made based purely on the language because the platform matters so much.

4. So, ignoring the platform for a moment, what do you have to say about F# compared with other functional languages?

I think F# made some sacrifices to be on the CLR and those sacrifices are worth it because if it was just a native programming language it wouldn’t be nearly as useful and there are already a lot of great languages in those spaces. Also, F# has a unique feature called Type Providers which let it give types at compile time to various data sources in other languages, it’s extremely powerful.

5. In your write-up in the conference brochure it says that you are a functional revolutionary. Can you explain what that means?

Sure. I graduated college and I started off working at a .NET company doing C# and I naturally gravitated towards the functional style without really ever learning much about functional programming at all, I found immutability to be really great, as soon as lambdas came I loved them, delegates I used them quite often , sometimes people in my company they did not like it as much as I did, but when I found F# I was instantly in love, it was everything I wanted in a language up to that point, it just supports all of the immutability in such a nicer way than C# and it just lets you build programs in a much more robust way and it has the types to back it up, that’s what really sent me down the path and since then I’ve come to realize that most programmers, in the United States at least, haven’t really heard enough about functional programming, I kind of decided I had to tell them or at least get the word out a little bit, help others tell them.

6. So, what might help get that word out? What is the best way to proselytize about functional programming?

Make awesome things in functional programming languages that other people can see and talk about. I would say that’s the number one thing. User groups, training events, conferences, talks, they all help but it isn’t enough though. You need to show people that it’s awesome, you need to show them examples of why it’s awesome. No one is going to just pick something because you say it’s good.

7. What about performance issues for functional programming? I know that is a concern of many enterprises, that an application written with a functional paradigm might run slower than one written in an imperative paradigm. What would you have to say about that and what’s the overall hit that management takes for such issues?

In general, it’s negligible or faster because at least for pure functional programming languages you can do all kinds of optimizations that you can’t do otherwise, I recommend taking a look at the Haskell webserver benchmarks that have been recently published, they are just killing everything else, and look at what’s being used, look at what else it’s popular, PHP, NodeJS; Haskell is a couple orders of magnitudes faster. And if you look at F# versus C# benchmarks, F# in most cases kills C# and you also have the ability to explicitly inline, so it gives you the ability to actually control your performance even more. I don’t think it’s founded, maybe at one point in history when the CPU wasn’t sitting there so much waiting for the cache, starving for data, that was more of a concern, but with modern architectures and modern problems, it’s a negligible problem.

8. So, now I hear a lot about the advantages of immutability that you find in functional languages given the fact that we are working with multi core architectures these days; can you speak to that, is the true potential realized or are we still years away from being able to take advantage of what’s available in the functional paradigm?

I would say it’s the other way around, I think we are years away from being able to take advantage of all that horse power, mostly just because starvation. You can blast through certain problems, there are certain types of math problems that are iterative that can keep things locally but most problems aren’t like that, today’s CPU spend a whole lot of time doing a whole lot of nothing, so I don’t know, I just don’t think it’s, I kind of lost track of the question, I’m sorry.

9. Well, let me rephrase my question. If you want to take advantage of multi core architectures, then as a general rule, is it better to be thinking functionally or to be thinking imperatively? Better meaning does the functional paradigm lend itself better to parallel processing because of the use of immutability, software transactional memory and so on and so forth?

Definitely. I mean, I can’t believe I let that one slip. So, part of the problem with immutability is that little changes can ripple through your program, especially with eventing. So, if you change a thing on a class, it fires an event, changes things on these other classes and the mutation ripples through your program, that’s bad enough. But now imagine that you have shared memory. Now, the problem is that those ripples are not thread safe and they are going to collide and cause all sorts of problems, or end up locking everything up really, really badly and it’s going to kill your performance. So, in general immutability in these cases is a huge benefit, also if you look at it algorithmically, one CPU is doing work on one chunk of data or maybe several cores are doing work on one chunk of data, you still need to distribute, if you’re using big computers, too many CPUs and you need to break up the data in a way that’s amicable to that. But if things are overwriting or vying for locks it can be a huge nightmare.

10. And functional programming helps a lot?

It helps quite a bit. I mean it’s not cure-all for all problems, like everything in computer science there are always certain problems that are solved better in certain ways, but it’s more of a way that you can fire and forget on most problems and not have to worry so much about the interactions of these different threads on your program.

11. What about hiring developers, developers who know the functional paradigm well?[...]Should managers be thinking about the personnel issue at this point?

Barry's full question: What about hiring developers, developers who know the functional paradigm well? Let’s say that a company locks itself in to a functional programming language such as F#, my impression is that they are going to have more trouble finding qualified people than they would with an imperative language such as C# or C++, is that a significant advantage to imperative programming as a foundation for an application or is it something that you would say “hey, better to use the best language for the job”? Should managers be thinking about the personnel issue at this point?

I definitely agree the personnel issue is a big deal and there are certain things you can do to get around that, but it’s not always possible, depending on where you are in the country, on what’s going on. I would say it’s one of the biggest challenges facing functional programming, is getting people trained up. But we’ve had a lot of success with that. For example, at the F# tutorials last year in New York City we brought one junior C# programmer to the tutorials and he spent two days learning F# and he left, he went back and he wrote some awesome stuff in F# and it was just like that. And I think that the advantage of using a functional programming language on the same platform that you already know, is that it’s much less of a leap until you learn that, and then you can take the principles you learn there and apply them to other functional programming languages in the future if you like, but it’s a big leap to go to a whole new platform, a whole new set of libraries and a whole new language all at the same time.

Barry: So, it sounds as if to look for a good F# programmer, if you can’t find one, what you need to do is find an existing C# programmer and train the person.

Exactly. If you talk to them and they love to use lambda expressions, well, they’re a great candidate or maybe they use LINQ a lot, that’s a great candidate for learning F#. If they’re still using their Gang of Four book every day, they’re probably not such a great fit.

12. I think you’ve already answered this question, but I want to ask it in a slightly different way. Is functional programming ready for the enterprise?

Oh, more than ready, it’s been ready for a long time, just the support wasn’t there yet, the community support, look at Scala’s success, I guess it depends on how you define enterprise, here in New York City a lot of finance institutions are using a whole lot of functional programming, whether it be OCaml or Haskell or F#, they have hundreds of programmers using these languages. And then you have the pseudo functional, maybe pseudo isn’t fair, like LINQ and such. There are two attack angles going on here, one is from the inside and one is from the outside. I felt like at Microsoft, Erik Meijer was leading the charge on bringing functional programming into the other languages, that’s why C# has so many of the great features it has today. At the same time that trains up people from the inside so that they can move out into actual functional programming languages.

13. Let’s talk for a moment about pure functional programming, Haskell. Should one think about doing pure functional programming for real problems?

I never see a problem with someone thinking about anything.

Barry: Should someone think seriously about it?

I think it’s a great mind expanding use of your time to write a bunch of Haskell, and I think actually some problems can be solved very well in Haskell that can’t be otherwise, for instance some of the machine learning stuff that’s going on in Haskell right now they are seeing performance that just blows the Java equivalents out of the water because they can do some smart things with the math because of the data structures underneath. And I can see that as being a great inroad for Haskell in that case, but I don’t see most companies adopting Haskell mostly because the dirty secret of the Haskell community is that even Haskell programmers sometimes have to use dirty tricks to get the job done, there isn’t an FFI, sometimes it just depends on what is important to keep safe and what is more important to just have now, at times you have to make tradeoffs like in anything in life, there is no one answer to rule them all, it all depends on your requirements and your situation.

14. Another thing that you mentioned in your talk was asm.js. Do you want to talk for a moment about that and what its advantages are and what about it is so cool?

Well, the cool thing is from my perspective, it’s cool for a lot of reasons, most of all the idea that someone could port a game written in C++ directly into JavaScript, it’s kind of mind blowing, you get these games that came out on consoles two years ago and could show up in your browser tomorrow, that’s a cool feature. But more cool from my perspective is the fact that I might be able to use a language that I enjoy in the browser and JavaScript is not one of those languages. It’s a personal preference thing.

15. Now, your talk at QCon is about polyglot architectures. Can you remind our viewers what that is and what are the key points to remember when working in a polyglot project?

A polyglot project is one that uses many languages and so the key points to remember are mostly about keeping them balanced and keeping them segregated in such a way so that they don’t really stump on each other and cause you incompatibility problems. It really depends on your platform and what the languages are because let’s say you are like my company and you use a .NET stack, so you’re polyglot, you’re using C++ CLI, you’re using C#, you’re using F#, you’re using maybe even VisualBasic and maybe some SQL in there and some Iron Python, Iron Scheme, Iron Ruby, there is a ton of stuff that you could be using all on the same project that plays really nice because it’s all on the CLR. I feel that once you get off of the same platform then you have to be a lot more careful, because you have to worry about serialization, moving data around and potential conflicts, it becomes a lot more messy.

16. Do you have recommendations for what to do to overcome the messiness?

I’d recommend mostly staying on the platform that you need because most of the times it’s not worth it, sometimes it is but other times it’s not. One great example of where it’s worth it is with data science and this is a big problem. We do some data science for anti-money laundering and you have pretty much the big three: R, MATLAB and Python, all the things you care about are really in one of those, but it’s sort of a guess as to which one has what and so if you want to be a good data scientist, you have to be able to use all three and then they all have very different syntax and different properties when it comes to making good languages for actual development and the solution as I see it from my perspective is they need to be unified through some sort of layer, in this case this layer is, for me, F# Type Providers.

17. Can you say briefly how F# Type Providers handle that, what mechanism is being used there?

Sure. An F# Type Provider is a compiler plugin, sort of like a limited kind of macro and what it does is at compile time it goes out to any data source you want or anything that it can talk to, it’s a normal code, you’re running normal code, and it gets types from it, puts it right in your VisualStudio. For instance, I’ve been working lately on the MATLAB Type Provider and you can dot into the MATLAB packages and functions and you get your help right there with IntelliSense and your F# package. Of course it’s not all roses, you have to tell MATLAB how many return arguments you want and so that can be a little bit, it’s not beautiful like normal F#, but it’s a huge step up and then if you have let’s say a Python, there is already an R Type Provider and then the MATLAB Type Provider, you can just seamlessly do computations in one of them, take the data out, put it in the other, do some computations, take the data out, you can just chain them all together to use them in any way you like, it eliminates the walls between them.

18. So, is this what you mean by generating static types from dynamic data sources or is that something different?

It’s totally what I mean. Type Providers when they came out they were totally targeted at going into SQL and giving you typed SQL tables with Joins over them, with more static versions of LINQ than C# has. Now, Howard Mansell figured out that you could do this to programming languages and he did it with R, and it was amazing, totally blew my mind and then I decided I need to do a MATLAB one and then all of a sudden, a vision came together, I could do all of my data science in F#, and F# is really good at bringing in data from various places and transforming it to the right way and most of these languages are not so great at that, Python is passable but not MATLAB and R, definitely not. And so, you could take your data, munge it up in F# and then put it into those as needed directly, do your algorithms, take it back out and put it in the other data source and it’s beautiful, seamless and as typed as it can be.

19. So, you mentioned this is one of the things you are working on now or thinking seriously about, thinking hard about, what are your current thoughts about this whole idea of generating static types from dynamic data sources?

I think it’s extremely exciting, because one of the biggest problems with statically typed functional programming languages for a long time has been there is always a boundary that you have to draw, a line in the sand where on one side a really messy world exists and on the other side you have your beautiful functional paradise, so you have to have that buffer there so you can deal with the real world because you don’t want it infecting your entire program but what it essentially does it takes away that ugly end piece and replaces it with something beautiful.

20. So, this kind of an adaptor between the types of one language and the types of another?

Yes. Of course it’s limited only to the degree that F# supports types. So, you couldn’t for instance make a Haskell Type Provider and have it work very well because Haskell has very fancy types and F# wouldn’t be able to represent them well. But I think it’s a huge step forward to have what we have.

21. Are there any weaknesses you see currently in functional languages that you would like healed or worked on by the community?

I’d say some people would just yell me saying “use emacs”, but the IDE situation is abysmal. F# probably has one of the nicer IDE experiences, maybe Scala with IntelliJ, but in general the functional programming IDE, there is so much opportunity for analysis. Functional programs are easier to analyze than statically typed counterparts. So we really should have these beautiful factoring tools and code visualization tools, all kinds of tooling, but it doesn’t really exist. So I think that’s where the imperative object oriented community has spent their attention for a long time and they have a lot of nice tools and I would like those.

22. And in the functional world, what are people working on most now? You mentioned that you see a lack in tooling, what lack is being filled most, what are functional programmers, functional developers thinking about most for the future of their languages?

It totally depends on the person. I think it’s hugely impacted by what problems they are trying to solve day to day. For me, I see this beautiful opportunity to make my data science work way less painful. For someone else, it might be they have huge volumes of real-time data they need to process because they do finance and they need to have that super fast and nice syntax. Every single slot to fill has a different thing. It’s a big opportunity to have companies that come along and write these tools because we are going to pay for them, but the user base has to reach a certain threshold to make it worthwhile for the companies, so I think the opportunity is there for Scala right now and potentially for Haskell, based on the user group numbers we are seeing. But there are a lot of functional programming languages and it would be nice if they could share or something like that. I guess it’s why they tell me to use emacs.

Barry: Rick, thank you very much.

Thank you.