Larry Wall interview to Linux Format. Aug 14, 2009

Mike - 11:06am on Friday August 14th 2009

What does it take to design a computer programming language? Good knowledge of coding, certainly. A degree of linguistic understanding, perhaps. But what about a firm grasp of whipuptitude and manipulexity?

These are terms that only exist in the lexicon of Larry Wall: long-time geek hero, an open source developer before the term 'open source' was used and father to a close family who even go to O'Reilly's OSCon convention together.

But the Perl Foundation's latest project, Perl 6, seems to be under never-ending development. Guido van Rossum, creator of Python, has called version 6 "Perl's cross to bear", and certainly Wall's team are making massive changes from Perl 5. So we met up with Larry and his family to find out what's taking so long and what we can expect from Perl 6. And yes, he reveals the release date at last...

Linux Format: The Perl Foundation has funded various Perl developers through grants for a year's work or half a year's work, and a number of people ­- you included ­- have written greatly successful O'Reilly books. But do you feel that there's enough funding for Perl?

Larry Wall: The problem from my perspective is that [the work on] Perl 6 has essentially been basic research, and the business climate in recent years has not been terribly conducive to basic research. Nobody wants to invest in that when budgets are so tight. Essentially I have been officially unemployed for not quite five years now. There's never enough funding.

LXF: And I guess when you're out talking you're not programming, when Damian [Conway, core Perl developer] is out talking he's not programming. Does having to work as a convention speaker take you away from what everyone wants to see you doing?

LW: I don't actually get paid for speaking. Damian does, and I do a little consulting on the side to try to make ends meet, but that does take time away from the design of Perl 6. The part of the implementation I'm doing is the Perl 5 to Perl 6 translator, and I haven't been able to work on that much lately.

I've had a lot of infrastructural help from the Perl Foundation and from O'Reilly. For instance, O'Reilly hosts our weekly teleconference calls and has hosted some of the hackathon- type design sessions. It's a bit hand to mouth, but you know, whatever works.

LXF: Did you leave O'Reilly after the dotcom boom had ended, when people stopped buying books so much?

LW: O'Reilly had run into really tough times because of the plunge in book sales, which was already starting before 9/1 but very much accelerated at that point. I knew that I was one of their fluffier employees from a standpoint of their core business, so I was not at all surprised to get laid off.

People sometimes say, "Aren't you angry at Tim O'Reilly for laying you off?" and I say, "No, you don't understand." The years that he hired me he essentially paid me to do what I wanted to do. He essentially gave me a scholarship for those years, and I'm completely grateful for that, for what he was able to do, and so that's my feeling about Tim O'Reilly. I'm on very good terms with him.

LXF: So when this happened about five years ago you were just starting to kick off Perl 6, then?

LW: Perl 6 had been kicked off several months before that point, but the parting of the ways with O'Reilly really had nothing to do with that.

LXF: It still must have been a setback for you.

LW: Yes it was, but we were patient ­- we wanted to make the design take long enough. There's a saying in the software design industry: "Good. Fast. Cheap. Pick two." Well, we're an open source project, so by necessity we have to do it cheap. Therefore it degenerates to "Good or fast. Pick one." And we chose to do good, which means it won't be fast.

Our unofficial slogan is: "Perl 6: second-system syndrome done right". [Second-system syndrome is when coders try to make a 2.0 release much better than the 1.0 release, but end up adding lots of bloat.] But the only way to do second-system syndrome correctly is to take enough time to actually think through all the second and third order effects.

There's never time to do it right but there's always time to do it over. We decided to break that rule too, and essentially chose to take the time that it would have taken to do it wrong then do it over, to do it right in the first place. We definitely opted for thinking things through deeply, with the full knowledge that we would take a hit in market share to other scripting languages, but we figured we would probably get that back and then some once we got Perl 6 out the door.

So that's the plan. It still seems to be working, though I daresay we didn't actually expect the initial phase to take five years.

LXF: So this is the initial phase?

LW: Yes. We're done with about 80% of the design ­ now we're working on the second 80%!

LXF: Along with second-system syndrome the wise men also say you should plan to throw one version of whatever you're doing away. Are you confident that Perl 6 isn't the version you'll be throwing away?

LW: Well, our hope is that Perl 5 was the second system; this [6] is actually the third system. The second system in this case was sorely limited by the fact that I was designing the whole thing and implementing it. It was subject to my limitations of vision.

You can certainly get second- system syndrome by increasing the number of programmers working on the project, but at the same time there's also strengths in having more viewpoints. We decided to structure the Perl 6 redesign to take advantage of those strengths and try to avoid adding more programmers to a late software project simply by defining it as 'not late yet'.

We'll never be late in that sense: it'll be ready when it's ready. We're just going to do it right, and when we see places where it doesn't seem to be right we'll take the time to back up the wrong things and do it right again until it gets there.

That being said, we are always impatient for everyone else to have this, and so a lot of our design decisions at this point are: "Within the constraints on the current design let's just do the conservative thing, where we won't commit to something we're not sure about but will leave the space to go in that direction if we decide to."

It's easy to try for too much and commit yourself to an interface that you'll regret later.

LXF: You're having to get tough...

LW: At this point in the design it's definitely a design slush. Not a design freeze but a design slush, and it's getting more solid as the implementers come back and say, "This works great, we love the way this part works, but we don't know how to implement this." And we say, "OK, let's simplify here or there." Or in some cases we complexify here or there if the greater complexity results in greater simplicity at the end of this.

A lot of that design trade-off is going on right now. There's a sense in which the fun work is done and the grunt work is starting, but for some reason we seem to have recruited some people who think that the grunt work is fun. That's a healthy sign.

LXF: A wise person said to me recently ­- I think it was your executive secretary ­- "Larry's very good with times and dates, as you can see with Perl 6." Did you really think five years ago that you'd be where you are now with Perl 6? Or were you thinking that it would all be done by 2003?

LW: Five years ago I thought that we might have 20 RFCs [requests for comments] for suggestions about how to patch up Perl 5. Turned out we had 15 times that many, so it became apparent over the first couple of months that the redesign had to be either smaller or larger than we had first envisioned, and we chose to do the larger redesign. And it really set me on my ear for a while, just trying to digest these 361 RFCs that we got.

Gloria Wall: You spent a couple of months in paralysis.

LW: Yeah. I was just thrashing round, trying to find a handle on all of this stuff. Each of these RFC authors had a particular thing that they wanted to fix about Perl. Some of them were contradictory. Many of them proposed solutions that contradicted other solutions or were incompatible in various ways.

They all had tunnel vision as to what they were fixing. None of them could actually see a new Perl 6 as a whole and integrate into that, so it wasn't until I came to the realisation, months later, that I had to find a small end to start chewing on, that I could actually find a way to work through it.

Eventually I classified all these RFCs into the same order that issues are handled in the Camel book [Wall's classic O'Reilly text, Programming Perl]. It is sort of a small-to-large approach, but I realised that the order you want to present a language to people is the same order you want to redesign it. Certainly there were some forward references in the process, and we had to do a lot of work ahead, but by and large the design has been done in that rough order.

GW: Also, Larry wasn't expecting to be sick. That lasted a year out of the project.

LW: Yeah, a couple of years ago I ended up having two stomach surgeries. Two because the first one didn't work. Well, it worked in the sense that they chopped out the tumour that needed to be chopped out, but not in the sense that I couldn't actually eat or drink anything for a period of six weeks. During that time I was subsisting away on what could be pumped into my veins.

Back-to-back surgeries like that take a lot more out of you than you think they do. Two months after my second surgery I thought I was back to 100% but then half a year after that I looked back and said no, I was not at 100% then either. I don't know if it was that whole year, but it was a significant portion of the year.

On the other hand, that did give the people doing the implementation more time to work things out that they were working on, so it may not have been a bad thing. All potentially part of the masterplan...

LXF: You said that you expected to lose market share, given the gap between Perl 5 and Perl 6, yet Perl is more popular than ever. I haven't seen any sort of decline at all, so I wonder whether you are encouraged by that?

LW: Yes, I'm encouraged by that. From book sales Perl appears to be losing a bit of ground, but I don't know how representative that is. There's just been a new round of books about Perl 5, and it remains to be seen how they will be received.

LXF: I think Tim [O'Reilly] said something like Perl book sales are up 19% in the last couple of months thanks to the new Learning Perl and...

GW: Well you know, the percentage numbers are down but the absolute numbers are still flying high.

LW: On the other hand you don't know how much book sales are going down simply because of online documentation getting better. There are a lot of non-quantifiable things in open source software. If you had a marketing organisation and you had to record every sale then you would know how many people will have your product ­ except for the pirates.

In open source... well, I don't want to call everyone pirates, except in the Gilbert and Sullivan sense, but it's a lot harder to actually track who's using it. I see a lot more encouragement, particularly now that we are getting a Pel 6 compiler (written in Haskell of all things). I see a lot of excitement starting to build now. A lot of people were discouraged a year ago because it looked like it was going to take forever to implement this, but that's because all our implementation efforts were from the bottom up trying to build a VM [virtual machine].

Now with this new top-down implementation in a very high-level functional programming language they're making very rapid progress on an actual bootstrap and prototype compiler. Which will then be quite easy to translate into Perl 6 once it can do enough of Perl 6 to do that, and it's actually very close to that now.

LXF: That sounds good...

LW: While I can't predict that a year from now we will have a rock-solid platform, we will certainly already be self-hosted and running on a number of different platforms, only one of which is Parrot.

They already have back-ends to run on top of JavaScript, Parrot, and there's an effort afoot to run Perl 6 on top of the Perl 5 engine. That seems a little crazy to me and I don't think it will run terribly efficiently, but it's an interesting vision to have this language, which, for a change, is defined by specifications and by a complete validation test suite, and not really care what platform it's running on.

LXF: I particularly like the simple things about Perl 6. I like quantum superposition, and I like reduction operators ­- they're brilliant, I'd love to program like that.

LW: Well, that one's just straight out of APL, but unlike APL we're trying to optimise for human readability for people who are not necessarily familiar with actual operators.

You look at APL and you actually have to know what each operator is before you can understand the program. Just glancing at the Perl 6 reduction operator it's sort of apparent that there are these square brackets that tend to have things to do with lists or subscripts or something, and there's this operator in the middle of it, and it's nicely delineated and bracketed like a visual pill.

I'm a great believer in visual distinctions, so it really sets itself off as something that looks like nothing else [programmers] have seen: you know it's different, but it's really easy to see what the core part of it is too. We were very happy with that particular bit of syntax.

LXF: What do you find particularly excites you about Perl 6? Apart from the idea of finishing it!

LW: I think it would have to be the parts that are designed to evolve faster than even Perl 5 could evolve. Perl has always been about evolvability. That's partly why we have sigils [$, @, %] on the front of our variables, so they are often in separate name spaces from the verbs. And we added a lot of extensibility to Perl 5, which was why we ended up with the CPAN [Comprehensive Perl Archive Network, a public store of Perl code to download and use], which is the envy of all the other languages.

Nevertheless, there were many ways in which Perl 5 was coming to its limits. We had ways of mutating the language, but only very crudely. We had a mechanism called source filters, but the trouble is that the granularity was far too large, because it was from here to the end of the script. You could filter through a filter, and that meant that every source filter had to reparse Perl to do anything with it. You couldn't stack them, because they would interfere with each other.

In Perl 6 we actually give the programmer control over the individual grammar rules and even sub-rules, so that you can replace little bits and pieces of the grammar. This lets different kinds of grammatical mutability sneak into the language, and lets people experiment with different syntaxes and different ways of attaching those syntaxes to new kinds of semantics. And then we can have a lovely Darwinian gene pool that will pick the winners and also pick the losers, and that will contribute to the evolvability of Perl 6.

LXF: More user-friendly, perhaps?

LW: Part of the new evolvability design is the notion that we need better control over interface versions, so that different modules can for instance have requirements on other modules, and if two modules require the same module but happen to require different versions of that module, those can both coexist. Unless of course those two modules require some kind of exclusive access to a resource that cannot be shared ­ then they'd have to negotiate.

LXF: You said that Perl 6 was your one chance to break backwards compatibility. Do you think Perl 9 might be the same thing again?

GW: There's not going to be a Perl 9.

LW: I'm hoping that we're putting enough mutability into the language that all subsequent variants of Perl use this particular version of Perl. Maybe it's called Perl 7 or 8 or 9 or whatever, but that would probably be more for marketing reasons than anything.

What would probably happen is that if there were a popular module that mutates the language or the syntax or semantics in some way and gains great popularity, we might want to make it the default the next time around. But something like that would be a breakage of backwards compatibility, so it would require a new major version number.

LXF: You might leave a lot of people behind on Perl 5.8.

LW: That's all fine. Perl 5 is open source software and they will have that code forever. As long as someone's willing to maintain it, that's wonderful. That's part of the reason we actually felt secure in taking the hit for doing the design cycle. Because Perl 5 is a very stable base, it's a well known quantity. It is not evolving very fast because it's actually harder to hack on the Perl 5 internals ­ it's becoming more and more stable over time. If our experience of Perl 4 is any indication there will still be people hacking on Perl 5 five years from now. We're not terribly interested in telling people what they can't do.

LXF: You're not worried about the chance of a fork ­- people turning 5.8 into an alternative Perl 6 by adding a couple of new features here and there?

LW: I wouldn't view it as a fork, that's fine. I wouldn't really have a problem with that. Perl 6 is trying to make it easier to define your own favourite version of Perl. I don't see that people are going to be greatly motivated to try to extend Perl 5.

LXF: Finally, when will we see Perl 6? Give us a firm release date!

LW: Let's see... 30 July.

LXF: What year?

LW: Not telling!