Table of Contents
- Current Languages
- Identifying Requirements
- Developing a Custom Language
Over the last few days, an instructor and I have been discussing the merits of Lua, Python, JavaScript, and a few other popular scripting languages for their use in game engines. The predominant king of the field right now is certainly Lua, as can even be evidenced in part by my prior posts on the subject. Some of the topics we covered include Lua’s increasing lag behind the speed of modern JavaScript engines, Python’s design issues and lack of focus on being a dedicated embedding language, and what all of the available choices lack that we as game developers really want.
Current Languages
First, let’s look at some of the current popular scripting languages available today.
Lua
Lua’s popularity in the gaming industry can be attested to a few primary features. Perhaps first and foremost, Lua is down right easy to embed. It doesn’t pretend to be a Real Language like Python does; it may have all of the language features necessary for general application development, but it isn’t packages with a huge standard library and its stand-alone interpreter is nothing more than a few simple wrapper around the Lua library. It’s small and has no dependencies. This makes it excellent for embedding in any larger program. It also was once at the forefront of interpreted language efficiency, practically pioneering the register-based interpreter design that is now popular with many high-performance interpreters. Lua is also highly portable, needing nothing more than an ANSI C compliant compiler and environment. Unfortunately, Lua’s dedication to remaining ANSI C compatible is also its biggest downfall, as the requirement has forced it to lag far behind the state of the art interpreters available for other languages. Last but perhaps not least, Lua is very liberally licensed, removing any potential legal headaches when using it in a commercial game.
Python
Python has also seen some popularity in the game industry, though almost exclusively at the lower end of the scale. In most cases, the engines and games making use of Python are using it as the primary development language around a smaller C/C++ core, where as Lua is used mainly as an extension and logic language embedded in a large and full-featured C/C++ core. Python is in many respects a beautiful and powerful applications programming language, though it is not without its faults. Python makes it quite easy to put together casual and hobbyist games and through that has seen some small uptake in smaller scale commercial games, such as those based on Panda3D. Python has two large issues, however. The first is that Python is… well, large. It is a complete application programming environment with a massive standard library, the vast majority of which is totally undesired in the embedded use case. Secondly, Python is heinously slow in a way that can’t easily be fixed. Some of its language semantics such as automatic promotion of integers to BigNum types on overflow actually make some of the new mega-fast interpreter techniques unusable, or at the very least significantly less effective.
JavaScript
JavaScript has actually seen very little uptake in the hardcore gaming industry. The Unity 3D game engine is using a customized variant of JavaScript that compiles to the Mono IR and then can be JIT or AOT compiled to native machine code. Their extensions include optional static typing, which is a feature that both I and my instructor are in favor of in interfaces. The strengths of JavaScript over the other contenders is actually quite clear, though those strengths are relatively new. Perhaps most importantly is that JavaScript is really fast. Unity’s custom implementation is one of the fastest for sure, but the freely available engines such as SpiderMonkey or JavaScriptCore both include very heavily optimized runtimes with complete JIT compilation engines. The slowest of the popular JavaScript engines is still many times faster than the best Lua can do today, and the fastest of these engines is close on the heels of C/C++. JavaScript through its close cousin ActionScript has actually become one of the single most popular languages for game development, as Flash is used for a very large number of games; many of which you may not even realize are Flash games. Quite a few large triple-A titles have used embedded Flash for portions of their game, in fact.
Others
A few other languages are around and used in games. C# through Mono has seem use in a few games, and Thief made use of a Java interpreter for its scripting. Languages like GameMonkey Script, AngelScript, and Squirrel have been created to directly serve the embedded/games industry but have not picked up much steam. There has also been a fair share of use of languages like LISP, Scheme, Forth, and so on over the years, though their popularity has waned. In many cases, engines created their own scripting languages and runtimes. These custom script languages can run from simple modifications of established languages, such as UnityScript, to completely unique languages like the venerable QuakeC or UnrealScript.
To a very small extent, a few other popular general scripting languages have been used in the gaming circles. Such languages include Ruby, Perl, PHP, and TCL.
Identifying Requirements
With so many options to choose from, it can be a bit daunting to select an option for a new game engine. Simple going with Lua is not at all a bad choice to make, as it has proven itself good enough time and again in the professional world. Your needs or simply your preferences may drive you towards one language or another. The most important thing to do is of course to identify what you will be using a scripting engine for and what you need the scripting engine to do. You may well have a requirement or three that drives you entirely towards a particular existing solution, or perhaps even necessitates a custom solution.
- Speed - Game developers are really concerned about speed. Not surprising given the level of graphical detail, physics simulation complexity, complicated game logic, and low-latency networked multiplayer many games need to support. If your project has a good need for writing a large amount of performance-critical code in a scripting language, you are going to want to aim for one of the faster options. Right now, this pretty much means going with JavaScript or C#.
- Familiarity - A good developer will have no trouble picking up a new language. Learning all of the advanced tricks a language has to offer and getting a good grasp on the language's more useful standard library routines may take a few days at most while the core language can be picked up in hours. Unfortunately, not every team is blessed with a ton of truly great devs. Sometimes you just need to allow inexperienced developers to work on simple logic or animation scripting tasks. While some languages proclaim themselves to be easy to learn, in reality all programming languages are incredibly complex to pick up by an absolute beginner. An inexperienced developer is better served by a familiar language than an alien "easy to learn" trap. If your developers already have some familiarity with JavaScript, Python, or so on, it may be best to pick accordindly.
- Portability - Game developers deal with multiple platforms on a far more frequent basis than any other group of developers in today's computing industry. Three popular home consoles, two popular handheld gaming systems, a wide variety a incompatible mobile phone platforms, Web/Flash games, and the three popular variants of desktop/notebook operating systems. Common architectures include ARM, PPC, and x86, with x86-64 seeing a slow rise in use, along with a plethora of lesser-known architectures. These platforms range from big to little endian, from 16-bit to 64-bit instruction sets, with one core to an ever increasing many, and with a variety of unique quirks, limitations, and power features. Your target platforms will clearly limit your options. The iPhone does not allow dynamic script execution, for example, so you will be required to use a script language that can be compiled to native machine code, e.g. using Mono as a runtime. The XBox360 does not allow JIT-compiled code, so you may find yourself needing the fastest pure-interpreter language. Other embedded platforms are just very limited on CPU and memory and so need extremely small and compact languages.
- Special Features - Sometimes you just need a specific set of special features that lies outside of the least common denominator amongst all the popular options. Perhaps you really need or just really want a form of static typing. Maybe you need really, really fast vector math. Perhaps you just absolutely need first-class closure objects. Could be you just have to have UTF-16 string support. Whatever the case may be, if you know you need something out of your script language that only one or two options support, you may have to give up on a personal preference and go with what gets the job done. This seems fairly obvious and perhaps even redundant as advice, but I have seen a few teams pick a language that simply lacked a necessary feature because they didn't evaluate what they were going to use scripting for.
Developing a Custom Language
In the end, you may decide that you have a strong case for developing a custom script language for your engine. If you’re a student or hobbyist, you may simply decide to write your own for the sake of learning how such things work at an intimate level, which is something I recommend everyone do at least once given the opportunity. A custom language can fill the exact niche you need and meet all of your requirements, assuming you’re willing to sink in the (likely large) amount of time that will be necessary.
As a precaution, I do want to note that designing your own script language is not trivial. Simply determining the language semantics and grammar can be extremely difficult if you have little prior experience with language design. Writing a simple parser for a grammar is relatively straight forward, but performing proper semantic analysis, optimization, and opcode generation can be quite a daunting task for any but the most simple design. Finally, keep in mind that developing a truly high-end efficient runtime (including an efficient interpreter, JIT engine, and garbage collector) like those you’d get with a good JavaScript engine or a Mono-based language is a many, many man-years long effort without reusing a good portion of the existing technology available. If you are going to reuse technology, you may consider simply modifying an existing language to suit your needs.
That aside, developing a custom engine gives you the opportunity to solve a few problems that no existing language currently does. My instructor and I were contemplating a purely interpreted language with a small embeddable core with support for efficient vector math and type hinting, for example. We need a purely interpreted language for XBox360 compatibility, we prefer small embeddable languages for maintenance and ease-of-development reasons, and fast vector math makes it far more feasible to do various animations and particle effects in scripts. Targeted use of type hinting just saves a lot of time and effort in the long run after years of experience on both our parts, and it also allows the engine to potentially optimize the code better. No existing freely available language meets all three of those requirements. On top of those needs, we also have a few personal preference bullet points, such as having a comfortable C++-like syntax familiar to student developers, a strong optimization pass before generating the bytecode, and non-obnoxious license that lets us use the languages however we want without legal questions.
Given my interest and experience in the field of language design and implementation, I may well take on a project to develop such a language over the summer, just for kicks. I will be clear though that even replicating something as small and simple as Lua would take a few weeks of full-time work. Implementing something more efficient than Lua, using direct threading or context threading, will easily take a few weeks more. Tacking on a JIT engine for use on PCs is at least relatively easy given the availability of several existing high-quality JIT engines from Mozilla, Apple, and others. Adding in a good set of high-end optimizations and a solid debugging and profiling system turns it into a several month task at best. That is again assuming I could work on it full time, which I clearly can’t. I’m no stranger to this area, either.
If you are considering a custom scripting engine, realize that it is truly a large task that will take considerable development time to accomplish a fully usable and complete implementation of any non-trivial design. If that’s what you need to do to nail your project’s requirements, then that’s what you’ve got to do, but I can’t recommend strongly enough that you evaluate the existing solutions and use one of those if your project is commercial or otherwise has development deadlines to meet.</div>