-11650

030020

Trip Report: One Way From APL to J and K

J/K	A few words about treating J and K together may be in order. Both languages were important to me as I gradually moved away from using APL as an important part of my programming repertoire.
	Compare	J and K are quite different languages, but they share enough characteristics to make it not completely foolish to talk about them together. Their similarities are at the conceptual, not programming language, level. Their differences are structural and (most probably) in deep structure implementation. In what follows we'll use J/K when the arguments apply equally to either. Most of the time, and for the purpose of being used as an APL follow-on, either would serve our purpose well.
	Elsewhere	The issue of what differentiates J from K has been treated in some depth elsewhere. The purpose of the discussion here is not to emphasize their characteristics, but rather to suggest why these languages have replaced APL in my implementation repertoire.
	Attraction	There was nothing particular about J/K that caused me to be attracted more to one than to the other. As should be clear from the discussion below, I have used both of these languages in circumstances where I might have previously used APL. It may be relevant that I first learned J, but since I now probably do more K programming, I think of myself as having equivalent skill in the two languages.
	Summary	J/K represent reasonable alternatives to APL. While it is slightly unfair to treat them together, I have found them a productive alternative. And for the purposes of this discussion treating them together simplifies discussion.
Personal Background	Insofar as it is relevant to this discussion some aspects of my background may have an impact on the way the circumstances have an effect on the choices that I have made. So in the interests of disclosure, some short items may be relevant.
	Taught APL	I have taught APL to some thousands of students over more than a decade of teaching introductory computer courses. Most of this has been to BA/SB students in Economics and to MBAs. So I have seen lots of different people work with, struggle and learn APL. And a lot of the acceptance of APL in the financial communities of the early 1970s was due to students that came out of these, or similar, programs.
	Lots of Languages	I know, and regularly use, lots of languages. I suppose my principal implementations were generally done in C, but early work was in Fortran, and more recent work has been in perl. I write a fair amount of code and more than a little descriptive and expository text. Many of the problems I analyze have a quantitative aspect, although few of them involve any profound or deep mathematics. I know lots of computer languages, starting with several assembler languages in the 1950s, and now use lots of more `modern' tools.
	Long Consulting Experience	I have spent a long time consulting for a variety of organizations, both large and small. In the course of working on these assignments, various programming tasks were almost always at least a part of the problem. In addition some of the assignments were working on electronic document creation and dissemination at large financial houses, thus mixing editorial, analytical and technical problems in about equal measure.
	First Encounters	My first encounters with J/K started about a decade ago (J, principally, as K came along somewhat later). The earliest versions of J that made it into public circulation date from about 1990, and my regular use of J began a couple of years later, around 1992. K was made public a good deal later, but now we have had several years of K usage as well.
	Old APLs	For the past several years I have been using J/K, so I have little experience with the `modern' APLs. The last APL I bought for a corporate client was probably in the middle 1990s, and it was an adequate APL, though I found it somewhat clumsy to use. Since finding J/K I haven't given much serious thought to the capabilities of current APLs, although I have continued to follow comp.lang.apl, so I am reasonably certain that nothing has happened that would cause me to investigate it seriously again.
	What I don't do	Contrary to many APL users, I don't do a lot of matrix mathematics and I don't do a lot of statistics. Of course some of my problems will make use of the statistical capabilities of these languages, but most of my uses are what might well be described as`non-mathematical'. However, even if the problems are non-mathematical, they are, nevertheless, quite real.
	Summary	My work may make me more interested in applications that are not intrinsically formulated as matrix problems. I tend to process a lot of text and do a fair amount of work with long integers. So I cannot claim to be a normal APL user. However, my work load is quite typical of that of a reasonable number of people who do some analysis and write a wide range of reports that often involve some quantification. It is probably appropriate to note that much of this work is better directly supported by the facilities in J/K than is the case in APL.
The Pull to J/K	Lessons learned from Knuth (and others) and which may be a deeply ingrained part of MIT undergraduate education, suggest that it is usually worthwhile to keep a sharp eye out for what the `smart guys' are doing. As one might expect there are `LISP smart guys', `perl smart guys', `python smart guys', etc.. Among the APL `smart guys' Iverson plays a special role as `inventor' as well.
	People	I was using APL fairly actively in the early 1990s. I guess the first clue I got about J's existence came from noticing that Iverson had been working on a language called J. If I recall correctly, the source to this little language was actually available, and I was particularly charmed by the fact that it would run on my HP200Lx handheld (as would APL---in the Manugistics version). Iverson is certainly one of the `smart guys' that it is always worth paying attention to. Although I didn't know it at the time, Roger Hui is also in that league, and the combination---with help from lots of talented others, no doubt---produces a very effective piece of software. It is also, given its authors, an intellectually cohesive piece of software, and this shows up regularly, often in important circumstances.
	Typography	Another nattering problem with APL has always been the typography. I still have some code---probably written in the early 1970s---that allows me to (with tedious and trying exchange of typeballs) put APL code into typeset documents. Within the past month there was another of the seemingly interminable discussions of APL and typography on comp.lang.apl. I figured, I guess, that if Iverson decided that his ancient typography wasn't worth saving, I probably wouldn't profit much to spend time second guessing him, so following his lead made sense.
	Flow	APL never seemed, to me at least, to represent the flow of control very comfortably. While I understand that now there may be much better representations for this in modern APLs, I never cared much for right arrows or line numbers or for the odd machinations that always reminded me most of my machine language days where we would program jump tables and mistakenly think that we had done something profound.
	Former Students	While I was beginning to play with J I started to hear a lot about K, principally from some of my Wharton students who were spending a lot of their time working on some of those derivative financial instruments that would later checker their careers. I started to hear about demos where a few keystrokes and almost no time were required to do things that took ages not only in the Oracles and SQLs but also even in the APLs that were then still commonplace on Wall Street.
	Whitney	A name which kept popping up in these contexts was that of Arthur Whitney. After a bit his links to A+ and to Iverson and Hui made clear that he was another of the `smart guys' that was worth paying attention to. Around this time comp.lang.apl was full of debate and discussion about J, and Stevan Apter (another of the real smart guys) started to talk about K with some frequency. A problem for the rest of us, at that time, was the fact that nothing (that I knew of anyway) had been published about K. So these discussions were not always very informative, often ending with some startling revelation from Apter which was nevertheless unprovable, as few of us had any access to K. And the snippets of code that were exposed were generally undocumented, and given the rebarbative appearance of K, often woefully obscure. This all changed when Whitney decided to release an `evaluation copy' of K that was functional enough for most of us to use to develop an understanding of the language and to handle the kinds of problems that had a personal, not corporate, scale. This came along with documentation that was very effective in explaining the elements of K once one learned how to read the terse writing that described the language.
	Activity	As the J/K people began to leave comp.lang.apl traffic on that newsgroup dropped precipitously. For many of us the signal/noise ratio collapsed, and discussions of highly intellectual subjects like `Are APL people disproportionately hirsute?' started to occupy a significant proportion of the traffic.
	Keyboard	That the APL keyboard is a nuisance is hard to argue. Even after nearly 30 years, my fingers still know where the proper keys are for the APL functions. But that's not of much use to me, and it is a positive hindrance to anyone who's fingers aren't similarly programmed. When I was still writing APL code I even learned how to read the incredibly odd expressions that resulted from the various renderings of the APL glyphs. But this is all an extra burden with very little payoff for me. And indeed it has no payoff whatsoever that I can see to any `new' person who might encounter the language. Neither J nor K impose such a burden on the learner, although they do, I suppose impose some burden on the `old-APLer'. But then there are fewer and fewer of them to worry about every day.
	Off-Line Editing	Special character sets are also a real nuisance to edit outside of their `home' environment. It is generally not an easy proposition to hand an APL function to a conventional editor. With J and with K this is an everyday occurrence. While J has its own IDE, K does not, and both of the languages are very easy to handle in any conventional ASCII-oriented editing environment.
	Off-Line Everything Else	And then there's the problem of everything else that one might want to do outside of the specific implementation environment of a particular programming language. For example, I regularly have K code that writes perl and perl code that writes K. They're all ASCII based, so none of this seems either unusual or difficult. To illustrate with a specific example, I happen not to like K's regular expression handling much, but that doesn't cause me much any real difficulty as I can just use the kind of `bridge' discussed to accomplish the task of making this easy.
	Character processing	I have never much liked the way that APL handled strings as variable length entities. J/K (particularly K) handles strings much more easily and naturally. Since strings, and text files, are a very important form of storage and an important domain for computer manipulation of data, this is particularly important to me.
	List Structures	Some problems just don't fit into a rectangular matrix paradigm very naturally. In such cases the list structure orientation of K is particularly effective. In the view of K, matrices are special cases of the more general concept of list of lists. In general I'd say it's easier for K to adapt to the special case of matricies---they are regared as rectangular lists---than it is for J, or APL for that matter, to treat ragged lists---by boxing them in homogenous structures.
	Cost
	Summary	This is a long list of things that have interfered with my use of APL in an even wider set of circumstances than seemed to be natural. These difficulties have, for me at least, now been pretty well resolved by using J/K. And the transition hasn't been very difficult.
Keep the baby	APL has taught us lots of important lessons. It's important that we don't throw them out if we are going to move to something else.
	Terseness	Perhaps the most important of these lessons is the value of terseness. When a computer language is terse, you can see a lot at one glimpse. APL is admirably terse. So are J/K, thus preserving the value of this important aspect of the original language.
	Clarity	One of the things that moved Iverson in the direction of defining APL was clarity and lack of ambiguity. Again both J/K continue in this tradition.
	Interpretation	Interpretive languages are so powerful, and in the case of languages that are terse as J/K are, this does not require that we sacrifice the power and expressiveness of being able to write code `on the fly'. Interpretation is a technique that seems to allow a straightforward expression of matters that would otherwise be very complex.
	Interactive	The ability to sit at the console and `play' with problem formulations and solutions is also immensely powerful, again particularly when the languages involved are terse. Terseness is attractive when you are actually typing commands on the console directly. And a little experimentation often proves to be a key in getting to an effective view of a problem domain, as well as an important step in often avoiding hours of debuggin effort that might otherwise be required.
	Non-Iteration	All of the languages coerce one into thinking non-iteratively. While J/K can be used with conventional loops, and possess loop-like constructs, they generally coerce you into thinking of problems in some array or list oriented form. For many problems this is a particularly enriching experience.
	Intellectual Cohesion	APL, J and K are all remarkable for their intellectual cohesion. Perhaps this is a direct consequent of Iverson's leadership, or perhaps it has wider causes. I don't know. What I do know, however, is that almost everything that ought to work actually does work and that is something that doesn't happen in environments that lack a fundamental underlying cohesion.
	Simple Rules	These languages also do amazingly complex tasks with equally amazingly simple rules. The fact that the rules are so simple makes understanding the overall structure easy, although it seems, for me at least, to be a lifelong study to understand the elaborate consequences that are obtainable with even a very simple set of rules.
	A Negative	There is also one important negative that has also been inherited from APL. There is considerable evidence that lots of people find languages of this type hard to get their heads around. This is no surprise. Some people find rigorous structures easy to understand and use, and others can work a lifetime and not find them anything other than a hindrance. J/K shares with APL the fact that it is hard to think about, but very effective to use, if you can get your head around it.
	Summary	It is fairly obvious that APL provided an extremely interesting and provocative view of an important subset of computer problems. For a while some of us were naive enough to think it was a better way rather than a different way. That idea, although it seemed to gain some currency for a few years, gradually withered away in the face of mounting evidence that it wasn't better but rather just different. However, there is enough value in just being different that should be exploited by people capable of doing so.
Programming Language	For me the most important use of a programming language is how well it functions as a practical matter solving real problems.
	Conventional	J is a fairly conventional programming language in many normal respects, and as such seems, right out of the box, to be more comfortable to me than APL. Representation issues aside (they argue against APL) the control structure of J seems quite normal to a C programmer. While some things (implicit array iteration, for example) are quite different, normal programming structures look like normal programming structures. They are easy to recognize and not troubling. With K this is less so, as control structures happen to look different than we might expect, but once one `gets the hang' of it, they read quite normally, too.
	Efficacy	Both J and K are efficacious. Somewhat to my surprise this is particularly true of K. At first I found K hard to read and write, but now it has become a rather commonplace activity for me, one that I face with no more trepidation that I do facing a task with any of my other programming tools.
	Library	J has a large library of functions that are available to users. Over time lots of packages that deal with things like `fractal mathematics' or `concrete mathematics' have been developed and are available for use. K has much less of an available library, but its functions are more financially oriented, and this is an important area.
	Summary	I find J/K easier to use to represent `conventional' programming structures than was the case with APL. As a result, they serve the function of being a `programming language' better for me.
Mind Bend	These languages are more than just programming languages. They also influence the way we think about problems. As such, if they are successful, they open new ways for (at least some of) us to tackle the problems we have at hand.
	New View	All of the languages in this class can generally be regarded as instructive from the standpoint of having some possibility of giving us a new view of problems. The whole notion of looking at things other than iteratively can be enormously productive, and all of these languages are conducive to that.
	Learning	However, I find K particularly startling with respect to giving me new views of problem solutions. And I say this without even using K in what is arguably its most significant problem domain, namely that of data base management.
	Conceptual Terseness	There is also an almost `mystical' aspect to terseness that is hard to describe. When you have terse code, and see simplifications that make it even more terse, it seems that you are often getting closer to the core of the problem. Thus shorter is, often, not only faster, but truly better. I find my short fragments survive environmental change even better than long elaborate pieces of code. I suppose this is because the more we choose a solution structure that parallels the structure of the core of the problem, the more likely we are able to adapt to circumstances as they evolve in reality.
	Summary	If you have the kind of mind that can make effective use of any of these languages, then the languages themselves can help you shape and structure your thoughts in ways that lead to them being tidy and effective. In my experience not everyone can, or wants to, do this. But those who can and do find using the languages, in places where they naturally apply, to both help themselves express their understanding of the situation, and to help them modify and make more elegant that understanding. In my experience, this is less often the case with other computer languages, even when they are very helpful in solving practical problems.
Community	Community size can be important. Unless a language is being used only to maintain legacy systems, new jobs are likely to be created in some proportion to community size. Similarly, the more `others' there are who use a language, the more likely their work may be able to help us. So size counts.
	Conferences	There is little to distinguish among the languages with regard to conferences. First, APL, J and K people tend to go to the same conferences (if any). Second, attendance at these conferences appears to be slowly declining over time. Since I don't go to conferences this has no particular impact on my decisions, other than if there appeared to be a major difference between the languages in this regard. Since there isn't, as of now, any such detectable difference, conferences are not important in making the decision to move away from APL. Conferences also seem to be of declining importance at least partitially due to the growth of the influence of the Internet and to the general decline in travel. Modern communications technology obviates the need to physically gather the clan. In addition the publication of active results in conference proceedings has now been effectively obsoleted by the availability of truly current research documents on the Internet.
	Newsgroups	Newsgroups are another story. When I started to follow newsgroups on the Internet about a decade ago, comp.lang.apl had a reasonable amount of traffic with a respectable signal/noise ratio. Then, in the early to middle 1990s, when the J group split off, traffic on comp.lang.apl began to decline. When Whitney brought up kx.com's website and set up a list server to handle K related mail, traffic on comp.lang.apl declined even further. At the present time, discussion on the J forum is quite robust. The K list server has bouts of high activity that punctuate slow periods. In both cases the signal/noise ratio remains good to very good. There is little traffic on comp.lang.apl anymore, and what little there is often of no substantial technical content---ranging from discussions like How can we prop up APL? to discussions of why managers are clearly ignorant for not seeing that 30 years of lack of success in building effective APL communities in organizations is really a harbinger of a wonderful future for APL.
	Membership	The only professional society that has any bearing on this business seems to be SigAPL, and its membership has been in a long slow slide for many years. There are essentially no corporate members anymore, and regular memberships seem to generally decline at a rate of 10-15% per year. I know of no corresponding professional societies that might usefully contribute any `membership statistics' to let me make the same evaluations about J and K, though, so again this doesn't provide, in and of itself, any reason for choosing one over the other.
	Education	I have not made a careful study of the use of any of the languages in educational settings. However, I have at least some residual experience with the use of APL in educational institutions in the US, but no real experience elsewhere in the World. So my view is limited at best. It does seem to me to be safe to make a few observations, though. First, there was a period of time when APL was taught at several of what we might call elite institutions in the US. For the most part this was in the late 1960s and early 1970s. With the arrival of the spreadsheet, most of these efforts became increasingly moribund, and as far as I know, virtually all such usage had ended by 1990. A number of smaller, less well-known (and therefore less influential) institutions have since taught some small amount of APL and a little J. One institution in the UK appears to teach a little K. However, taken across all institutions in all countries, the amount of any of these languages that is being formally taught is extremely small, and it is unlikely that the languages will be able to maintain any substantial influence because of schools knowledge. Again, this appears to be generally equally true for all of the languages, so there is no differential effect here that would cause anyone to choose one over the other because of the availability of school trained people who understand the language.
	Summary	Although it is very difficult to get one's hands on any real `figures', I believe that the APL community is decreasing in size and that both the J and K communities are growing. Practically speaking, however, I should make the point that it is my impression that the growth in J and K just about explains the decline in APL, so net I see very little overall movement with respect to the whole problem domain. Indeed over the last 40 years or so, I see little indication that the size of this community, taken in total, has changed in size much at all.
Documentation	No one doubts the importance of good documentation. The only problem is that what one person regards as good, some other person will regard as horrible. If you accept that argument and think about its implications, then there may be a dominant reason to produce lots of different kinds of documentation in the hope that it might support different modes of learning in people.
	Books	There are few books of much interest for any of these languages. It amazes me to see people still refer to the APL text I used when teaching it in the 1970s as `best book around' for teaching APL today. Even if you go to the bookstore with the largest collection of computer books, you are unlikely to see any books about any of the languages discussed here. So there's not much to choose between on this measure.
	Drafts	There are some drafts of books about J in circulation in the J User's Group. This is some sign that there are people willing to write about J. These books seem to be quite good in their own problem areas, but whether they will end up by serving much of a market or not still very much remains to be seen. However, at least there is some activity and this very activity provokes interest and discussion.
	Journals	I don't know of any Journals that support either J or K, so here APL has a clear advantage, if the journals that exist happen to serve your interests. I found over the years that I used journals as a means of useful communication, their impact and role was declining, and with the arrival of the Internet I now generally find them obsolete. But your interests and experiences may differ. In any event, if Journals matter to you, then that would probably suggest points in favor of APL.
	Papers	The use of any of the languages in academic papers is limited. It may also be rather `topic specific'. For example, most of the research work done with the huge data bases that are commonly processed by K would generally be in the financial area, while most of the APL papers might have some focus on actuarial and insurance issues. For my purposes, none of the languages are heavily enough used in papers for it to influence my choice of working language.
	Conventional Documentation	I'd say that I personally prefer J/K's conventional documentation (particularly that of K) to any APL I know, but that may be largely a function of the kinds of problems I tend to work on.
	On-Line	J/K has on line documentation that is far superior, for me at least, to any APL documentation I ever encountered. But documentation is always a matter of taste, and your tastes may not be the same as mine.
	Summary	Documentation obeys several well known laws. For example, there is never enough of it. There is also too much of it. It is never detailed enough, except when it is too detailed. It is never up-to-date except when it is written to incorporate features that have not yet been put into the software. In short, documentation is the thing that we blame for whatever other problems we are actually having using a system. Of the systems discussed here, I personally prefer K's documentation to J's, but only a little. I far prefer J/K's documentation to that of any APL I know. But I could well understand if others would feel differently. For example, J's documentation is quite idiosyncratic, wonderful once you `catch on' but often quite useless until you do. Thus it is hard on new users, but a real blessing to those of us who call upon it daily.
New Problems	Up to this point we have been dealing with the issues of J and K as an alternative for handling tasks that we might have otherwise handled in APL. There is a whole additional class of other considerations, namely tasks which we wouldn't have tackled in APL, but which prove to be quite feasible in J/K. Here are some.
	Data Base	K is particularly rich in data base functionality. Indeed, managing large (billion line) data bases is the primary domain of application, and the speed with which it can do so is already of nearly legend status.
	Net Connection	Connections to the net are also an area of increasing importance and networks become an everyday part of the structure of both organizations and with respect to inter-personal communication. J and K both allow the easy management of TCP/IP message stacks and make the transmission and reception of messages across the Internet quite an easy thing. In addition, K's way of handling this problem often results in complex systems which can be easily debugged on a LAN, and then with a very simple change of command parameter suddenly turn into servers available on the net. Some of this code is startlingly simple.
	Deployment	Deployment is also an issue of importance to anyone who has systems that need to be installed at lots of sites. Not only can the instillation prove difficult, but the maintainence of such systems can easily become quite a nightmare. K offers some truly incredible facilities for deploying operating code. Indeed, the distribution of the whole system itself is a system administrator's dream, as K only requires a 200K executable that is split into a couple of modules.
	Summary	J and K both have their place even if they are not being used to replace APL. The fact that they can serve these purposes in addition to replacing APL is just an added advantage.
Conclusions	There are some lessons. Like all lessons, their significance changes, but their core is now quite stable, and I don't think that I've seen reason to materially change these judgments for at least several years now.
	Usage	By using J and K, I virtually Never Use APL anymore, and in the (increasingly rare) circumstances that I actually encounter some old piece of APL I now regularly re-write it into one of the newer languages. As time passes, and I have less and less legacy code this becomes an increasingly rare process.
	String APL	K has become a sort of `string APL for me. With only a few keystrokes I am able to pick up a file, read it into memory, manipulate it in some significant way and then write out the result. About the only `serious' string facility that I miss in K is regular expression handling. K's capabilities in this dimension are not very sophisticated, but K makes it easy to call on other facilities that might be present in the environment to perform such functions.
	Integer APL	I find J to be particularly effective for handling integer problems. K, by design, is not very good at integer problems. Integers are thought of principally as `pointers' to storage blocks. They are not much used to compute with. in J, on the other hand, the algorithms thru which the handling of such numbers are both rich and effective, and the integration of extended precision integers with other data types is well thought out and easy to use.
	Rich Number Types	J, in particular, is very rich in numerical data types and in functions that support basic mathematical constructs. For example, not only is there full support for complex variables, there is support for rational numbers, factoring, primes, polynomials, etc.
	New Users	I would unhesitatingly point new users towards either J or K rather than towards APL. My principal quick arguments would have to do with typography, keyboard, cost, and the availability of an active and interested support community on the Internet.
	Old Users	I really don't know what advice to give old users. Actually, that's not true. If pressed my advice would be to abandon APL and invest the time and energy required to learn whichever of J or K seemed to offer a natural structure that was best suited to the kinds of problems that an old user regularly had to solve. I would never presume to think that someone else would learn things with the same difficulty or ease that I do, anyway, but as testimonial experience I can say that I could use J in about a week and K in about two weeks. But, of course, such early usage was crude, cumbersome and caused substantial discomfort and `pain'. However, in both cases, productivity came fast.
	Future Prospects	As to future prospects, most of the work I know for APL programmers relates to legacy systems. Since there are no real J/K legacy systems, most J/K work is at least new work. For some this would likely be more interesting.
	Summary	The trip from APL to J/K has been a journey well worth taking. I have surely learned as much from this trip as I did when I encountered APL itself for the first time in the 1960s. In the process of learning a lot, I also find myself in possession of some programming tools that I use almost daily in practical ways that range from quick simulation models to HTML screen scraping tasks.
	Summary	I am unclear about whether J and K are truly languages of the future. What I am clear about, however, is that they are both enormously useful and effective languages---well worth the amount of energy that I had to invest to learn how to use them. I am equally clear that APL is a language headed for oblivion. I cannot speak to the issue of whether it is worth you investing the energy to give yourself an alternative or not. I can say, however, that it certainly was worth my effort for me. And one of the lessons that I have learned from years of experience in this area is that it's very hard to make a profit by second guessing the smart guys. In this case, at least, I decided to avoid that problem and have, so far, been quite happy that I did.