Programming, Software and Code

> "We should not be so foolhardy as to think that we can't have made a fundamental mistake with PASM."

PASM might not be a mistake but perhaps the assumptions Parrot is based on?

> "The answer is that they couldn't have known what we know now about VM design."

I'm not sure that this is true, who gave the link to
ftp://ftp.create.ucsb.edu/pub/Small​talk/Squeak/docs/OOPSLA.Squeak.html for what purpose?

> "the experience as a community to write a higher-level language compiler"

I hesitate to point to the awful truth, but ... almost none of the HL-languages targeting Parrot are close to being complete (useful, efficient, ...) but far away from it. Most of them are "dormant", "inactive", "retired" or just toys. https://trac.parrot.org/parrot/wiki/Languages

I'm a bit torn on this whole thing. I can see tremendous advantages on a simpler low level language IF:


1) the operations it performs have a contract that makes it clear when machine-specific features are in use inside an op, so JIT knows it's OK to use them.

2) ops do not need to pass everything through a return/parameter hoop. They should be able to float some registers -- both abstract L1 registers and architecture feature registers -- to the next operation. And JIT needs not to stomp on those either.

3) ops should be coroutines, in that a single opcode would normally be invoked multiple times to complete a task in steps, but if JIT is told how, it could also

4) ops should, except for very rare cases, restricted to a very small runtime. That means if you are for example strncmp'ing two strings in an op, and they are big enough to cause huge latency, your op should yield back somewhere in there and continue the job.

...but all 1,2,and 4 of those suggestions were sort of met with "meh, we don't need that." But you do. If you want any sort of control over the applications latency, something, somewhere has to step in and tell developers "this is how much work you can do in one chunk, anything longer has to be split up." Machine language and microcode both come with cycle counts in the documentation and a complete and thorough description of every side effect and possible outcome. L1 could be an opportunity to bring the code and documentation up to this standard. But it doesn't sound like that's what's on the agenda.

There are also a lot of advantages to moving away from C for PMC code. For one, a different language could clean up pointer references so it's harder to reference pointers without passing them through GC hoops, if the particular GC that is running wants that to happen.

It could define a framework to actually formally specify the structures we are using -- e.g. these types of structures are always from this type of GC pool, that type of structure is does or does not have a buffer that might be mutated by GC. It could un-uglify working with the internal API and hideall those nasty C-namespace Parrot_subsystem_of_this_other_subsystem_just_friggin_do_it_already()
nonsense.

But is NQP maliable enough to do so? Is that really what Close is aiming at? We don't know that yet.

At the same time I'm not entirely convinced that the barriers presented as arguments against PIR are all that insurmountable. PIR(PASM), as you have noted, is also a pretty compact representation, moreso than L1 would be.

I'd like to encourage those who would advocate for the "let's fix PIR" to suggest how PIR might meet as many of the stated goals of L1 as they can, rather than blow them off as nonconcerns -- because they are well founded criticisms of where we are now.

More good questions!

1) if we use a JIT engine like LLVM or libJIT, those will be able to optimize in machine-specific ways that they know about. I'm not sure if this answers your question.

2) I was actually thinking about this topic today. Do a google search for "context threading" to see ways that ops can be dispatched to help minimize instruction cache hazards. I think for most compilers we could also specify that ops are dispatched with "fastcall" semantics, and hopefully the optimizer will prevent necessary registers from getting clobbered on every dispatch. I was actually going to play with "fastcall" tonight to see if it would yeild any speedups at all.

3) I don't think we need them to be coroutines. Coroutines require a certain notion of saved state which would be too expensive for ops that are called so frequently. A good fast dispatcher to normal subroutines should be sufficient.

4) Parrot doesn't make any hard realtime guarantees, so it doesn't necessarily make sense to make individual ops conform to a particular time profile. This is especially true of "call" ops which can call arbitrary C functions (which themselves could be very expensive).

And why not a human-readable pbc instead of a new language?

My only comment is... given how much investment there's been in PIR/PASM to date, would it not be possible to actually do the "extra stage" approach first, get some PASM/PIR -> L1 translation in place, and look for the other opportunities later on?

Reading this, I kept asking myself, for those pesky low level things you're going to invariably need to do implementing your HLL, if not PIR/PASM, then what? Allusions are made to a yet-to-be-fully-conceived "other" language to implement HLLs in, but PIR makes up the majority any language which can't rely wholy on the builtins provided by parrot.

Sequence I would think least destabilizing, while still allowing for the refactoring of things...

- you get to focus on implementing a single language -> L1 path first
- there is a ton of existing code you can throw at it for testing, tests around those, etc.
- lets you focus mostly on L1 and the VM side implementation at the outset
- once thats in place, nothing to stop you from building out new compiler bits that target L1 directly
- new intermediate languages built "natively" to L1 emerge with aim to be what you implement an HLL against.
- HLL authors flock like druken dwarves to a keg to migrate their languages to exploit these languages, due to the sheer pwnage of their capabilities.

Sure... none of that existing code will be able to take full advantage of what L1 offers, and there may be some speed impacts, but it will all still continue to work, the HLL authors get to enjoy a period of stability while the new approach matures, and migration can be taken in pieces in many cases once it's all ready.

I think a period of said stability is important for building a community around all of this, everyone would love perfection, but for the moment, "working the same as it did last month" is an important milestone you may want to consider for a time, preserve some of the runtime as it is, offer a compatibility path as the new is introduced, and you will get people accumulating around this.

I leave you with a story, myself as a newbie into the *nix world and wanting to pick up some languages within. At the time, I was attracted to TCL, and taugh myself some, and went out to buy a book on it. Got home, sat down, and after running into a few snags, discovered that what I was trying to teach myself wasn't even syntacticly valid on currently shipping TCL versions. I turned then to Perl5 and never looked back, because I wasn't going to bother with something still going through that much flux between releases.

I can't help but feel the same thing every time I take a stab at parrot.

Anyhow, hope this was in some way interesting, from the perspective from someone on the outside who's been looking in for a while.
-

There are two good options right now for this magical "other language" that I am talking about. The first is NQP, the Perl6-like bootstrapping layer that PCT uses to help write compilers. NQP is nice because it uses Perl6 syntax but has no runtime library, so it's very light weight. Because it's modeled on Perl6, it doesn't have a lot of the constructs that we would need to make the L1 conversion (mostly pointer-related operations), but those could be added via some kind of extension.

The second option for this language is Close, which actually got posted online yesterday afternoon. Close is a C-like language that has a very small runtime and is very close to the underlying VM. Because it's modeled on C syntax, I think it would be less of a stretch to extend it to do the pointer stuff that we would end up needing.

The sequence that we are generally looking at following to implement L1 is this:

1) Write PMCs and Ops in NQP or Close or whatever, to figure out what syntax and capabilities those languages would need to have

2) Write a dynamically-loadable library of new L1 ops

3) Modify PCT to be able to output L1 ops instead of normal PASM ops based on a switch. Here we can test that HLLs and other programs work with L1.

4) Start compiling the PMCs and PASM Ops into L1, and testing that

5) Make the final switch: Move the L1 ops into the core, make PCT output L1 by default, replace all PMCs and Ops with their counterparts written in NQP/Close.

Thanks for all the comments!

Fastcall is close to what I'm getting at with the "coroutine" meme. Except using more registers, including the extra goodies of whatever architecture one is running on.

Say we defined abstract families of registers (and a small chunk of immediate state memory.) These would not necessarily be modeled in a traditional fashion like "general" or "fpu" but would likely each be an assorted grab-bag.

Over time we'd get a feel for what would make a good set of typical register families and they would each develop a personality -- e.g. family 1 is typically used for this, family 2 for that... and adjust per-architecture which family certain registers are assigned to.

Now suppose we had a special (non calling) L1 opcode that said: "stay out of these families, JIT, until further notice" or "start using this passing style" This opcode would be inserted before any opcode that did anything unusual.

Now the runcore knows A) it needs to stay clear of that family and B) that family should be treated as in-use for green-thread purposes and in certain scenarios may need a save/restore and C) how to shuffle operands around, if special needs are in force. And that's all available to JIT as well.

The point is the data could be kept in registers between operations. Certain sets of operations could be coded to expect this and use it.

So then lets take a look at one of those unusual types of operations. Say we have an operation that loaded family 2 with a set of data, then yielded, then when called a second time, summed the data in family 2 and placed the result in the family normally used to pass return values.

In a normal invocation, the opcodes would look like this when humanized:

USEFAMILY(2)
SUMWITHFAMILY2(foo, 8)
DONEFAMILY(2)

That would sum 8 values located at foo and the result would reside wherever results normally reside. It would actually call SUM twice to do it, however. The default would be to re-call until there is no yield.

Now suppose we had another opcode designed to work with SUM, that takes the absolute values of things already located in family 2. In this case we could get the sum of absolute values by:

USEFAMILY(2)
YIELD1 SUM(foo, 8)
ABSFAMILY2(8)
SUM(foo, 8)
DONEFAMILY(2)

The YIELD1 opcode modifier tells the runcore to proceed to the next instruction after the yield, instead of continuing to call SUM until it completes. We could have any number of interchangeable middle opcodes to jam between the two sums.

Now this example is not that compelling, but I am sure better ones could be come up with.

> And why not a human-readable pbc instead of a new language?

Such a language already exists: it' PASM!
"PASM is just a text form of PBC", see PDD06.

But I am following the developpement of parrot since 3 years now and I think that Whiteknight is simply right by thinking that PIR/PASM is not the correct abstraction level: the move from PIR to L1 is similar to a move from CISC to RISC.