heredocs in lisp

Shell script and Perl have heredocs. They are handy because you don't have to worry about quoting meta characters in a large body of data.

(Depending on which flavour you use, of course. You can interpolate variables if you wish.)

Extracted from Lisp newbie,

You could probably cook up a funky reader-macro to do heredocs in lisp. Probably a good first exercise in reader-macro-land.

Is it a good idea?

[before finding it can be done] Maybe the answer is "that would be pointless". Still, with perl I've grown accustomed to being able to paste a chunk of stuff into the source and then mess with it (for example, outdent it; this allows it to be indented in the source, which aids legibility). Indeed, being able to read the inline POD can be handy too.

I suspect that this is caused by "thinking in strings" instead of lists. Still, if I were writing HTML-from-sexpr source I would rapidly grow tired of "quoting strings".

Implementing it

Well since it is "probably a good first exercise", I won't ask you to show me how. Watch this space (but not too hard).

This is actually not as easy as it looks. For example:

    (list #>END 2 3
  some text
  END
   4)

should read as

(list "some text\n" 2 3 4)
. But to do that, you need to have the ordinary reader take over again between the marker and the start of the here-doc.

Lisp already has multi-line strings

Standard "string" syntax can contain literal newlines - as often used for function docstrings.

* (defvar *foo* "This is
  my sample
  multiline string!
  
  Whee.")

*FOO* * (write-line *foo*) This is my sample multiline string!

Whee. "This is my sample multiline string!

Whee." *

However this does require literal " and \ to be backslashed.

Lisp already has balanced comment syntax

The standard reader treats #| ..stuff.. |# as a comment. See http://www.lispworks.com/documentation/HyperSpec/Body/02_dhs.htm for details.

Is it Lispy to extend existing #| and ## reader notation in a form such as

(list
#1|put some text here
with whatever "quotes" you need
|1# 'and #1# "use it" #1# "often")
?

Advantage: it may be easier to teach editors what's going on, if they already understand #|...|# .

SBCL currently gives "WARNING: A numeric argument was ignored in #1|."

Motivation

[rework in progress on this unsigned text]

Also, as far as I can tell the motivation for the heredoc syntax is the ablity to do stuff like this in shell (note the redirection):

sed-or-awk <<EOF < source > target
...script...
END

There the clarity of having the redirection on a single line is worth something, but I really wonder if this is worth anything in lisp.

I'm well aware that Lisp has multi-line strings -- my point was just that although this is labeled as a "my first readtable" exercise, it's not even clear to me that it's possible without essentially writing your own reader. And while it's nice to be able to say

(let ((files #>this #>that #>the-other)) ... )
...
this
... ...
the-other
it's probably not worth the effort.

It's not much effort, if I correctly understand what you want here. Here's a quick off-the-cuff implementation (untested...may have typos, etc.)

 (defun read-string-to (terminal stream)
  (let ((current 0))
    (with-output-to-string (out)
      (loop
	(let ((char (read-char stream t nil t)))
	  (if (char= char (char terminal current))
	      (when (= (incf current) (length terminal)) (return))
	      (progn
		(write-string terminal out :start 0 :end current)
		(if (char= char (char terminal 0))
		    (setq current 1)
		    (progn
		      (setq current 0)
		      (write-char char out))))))))))

(defun read-heredoc (stream char arg) (declare (ignore arg)) (read-string-to (read-string-to (string char) stream) stream))

(set-dispatch-macro-character #\# #\> #'read-heredoc)

Just type #>foo> ... foo

Not quite...

* #>foo>
bar
foo

" bar " * (list #>foo> 1 2 3) bar foo C-c C-c ;; (i.e. interrupt it).

I know you can make a concatenated-stream out of the rest of the line plus the stream after the end-marker, but I can't think of a way to make the reader continue reading from this new stream instead of the one it's working on (short of hacking the SBCL source, that is). Once again, it's not that important, it's just not a newbie exercise.
Huh? The above example works perfectly, you just didn't finish your list. You wrote the equivalent of
(list " 1 2 3)
bar
"
What you should have written was
(list " 1 2 3)
bar
")
Note the closing paren. In your example, that's the same as
(list #>foo> 1 2 3
bar
foo)
Heredocs are a bit stranger than that:
~/src% perl
print <<a, <<b, "3\n";
text
a
more text  
b
^D
text
more text
3
~/src%

This is what makes them more than just fat quotation marks -- you can tersely include a large block of text as a string in the middle of an expression. One way to do this might be to have the heredoc dispatch macro just record the heredoc's name and an associated gensym, then install a reader macro for newline that would read in the document contents, and set the gensym's value to the text.

No you can't. You need to be able to detect the end of the current top-level form and start reading from there, not just the next newline; and that wouldn't actually work, either, because you'd need to somehow delay evaluation of the form until after the values were set (in case it was being typed at the REPL or loaded as source). I.e., it'd all be very ugly. ISTM better to use something like this as-is, with LET to bind the variables first. But maybe READ-HEREDOC should return a string-input-stream rather than a string -- in the shell, using <<END is redirecting stdin, not quoting a string. Also, it should put a newline at each end of the terminator string, so that it only ends when that string occurs on a line by itself.


Similar features elsewhere

CHICKEN ("A practical and portable Scheme system") has #<<FOO documented as non-standard read syntax. It works as a perl or shell programmer would expect (possible omission of trailing \n?). This is not Common Lisp, I only mention it for interest.

This page is linked from: Lisp newbie  

CLiki pages can be edited by anyone at any time. Imagine a fearsomely comprehensive disclaimer of liability. Now fear, comprehensively