Abstract: Functional HTML5 and XML library for the Scala platform

Tree construction

Unless otherwise stated, all code samples require a prior import pine._.

DSL

Pine offers a DSL that allows to create trees in terms of immutable objects:

val a = tag.A
  .href("http://github.com/")
  .set(Text("GitHub"))

a.toHtml  // <a href="http://github.com/">GitHub</a>

The bindings are derived from the MDN documentation. For all attributes, we provide getters and setters. set() replaces the children of a node.

Macros

Pine defines a couple of compile-time macros for increased comfort or performance.

Inline HTML

Inline HTML can include placeholders referring to Scala values:

val url   = "http://github.com/"
val title = "GitHub"
val root  = html"""<a href=$url>$title</a>"""

root.toHtml  // <a href="http://github.com/">GitHub</a>

A placeholder may also reference Seq[Node] values:

val spans = Seq(
  html"<span>test</span>",
  html"<span>test2</span>"
)

val div = html"<div>$spans</div>"
div.toHtml  // <div><span>test</span><span>test2</span></div>

External HTML

For loading external HTML files during compile-time, a constant file name must be passed:

val tpl = html("test.html")
tpl.toHtml  // <div>...</div>

Runtime HTML parser

Pine provides an HTML parser with the same semantics on all backends:

val html = """<div id="a"><span>42</span></div>"""
val node = HtmlParser.fromString(html)
node.toHtml == html  // true

HTML code is parsed during compile-time and then translated to an immutable tree. This reduces any runtime overhead. HTML can be specified inline or loaded from external files.

The parser has the following limitations:

• The DOCTYPE tag is ignored
• The input is expected to be valid HTML5

The parser supports the complete set of more than 2100 HTML entities such as " as well as numeric ones ("). These entities can be decoded using the function HtmlHelpers.decodeEntity(). If you would like to decode a text that may contain such entities, you can call decodeText() instead.

XML

XML has slightly different semantics with regards to self-closing tags. The following example is valid XML, but would yield a parse error when parsed as HTML:

<item><link></link></item>

Also, the typical XML header <?xml is not valid HTML. In order to parse documents in the XML mode, use XmlParser or the xml string interpolator, respectively:

XmlParser.fromString("""<item><link></link></item>""")

xml"""
  <?xml version="1.0" encoding="UTF-8"?>
  <rss version="2.0"
    xmlns:atom="http://www.w3.org/2005/Atom"
    xmlns:dc="http://purl.org/dc/elements/1.1/">
    <channel>
      <atom:link type="application/rss+xml" />
    </channel>
  </rss>
"""

As per the XML specification, Pine supports only the following four entities:

• &apos; (')
• &lt; (<)
• &gt; (>)
• &amp; (&)

The underlying data structures are the same for HTML and XML trees. Pine strives for simplicity and performance at the cost of implementing only a subset of XML's features. Please refer to scala-xml for a more complete implementation.

At the moment, we are aware of the following parser limitations:

• The XML header is optional and its attributes are ignored. The input is expected to be in UTF-8 regardless of the specified character set.
• DTDs are not supported. Therefore, additional entity or element declarations cannot be defined.
• Processing instructions (other than <?xml ... ?>) are not supported

Conversion

Some functions return Tag[_] when the tag type cannot be statically determined. A more concrete type is useful if you want to access element-specific attributes, like href on anchor nodes. You can use as to convert a tag to its correct type:

val tag = html"<div></div>"
val div = tag.as[tag.Div]

Unlike asInstanceOf, this function ensures that the conversion is well-defined.

Custom tags

So far, we have used elements from the tag namespace. For each HTML element, Pine defines a type and an empty instance, i.e. without attributes and children. If you want to support a new element such as <custom-type>, you could define it as follows:

type CustomType = "custom-type"
val  CustomType = Tag("CustomType")

The compiler feature we use here are literal types. Originally developed within Typelevel Scala, it is now part of Lightbend Scala 2.13 onwards.

Additionally, you can define methods to access attributes conveniently:

implicit class TagAttributesCustomType(tag: Tag[CustomType]) {
  val myValue = TagAttribute[CustomType, String](tag, "my-value")
}

Now, you can access and modify your custom HTML element while preserving type-safety:

val tag = html"""<custom-type my-value="value"></custom-type>"""
val ct  = tag.as[CustomType]
ct.myValue()                 // value
ct.myValue("value2").toHtml  // <custom-type my-value="value2"></custom-type>

Note that the type definition above is optional and you could also write the literal type directly:

val ct2 = tag.as["custom-type"]

TagAttribute takes an implicit AttributeCodec. If you would like to enforce more type-safety in attributes, you could define an enumeration and create an AttributeCodec instance for it:

sealed abstract class Language(val id: String)
object Language {
  case object French  extends Language("french")
  case object Spanish extends Language("spanish")
  case object Unknown extends Language("unknown")
  val All = Set(French, Spanish)
}

implicit case object LanguageAttributeCodec extends AttributeCodec[Language] {
  override def encode(value: Language): Option[String] = Some(value.id)
  override def decode(value: Option[String]): Language =
    value.flatMap(id => Language.All.find(_.id == id))
      .getOrElse(Language.Unknown)
}

implicit class TagAttributesCustomDiv(tag: Tag[pine.tag.Div]) {
  val dataLanguage = TagAttribute[pine.tag.Div, Language](tag, "data-language")
}

tag.Div.dataLanguage(Language.Spanish)

Rendering

A node has several rendering methods:

• HTML: toHtml is defined on every node and will return the tree as an HTML5 string. If the root node is an <html> tag, the DOCTYPE will be included as well.
• XML: toXml returns the tree as an XML 1.0 string. It always includes the XML header, specifying the encoding as UTF-8.
• DOM: toDom is only available in JavaScript. It renders the tree as a browser node, which can be inserted into the DOM.

Tree updates

Operations

A Node is equipped with a variety of functions to easily manipulate trees such as prepend, append, remove, clearAll, filter, flatMap, map and others. See the source code for an overview.

Referencing nodes

While the operations from the previous section allow you to modify the tree, they operate on either the root node or are applied recursively to all children.

If you would like to update a specific child further down in the hierarchy, Pine introduces the concept of tag references (TagRefs). These have the advantage that changes can be batched and applied efficiently.

In order to do so, you need to make the nodes you would like to reference identifiable, for example by setting the id attributes:

val node = html"""
  <div id="child">
    <span id="age"></span>
    <span id="name"></span>
  </div>
"""

Now you can reference these nodes using TagRefs. A TagRef takes the referenced tag's ID and HTML type:

val spanAge  = TagRef[tag.Span]("age")
val spanName = TagRef[tag.Span]("name")

There are more ways to reference nodes such as by class name or tag type. See section "Tag references".

Updating nodes

You can use the update() method to change the node:

val result = node.update { implicit ctx =>
  spanAge  := 42
  spanName := "Joe"
}

The changes (diffs) take an implicit rendering context. When you call update(), the changes will be queued up in the rendering context and processed in a batch.

result will be equivalent to:

<div id="child">
  <span id="age">42</span>
  <span id="name">Joe</span>
</div>

Replacing nodes

If you would like to replace the node itself, you can use replace():

val result = node.update { implicit ctx =>
  spanAge .replace(42)
  spanName.replace("Joe")
}

result will be equivalent to:

<div id="child">
  42
  Joe
</div>

Updating children

val node = html"""<div id="page"></div>"""
val root = TagRef[tag.Div]("page")

In order to render a list, you can use the := function (alias for set):

root.update { implicit ctx =>
  root := List(
    html"<div>Hello, </div>",
    html"<div>world!</div>"
  )
}

But if you would like to later access those child nodes they need unique IDs. This is particularly useful when you render your HTML on the server and want to access it in JavaScript, e.g. in order to attach event handlers.

First, we define a data type we would like to render:

case class Item(id: Int, name: String)

Next, we define a function that returns a child node given an item.

val itemView = html"""<div id="child"><span id="name"></span></div>"""
def idOf(item: Item): String = item.id.toString
def renderItem(item: Item): Tag[_] = {
  val id   = idOf(item)
  val node = itemView.suffixIds(id)
  val spanName = TagRef[tag.Span]("name", id)
  node.update(implicit ctx => spanName := item.name)
}

Finally, we render a list of items using the set method.

val items  = List(Item(0, "Joe"), Item(1, "Jeff"))
val result = node.update(implicit ctx => root.set(items.map(renderItem)))

result will be equivalent to:

<div id="page">
  <div id="child0">
    <span id="name0">Joe</span>
  </div>
  <div id="child1">
    <span id="name1">Jeff</span>
  </div>
</div>

Now, we can reference child nodes using a TagRef:

TagRef[tag.Div]("child", idOf(items.head))  // TagRef[tag.Child]("child0")

Updating attributes

As our TagRef objects are typed, we can provide implicits for supported attributes.

val node = html"""<a id="lnk">GitHub</a>"""
node.update(implicit ctx =>
  NodeRef[tag.A]("lnk").href := "https://github.com/"
)

Tag references

Tags can be referenced using:

• ID attribute: TagRef[tag.A]("id")
• Tag type: TagRef[tag.A]
• Class name: TagRef.byClass[tag.A]("class-name")

A TagRef exposes methods for manipulating nodes and their attributes. See its source code for a full list of operations.

Diffs

A Diff is an immutable object which describes tree changes. It is instantiated for example by the TagRef operations you have seen before such as := (set), replace etc.

So far, these changes were performed directly on the tree. However, for the JavaScript back end, we have an additional rendering context that can apply those changes to the DOM. This will be explained in the next chapter.

The full list of supported diffs can be found here.

Multiple occurrences

If you would like to perform a change on all occurrences of a TagRef, use the each function:

val div = html"""<div><span></span><span></span></div>"""
div.update(implicit ctx =>
  TagRef["span"].each += html"<b>Hello</b>").toHtml
// <div><span><b>Hello</b></span><span><b>Hello</b></span></div>

each can also be used in conjunction with any other diff type, such as attribute updates:

val div  = html"""<div><a href="/a">A</a><a href="/b">B</a></div>"""
val html = div.update(implicit ctx =>
  TagRef["a"].each.href.update(_.map(url => s"$url/test"))).toHtml
// <div><a href="/a/test">A</a><a href="/b/test">B</a></div>

HTML/CSS extensions

Pine's DSL provides extensions to facilitate interaction with HTML/CSS. For toggling the visibility of a node, you can use hide():

div.hide(true)  // Sets `style` attribute to hide the element in the browser

Token list attributes

There are certain HTML attributes whose values are encoded as space-separated tokens. class and rel are the most prominent examples.

These attributes have a special mapping in Pine that models their underlying sequential nature:

tag.Div.`class`("a", "b")           // Sets the classes "a" and "b"
tag.Div.`class`.set(Seq("a", "b"))  // Same as before
tag.Div.`class`.get                 // Returns the list of classes
tag.Div.`class`.add("a")            // Adds the class "a"
tag.Div.`class`.remove("a")         // Removes the class "a"
tag.Div.`class`.clear()             // Removes all classes
tag.Div.`class`.toggle("a")         // Toggles the class "a"
tag.Div.`class`.state(value, "a")   // Adds the class "a" if value is true, remove otherwise
tag.Div.`class`.update(_ :+ "a")    // Updates the classes

The same functionality is available on TagRefs.

Custom attributes

If you would like to support custom attributes, you can extend the functionality of any tag by defining an implicit class. This is the same approach which Pine uses internally to define attributes for HTML elements.

For example, to define attributes on anchor nodes, you would write:

implicit class TagRefAttributesA(tagRef: TagRef[tag.A]) {
  val dataTooltip = TagRefAttribute[tag.A, String](tagRef, "data-tooltip")
  val dataShow    = TagRefAttribute[tag.A, Boolean](tagRef, "data-show")
}

Web development

Pine may be used for web development. You can use it in various architectures, for example:

• Client-side rendered pages (SPA, single-page applications)
• Server-side rendered pages without client logic
• Server-side rendered pages with client logic
• Server-side rendered pages with client logic and client-side rendering

Please refer to our sample project which implements the last architecture.

Server refers to either the JVM or LLVM back end, whereas client refers to JavaScript.

All examples require a prior import pine.dom._.

Architectures

Pine advocates web development in the FP style. You are advised to split your HTML rendering into composable functions and share the code across platforms. Pine does not provide any abstractions for pages or components to maximise its use cases.

The fourth architecture is the most sophisticated and allows for the best user experience. For this, you have to define a shared protocol for the data layer as well as shared code for populating the templates. On the client, you evaluate which page the server rendered and then attach the event handlers. Also, when the user clicks an internal page link, instead of redirecting to it, you can use the shared template layer to perform the rendering directly in the browser.

This architecture has the following life cycle for a page p, which you could define in terms of four functions:

1. node(p): Creates an immutable tree node (shared project)
2. populate(p): Populates the tree with content (shared project)
3. attach(p): Attach event handlers, only called in JavaScript (js project)
4. detach(p): Detach event handlers, only called in JavaScript (js project)

Render JavaScript node

To render a Pine node as a JavaScript node, use the function toDom:

val div    = html"""<div><input type="text" /><input type="test" /></div>""".as[tag.Div]
val jsNode = div.toDom  // dom.html.Div

toDom returns the correct JavaScript type depending on your node type:

Text("test").toDom  // dom.raw.Text

You need to add the JavaScript node manually to the DOM to be able to access it via a TagRef:

dom.document.body.appendChild(jsNode)

Access DOM node

Use dom on a TagRef to access the underlying DOM node:

val text = TagRef[tag.Div]("text")
text.dom  // Returns browser node, of type org.scalajs.dom.html.Div

If you would like to retrieve all matching nodes, use each and domAll instead:

val input = TagRef[tag.Input].each
input.domAll  // List[org.scalajs.dom.html.Input]

Access DOM attribute

val text = TagRef[tag.Div]("text")
text.`class`.get  // Retrieves 'class' attribute from DOM node, of type Option[String]

Note that in JavaScript, DOM attributes may not represent the current state of a node. If this is the case, you can retrieve the value via dom:

val name = TagRef[tag.Input]("name")
name.value.get  // Returns value the DOM node was initialised with
name.dom.value  // Returns current value

Converting JavaScript nodes

It is also possible to convert regular DOM nodes to Pine:

val node = dom.document.createElement("span")
node.setAttribute("id", "test")
node.appendChild(dom.document.createTextNode("Hello world"))

DOM.toTree(node)  // Tag(span,Map(id -> test),List(Text(Hello world)))

Diffs

Previously, we used update to perform the changes on the nodes. To carry out the changes in the DOM, we have to use DOM.render:

DOM.render(implicit ctx => text := "Hello, world!")

Events

As an extension to content updates, you can set event handlers. In JavaScript projects, a TagRef exposes all event handlers which the underlying DOM element supports. These changes are side-effecting and therefore do not require a rendering context. The motivation is that event handlers do not change the visual page content. Therefore, instantiating Diffs and performing a batch execution would be redundant.

val btnRemove = TagRef[tag.Button]("remove")
btnRemove.click := println("Remove click")

It is possible to attach an event to all matching elements using each:

val input = TagRef[tag.Div].each
input.click := println("Any div was clicked")

See also dom.Window and dom.Document for global events.

Troubleshooting

Dangling rendering context

If you encounter a Dangling rendering context exception, this may be reminiscent of a dangling pointer in C. The underlying problem is the same: You set up a rendering context passing it a function, which adds diffs to the context. After this function returns, the diffs are processed. Now, the rendering context should not be used anymore.

Most likely an asynchronous event took place which re-used the implicit context from the scope and added a diff to it. One such example is:

DOM.render { implicit ctx =>
  button.click := box.hide(true)
}

When the button was clicked, hide will re-use ctx. The following fixes the situation:

button.click := DOM.render(implicit ctx => box.hide(true))

You can safely nest multiple DOM.render blocks. The inner-most block will always use the context from the immediate scope. It is advisable to limit the rendering context only to functions that change the DOM and take an implicit context. This could prevent problems as above since an implicit context would not have been found in the first place.

IntelliJ support

When loading the sample projects in IntelliJ, some references in the shared module may not be resolved properly. This happens because IntelliJ doesn't add a dependency from platform-specific modules (i.e. jvm and js) to the shared module.

To fix this, please go to File -> Project Structure... -> Modules -> project-Sources -> Dependencies. Then, add a module dependency to projectJVM/projectNative and projectJS.

Development

Benchmarking

The benchmark suite measures the performance of Pine's core functionality. It is available for the JVM and JavaScript back ends.

The benchmarks can be run on the JVM as follows:

bloop run pine-bench-jvm -- fast  # Fast profile
bloop run pine-bench-jvm -- slow  # Slow profile

The JavaScript suite requires two dependencies to be installed:

yarn add jsdom object-sizeof

If you are compiling to tmpfs, link the node_modules folder:

ln -s $(pwd)/node_modules /tmp/build-pine/node_modules

Then, run the suite with Node.js:

bloop run pine-bench-js -- fast  # Fast profile
bloop run pine-bench-js -- slow  # Slow profile

Since Node.js is lacking a DOM implementation, several benchmarks will be omitted. However, you can run the full benchmark suite in the browser. This requires the webpack dependency:

yarn add webpack webpack-cli

Next, create an HTML file in the build folder (e.g. /tmp/build-pine/):

<!doctype html>
<html>
	<head>
		<meta charset="utf-8">
	</head>
	<body>
		<script src="main.js"></script>
	</body>
</html>

And the webpack configuration webpack.config.js:

const path = require('path');

module.exports = {
  entry: './pine-bench.js',
  output: {
    filename: 'main.js',
    path: __dirname
  }
};

Finally, link the benchmark suite and bundle all external dependencies into a single JavaScript file:

bloop link pine-bench-js
yarn exec webpack --config webpack.config.js

You can now open the HTML file in the browser. The results will be printed to the browser console.

To detect performance regressions, the benchmarks are run with JVM and Node.js as part of every CI build. Since the benchmark suite uses the fast profile to speed up CI runs, it is advisable to also run the benchmarks locally in the slow profile. Similarly, the DOM benchmarks are not run as part of CI and should be tested manually in a browser.

Development

Manual

The manual was generated using Instructor. Follow its installation instructions, then run the following command:

instructor manual.toml

Generated with Instructor v0.1-SNAPSHOT