Building and Debugging FRP with CodeWorld and Reflex

Two weeks ago, I wrote about FRP with Reflex and CodeWorld. The integration has been popular, and I’m honored and thrilled that people find it useful and interesting.

(Aside: By the way, I neglected to mention last time that Joachim Breitner proposed this idea, and implemented it two years ago. I wasn’t able to merge his code at the time, because it was blocked on a Reflex Hackage release that took a long time. Ryan and I implemented the idea again at the NYHaskell CoHack, and I’ve been developing it further since then.)

However, the first implementation didn’t go as far as I wanted to.

• While Reflex helped with composability, the CodeWorld API didn’t live up to the goal.
• Reflex-based programs lacked the scene graph inspection features of the traditional CodeWorld APIs.
• Reflex-based programs were missing the on-screen exploratory controls that are available for traditional CodeWorld APIs.

I’m now announcing some changes to CodeWorld’s Reflex integration to address these problems. I’ll show you how the system has changed, and what you can do now that you couldn’t do before.

Composability and the builder monad

I was still brand new to FRP when I started this project. While I understood the core abstractions — Event, Behavior, and Dynamic, which I introduced in the last post — I did not appreciate some of the more structural choices that the reflex-dom library works.

Recall from last time that the key advantage of programming in FRP style, versus the simpler functional MVC style, is that the program can be assembled from well defined components that have inputs and outputs. (By comparison, in functional MVC style, a single interaction must be sprinkled through several parts of the program: its state must be added to the state type, it must be initialized in the initial state, its event handlers must be added to the program-wide event handling code, and its visualization must be added to the overall picture!) As I started to actually build non-trivial structured programs in CodeWorld’s Reflex integration, though, I found myself feeling like I hadn’t really escaped that world.

The original API for CodeWorld Reflex was this:

reflexOf :: (_ => ReactiveInput t -> m (Dynamic t Picture)) -> IO ()

Here, the inputs to the CodeWorld program are passed in, and one single picture is returned. The plumbing to thread the input everywhere, and especially to combine the resulting pictures everywhere was simply too much. The plumbing of output pictures obscured the way that higher-level interactions between components of the program were wired together.

The new API I’ve defined looks like this:

class ReflexCodeWorld t m
getKeyPress :: ReflexCodeWorld t m => m (Event t Text)
getKeyRelease :: ReflexCodeWorld t m => m (Event t Text)
getTextEntry :: ReflexCodeWorld t m => m (Event t Text)
getPointerClick :: ReflexCodeWorld t m => m (Event t Point)
getPointerPosition :: ReflexCodeWorld t m => m (Dynamic t Point)
isPointerDown :: ReflexCodeWorld t m => m (Dynamic t Bool)
getTimePassing :: ReflexCodeWorld t m => m (Event t Double)
draw :: ReflexCodeWorld t m => Dynamic t Picture -> m ()
reactiveOf :: (forall t m. ReflexCodeWorld t m => m ()) -> IO ()

Instead of passing an input object as an argument, the low-level input controls are available in the ambient monad. In practice, it was easier to just pass around the entire set of inputs anyway: it’s not really the caller’s business whether a control uses one or another physical input event. And instead of returning Dynamic t Picture as a result, the draw action can be used by any code running in the picture-builder monad to emit pictures to the screen.

This is similar to reflex-dom, too. There, if you want to add a checkbox or button to the screen, actually adding the button itself is handled as an effect in the DOM-builder monad, so that FRP values are only used for communicating higher-level interactions with the widget.

An example of this new API is here. Go ahead and play with it to see how it behaves.

{-# LANGUAGE OverloadedStrings #-}
import CodeWorld.Reflex
import Data.Text (Text)
import Reflex
main :: IO ()
main = reactiveOf $ do
    up <- button "Up" (0, -4)
    down <- button "Down" (0, -8)
    left <- button "Left" (-4, -6)
    right <- button "Right" (4, -6)
    pos <- foldDyn ($) (0, 0) $ mergeWith (.) [
        (\(x, y) -> (x, y + 1)) <$ up,
        (\(x, y) -> (x, y - 1)) <$ down,
        (\(x, y) -> (x + 1, y)) <$ right,
        (\(x, y) -> (x - 1, y)) <$ left
        ]
    draw $ uncurry translated <$> pos <*> pure (solidCircle 0.2)
    return ()
button :: ReflexCodeWorld t m => Text -> Point -> m (Event t ())
button label (bx, by) = do
    draw $ constDyn $ translated bx by $
        dilated 0.75 (lettering label) <>
        rectangle 3 1 <>
        colored (light gray) (solidRectangle 3 1)
    click <- getPointerClick
    return (() <$ ffilter onButton click)
  where onButton (x, y) = abs (x - bx) < 1.5 && abs (y - by) < 0.5

This is a very minimal rendition of a button, but notice that once it’s been defined, button nicely captures the logical structure of a button: it needs a label and a point, and produces an event indicating it has been clicked. All of the details of a button, such as ensuring that it has been drawn to the screen and that it listens for mouse clicks, are handled as details inside its implementation. Because of this, main reads nicely as a description of which buttons appear where, and how they affect the important state: the position of the movable dot.

It’s interesting to consider what happens if I want a fancier button that turns a different shade when one is hovering over it. I can just change the implementation to add this new feature:

button :: ReflexCodeWorld t m => Text -> Point -> m (Event t ())
button label (bx, by) = do
    hover <- fmap onButton <$> getPointerPosition
    let color = bool (light gray) (light (light gray)) <$> hover
    draw $ pic <$> color
    click <- getPointerClick
    return (() <$ ffilter onButton click)
  where
    onButton (x, y) = abs (x - bx) < 1.5 && abs (y - by) < 0.5
    pic color = translated bx by $
        dilated 0.75 (lettering label) <>
        rectangle 3 1 <>
        colored color (solidRectangle 3 1)

In the functional MVC model, such as used by traditional CodeWorld, Gloss, Elm, or Racket’s universe model, something like this wouldn’t be so localized! The button needs new state to know whether it’s hovered over, so a change must be made in a data type for the state. New event handlers must be added to change the hover state in response to mouse-movement events. And the picture rendering must be updated, as was done here. Using FRP makes this painless, and the new ReflexCodeWorld class lets you encapsulate even more and stay focused on the application logic rather than plumbing event and picture values.

More Effects

Reflex offers a few more options, as well, for effects in the builder monad. I’ve also extended CodeWorld to implement some of these: most notably, the type classes PerformEvent, Adjustable, and PostBuild.

The PerformEvent type class lets you schedule effects to happen as a result of events in the FRP system. It’s basically MonadIO for FRP! Using this, we can cheat a little bit, and build CodeWorld programs that escape the canvas!

{-# LANGUAGE JavaScriptFFI #-}
import CodeWorld.Reflex
import Control.Monad.Trans (liftIO)
import Reflex
foreign import javascript unsafe "alert($1 + ',' + $2)"
    alert :: Double -> Double -> IO ()
main :: IO ()
main = reactiveOf $ do
    click <- getPointerClick
    performEvent_ $ (\(x, y) -> liftIO $ alert x y) <$> click

The special performEvent_ takes an Event of actions to perform, and performs them when the event fires. The actions are in a monad with a MonadIO instance, so among other things, they can perform I/O as you can see here.

The first use I was able to make of this new PerformEvent monad was to use it to handle drawing the screen and interacting with CodeWorld’s debug interface (see below). This helped me to move more of the code into the FRP fold.

Adjustable extends PerformEvent with the ability to not just perform actions, but even rebuild parts of the FRP network itself. This can be handy when you’re dealing with dynamic data and want to create controls for each element of a list, or something like that. I don’t recommend that beginners try to use Adjustable on its own, but there are some higher-level functions built on this in the “Collection management” section of the Reflex Quick Reference.

Finally, PostBuild just offers an event that fires as soon as the builder monad has completed building. Again, I don’t recommend using it directly, but it’s used in some higher-level functions in the Quick Reference.

(Another type class that’s part of Reflex, but not yet integrated into CodeWorld is TriggerEvent, which allows you to create an Event together with an action that fires it. This can be seen as the complement of PerformEvent. Perhaps there’s some value to adding this class, but in general CodeWorld programs don’t have significant logic that lives outside of CodeWorld, so it’s less useful. The main example in the Quick Reference has to do with creating a time ticker, and CodeWorld offers a better one using requestAnimationFrame anyway, so I’m not in a hurry.)

FRP and Debugging

Two debugging features were also missing from the Reflex API.

The Inspector is a feature, accessed through the purple Inspect button on a running program, lets you browse the scene graph: the structure of primitive shapes, transformations, etc. that make up the screen. It even links from the screen and the tree view to the relevant source code. Originally implemented by Eric Roberts as part of Summer of Haskell a few years ago, this is a really powerful feature for debugging graphics programs.

The CodeWorld Inspector

The implementation of this feature was specific to traditional CodeWorld entry points, and the new Reflex integration originally didn’t offer the inspector. That’s now changed. If you create a program with CodeWorld’s reflexOf entry point, you will see the “Inspect” button on the bottom toolbar, and opening it will show you the scene graph and and let you navigate the picture exactly as you can with the traditional CodeWorld library.

A second debugging feature is CodeWorld’s on-screen controls. These controls give a user the ability to zoom in or out, pan, pause, fast-forward, and so on, with a running program. While the Inspector is great for examining the parts of the results, the on-screen controls are useful for exploring the behavior of the whole program. These, too, were implemented in a way that was specific to traditional CodeWorld entry points.

On-screen debug controls in a CodeWorld program

I’ve now implemented basic on-screen controls for Reflex programs, as well. To try it out, you’ll need to:

1. Write your code to the new reactiveOf instead of the old reflexOf entry point.
2. Actually, use debugReactiveOf instead of reactiveOf. This has the same type, so it’s just six characters of changes, but it enables the on-screen controls.

I actually had to reimplement this feature entirely, as the previous implementation was tied very closely to the functional MVC architecture. This was a non-trivial implementation effort with some interesting challenges, and it has guided my learning about how to build UI code in Reflex with better composition and modularity.

I wasn’t able to entirely reproduce the earlier on-screen debugging feature set. Certain controls depended on some knowledge of the state: either they kept a state log, or they requires a specific state type. The quintessential example here is the time-traveling debugger that Krystal Maughan wrote last summer, which keeps a log of past states, and allows a developer to scroll through them and find the exact point something went wrong! This is the cost of Reflex’s increased modularity, I suppose: since the complete set of program state is no longer part of the program’s type system, it’s not accessible or visible for inspection and debugging in the same way that functional MVC is. Perhaps there’s a solution to that, but it’s not an obvious one.

What else?

I’m still not done. There are a few details I’m still figuring out.

1. Performance isn’t where I’d like it to be. The new builder monad seems to do a lot of allocations, and the frame rate starts to get choppy toward the end of a garbage collection interval.
2. One challenge is supporting common UI idioms, such as keyboard focus, or propagation controls for events between controls. I haven’t got a good answer. I played around with a scheme for effectfully taking and filtering out events, so that a control could claim a mouse click and other controls wouldn’t see the same trick. But getting that to work with both MonadFix and Adjustable in useful ways is not easy. (It took me a while to reach this intuition, but Adjustable is largely incompatible with effects that are observable inside the network. DynamicWriterT is the answer to this, but it’s limiting.)
3. There are lots of places where I’d like to modify both the inputs and outputs to the program in a local scope. I’m thinking about API options for this, which would be added to the ReflexCodeWorld type class.
4. I am getting increasingly interested in putting together a more detailed intro to FRP and Reflex using CodeWorld, that walks through the parts with more deliberate examples and exercises and structure. I don’t know if I’m quite qualified to write this, yet, but I am definitely at the stage where I’m excited about the prospect of it existing.

In any case, this has been a great project for me. I’ve got a moderately complex bunch of Reflex code totaling nearly a thousand lines and tackling some non-trivial challenges. I’m learning more than I expected, and having a lot of fun. Here’s to finding out what happens next!