On coding and tools

Using dry-validation with Grape

2024-03-12T01:03:00+02:00

Intro

Grape is a popular web framework for Ruby, an alternative to Rails when writing an API-only application.

dry-validation is a flexible library for implementing data validation logic, something one tends to need more as their project grows.

When an API deals with a certain entity, the shape of endpoint parameters in it tends to resemble the shape of the entity itself. And if you already have validation rules defined for it somewhere else (or intend to do that later, depending on the order the functionality is built), you might like to use the same language/library/dsl for describing the schema and rules in both cases, to be able to reuse some.

Grape comes with its own terse DSL for describing parameters which starting with version 1.3.0 is implemented using dry-types under the cover. But that doesn’t make it much easier to use dry-validation (a layer above it). Here’s the open feature request.

Problem statement

Our goal here will be to declare the parameters for an API endpoint using a dry-validation contract, have it check and coerce the inputs, and in the case when the validation fails, send the response describe the failure in the same format as the built-in validation does, let’s say, for interoperability.

Here’s an example:

module Apples
  class API < Grape::API
    format :json
    prefix :api

    resource :orders do
      desc 'Create new'
      params do
        requires :order, type: Hash do
          requires :baskets, type: Array do
            requires :color, type: String, values: %w[green red yellow]
            optional :count, type: Integer, default: 10
          end
        end
      end
      post do
        Order.create!(declared(params))
      end
    end
  end
end

Our endpoint creates a nice big order of several baskets of apples, each from one of three varieties.

Here’s a faulty request and the response we get (for two attributes we messed up):

$ curl -H 'Content-Type: application/json' http://localhost:9292/api/orders/ \
       -d '{"order": {"baskets": [{"clor": 10, "count": "red"}]}}'
{"error":"order[baskets][0][color] is missing, order[baskets][0][count] is invalid"}

The error value enumerates all the invalid attributes, printing their full paths.

Custom type

Here’s how this structure can be described with dry-rb:

ColorString = Dry::Types['string'].constrained(included_in: %w(green red yellow))

BasketSchema = Dry::Schema.Params do
  required(:color).filled(ColorString)
  optional(:count).filled(Dry::Types['integer'].default(10))
end

class OrderContract < Dry::Validation::Contract
  params do
    required(:baskets).array(BasketSchema)
  end
end

One unsatisfying approach is to try the documented way to use the custom types by defining self.parse on the type. It will almost work, but there is no way to get the name of the “wrapper” attribute (order in our example):

params do
  requires :order, type: OrderContract
end

And Grape will prepend the wrapper’s name to the returned error message, which seems difficult to print right when there are several validation errors within our structure. So self.parse seems to be better suited for “leaf” values.

Simple delegation

Okay, referencing the contract in params seems problematic. Let’s drop that block entirely and do the validation inside the handler.

We create a new class which will call our contract to validate, raise an exception (our custom type) to signal failure, and in the case of success yields to the block. The error is formatted from the structure the validator gives us:

  ContractError = Class.new(StandardError)

  class ApiContract < Dry::Validation::Contract
    def validate!(value)
      res = call(value)

      if res.success?
        yield res.to_h
      else
        message = errors_array(res.errors.to_h, false).join(', ')
        raise ContractError.new(message)
      end
    end

    def errors_array(hsh, decorate = true)
      res = []

      hsh.each do |key, value|
        key = if decorate
                "[#{key}]"
              else
                key.to_s
              end

        case value
        when String
          res << [key, value]
        when Array
          value.each do |s|
            res << [key, s]
          end
        else # Should be Hash.
          errors_array(value).each do |subkey, s|
            res << [key + subkey.to_s, s]
          end
        end
      end

      res.map! { |arr| arr.compact.join(' ') }
    end
  end

  # ...

  class CreateOrderContract < ApiContract
    params do
      required(:order).filled(OrderSchema)
    end
  end

  # ...

  class API < Grape::API
    format :json
    prefix :api

    rescue_from ContractError do |e|
      error!({error: e.message}, 400)
    end

    resource :orders do
      desc 'Create new'
      post do
        CreateOrderContract.new.validate!(params) do |attrs|
          order = Order.create!(attrs)
          body order
        end
      end
    end
  end

This reaches our goal (correct response format). Whether to actually use Grape’s format or keep the nested structure, is user’s choice. But this way we control the response either way.

Custom DSL

The above is a little verbose, though. And it adds an extra nesting level. What if we wanted to make the usage more succinct?

Looking at how the params block is implemented, the class Grape::Validations::ParamsScope applies itself to the current endpoint by setting three “namespace stackable” attributes: declared_params, renamed_params and validations. We will use neither, using our own attribute to store the assigned contract.

If we just add a helper to fetch the validated input, we don’t need to convert the schema to what declared understands. This could also be extended later so that the contract and declared params are used together, if needed.

We also define an error type inheriting from Grape::Exceptions::Base, so Grape handles it appropriately by default, but that can still be overridden in some (or all) endpoints using rescue_from.

The new implementation:

module GrapeContract
  class ValidationError < Grape::Exceptions::Base
    # @return [Dry::Validation::MessageSet]
    attr_reader :errors

    def initialize(errors:, headers: nil)
      @errors = errors
      message = errors_array(errors.to_h, false).join(', ')
      super(status: 400, message: message, headers: headers)
    end

    private

    def errors_array(hsh, decorate = true)
      # ...
    end
  end

  module EndpointDSL
    def contract(kontrakt = nil, &block)
      unless kontrakt
        kontrakt = Class.new(Dry::Validation::Contract, &block)
      end

      self.namespace_stackable(:contract, kontrakt)
    end
  end

  module InsideRouteHelper
    def contract_params
      kontrakt = namespace_stackable(:contract).last

      raise "No contract defined" unless kontrakt

      res = kontrakt.new.call(params)

      if res.success?
        res.to_h
      else
        raise ValidationError.new(errors: res.errors, headers: header)
      end
    end
  end

  # Install these globally, for every endpoint.
  ::Grape::API::Instance.extend(EndpointDSL)
  ::Grape::DSL::InsideRoute.include(InsideRouteHelper)
end

Now it can be used like

       desc 'Create new'
       contract CreateOrderContract
       post do

or even defining the contract inline:

      desc 'Create new'
      contract do
        params do
          required(:order).filled(OrderSchema)
        end

        rule(:order) do
          next if value[:baskets].count < 10

          key.failure('contains too many baskets')
        end
      end
      post do

See this repository for the full example.

Viewing file history across renames

2024-03-05T02:12:00+02:00

This entry is also about an addition in Emacs’s VC support (development branch, Emacs 30).

Git doesn’t really track renames, but supports printing the commit log across renames using certain heuristics (seeing that a commit contains a deleted and a created file where a big enough percentage of lines are the same; copies are detected similarly), but this has to be enabled explicitly (the --follow option), and it can end up missing commits in some situations with complex branch merging history.

In Emacs vc-print-log (C-x v l) likewise had problems with renames, with multiple reports over the years and different approaches having been proposed. Adding the support for --follow helped somewhat, but the commands to view the diff (d), or annotate/blame (a), or check out that old version (f), stayed broken when the version had a different file name.

This difficulty is not specific to Emacs, though, and looking around, Github has recently adopted a solution for this: adding a “show history for that name” link when the current log ends. It’s not an original invention either: it started out as a Chrome extension.

Anyway, now we also have a button that allows you to jump to the history of the previous file name. Just make sure to set vc-git-print-log-follow back to its default value (nil), to be able to use it.

This is currently implemented for Git and Mercurial only.

Committing a partial changeset in Emacs

2024-03-03T22:15:00+02:00

One of the frequent questions for the built-in Version Control support in Emacs has been the ability to commit only a part of the changes in a file or several.

In the command-line, it’s simple (though not always easy): you call git add -p and iterate through the whole repository diff with y/n/etc. Magit also has an interface for staging and committing from the staging area, but that only works for Git and requires you to switch to its UI first.

Using VC

One of the less noticed additions in Emacs 29 is the ability to create a commit from a diff buffer. And since diff-mode provides functionality for editing the diff, you can drop the unwanted pieces first (they stay saved on disk) and commit the amended changeset.

Here’s how it works: you press C-x v = or C-x v D to show the diff for the current file, or the whole repository, edit the diff using commands such as diff-hunk-kill (M-k), diff-split-hunk (C-c C-s), and then type C-x v v in the same buffer to proceed to the familiar step “Edit commit message”.

I also recommend customizing diff-default-read-only to t so that the short key combinations work right away in diff-mode, including n and p for hunk navigation and k for removing hunks. Otherwise those only work after switching to read-only-mode (C-x C-q)

Anyway, you enter the commit message and press C-c C-c to create the commit. This works not only with Git, but also with Mercurial repositories, and the fallback implementation endeavors to support other VC systems as well (such as Bazaar and SVN), though using a slower method. The latter is not as well-tested (the latest bug report was regarding an incompatible command-line flags in OpenBSD’s patch), so more feedback welcome.

Using Diff-HL

The workflow described above doesn’t use the staging area, and if you are using Magit, you might prefer a simply faster way to stage hunks.

By popular demand, the new additions to the diff-hl package are the commands diff-hl-stage-current-hunk and, more recently, diff-hl-stage-dwim.

Either type C-x v S to stage the hunk at or above point, or prepend it with C-u to similarly pop up a buffer where you can edit the changeset to stage. Then do the commit with Magit or the command line.

High throughput Fizz Buzz in Ruby

2023-12-30T12:34:00+02:00

Stumbled on a StackExchange competition of sorts, where the top answer is in hand-rolled assembler that reimplements libc functions in a specialized way to avoid copying as much as possible, uses special memory alignment for storing numbers, and a bytecode generator and interpreter for the main loop. Reading it is a nice deep rabbit hole.

There was only one solution in Ruby, however, which seemed unfair. So I tweaked it a little.

It doesn’t do too bad in the overall comparison (see the summary comparison table inside the question), so I think it demonstrates how the IO performance is its own challenge, and how runtime adaptation in a dynamic language can succinctly overtake a naive solution in a statically compiled one, and get within an order of a magnitude of an average solution in a decently fast language.

Also, Ractors.

Debugging a syntax error in js2-mode

2021-06-19T03:27:00+03:00

I haven’t been writing much JavaScript code lately, so my personal motivation to work on js2-mode has waned over the years, but it’s a cool package with some dedicated user base. I’m sure there are people who have wanted to contribute, but didn’t know where to start.

Below is the story of debugging and fixing a certain bug which took a bit more effort than expected. It can probably serve as a primer both for hacking js2-mode and Emacs Lisp in general. Some pre-existing knowledge of Lisp is required, though.

The Problem

The bug related to the parsing or trailing commas in arrow functions, which is apparently encouraged in some coding styles. This worked in normal functions (the feature was added several years prior):

function foo(a, b,) {
  console.log(a, b);
}

But not in arrow functions:

(a, b,) => { console.log(a, b) };

An educated guess from one of the reporters was to check js2-parse-function-params and see whether it’s doing something wrong. This function parses parameters for all kinds of functions (expression, statement or arrow).

Late Introduction

js2-mode uses a recursive-descent parser written in Emacs Lisp, originally ported from Mozilla Rhino by Steve Yegge.

So most of its code falls into one of 3 categories:

Type definitions for the values representing the syntax tree nodes, using cl-defstruct.
The token stream and the tokenizer. Main entry point is js2-get-token.
The parser code, which is a bunch of functions that read the tokens one by one and call each other in a recursive fashion, according to the JavaScript grammar, while employing a couple of tricks to resolve the ambiguities (when you can’t tell in advance what kind of node you are looking at). And it constructs the syntax tree based on the tokens encountered.

js2-parse-function-params is one of such functions, it helps reach the list of function parameters and organizes them into a list, which is then attached to the function node.

Debugging

So it’s a good place to start¹. Let’s switch to *scratch*, M-x js2-mode in there and paste the problematic snippet of code. It flashes in red: syntax errors.

We switch to another window and visit js2-mode.el in it. Navigate to js2-parse-function-params’s definition and instrument it to be debugged with C-u C-M-x. The echo area briefly flashes:

Edebug: js2-parse-function-params

Switch to *scratch* and type space somewhere at the end, forcing the buffer to be re-parsed. Emacs stops at the beginning of the instrumented function.

We step through it, to the end, by tapping SPC, noting the return values displayed in the echo area.

Things look wrong, starting from the return value of (js2-peek-token) at the beginning of the loop, followed by js2-must-match-name failing later, and (js2-create-name-node) creating a node with some odd :name attribute.

The first return value was the best hint, though. It returned 165 (token types are numbers), which, if you C-s through the file, leads to this line:

(defvar js2-ARROW 165)         ; function arrow (=>)

That means the tokenizer says that the next token after the current is the arrow (=>) already. We can double check that by seeing the value of js2-ts-cursor (by typing e while debugging and typing the variable’s name in the prompt). Its value is 12, the position right after the arrow token.

But when we are parsing some construct, we really expect to be near its beginning (either at the first token, or before it). How come we are not there?

Let’s look at the list of errors. Type M-x js2-display-error-list, and it brings up a bunch of warnings, as well as these three errors:

line 1: missing ) after formal parameters
line 1: missing formal parameter
line 1: missing ) in parenthetical

Searching for the first two brings us back to js2-parse-function-params. These errors are to be expected.

The last one is more interesting. The error message key to search for is "msg.no.paren", and it appears in 3 functions.

Try instrumenting them all. Only one of them is entered by the code parsing this example: js2-parse-paren-expr-or-generator-comp.

What does it do? It checks whether the current token is followed by for, or a right paren right away, and when it doesn’t, it calls js2-parse-expr and then checks that the said expression is followed by js2-RP, which is the “right paren” token. That fails to happen in our case.

Look at js2-parse-expr. It calls js2-parse-assign-expr in a loop, as long as it finds new js2-COMMA tokens.

We instrument js2-parse-expr. Stepping through it, we can see that it ends up calling (js2-parse-assign-expr) twice. The first call returns a js2-name-node, totally expected. The second call, inside (setq right (js2-parse-assign-expr), returns a js2-function-node (the return value printed in the echo area looks complex, but we can make sure by evaluating (aref right 0)). That’s a surprise.

To see how this happens, let’s go down the rabbit hole. First of all, reset all instrumentation by calling M-x eval-buffer in js2-mode.el. Then let’s simplify the code so the parser does less work for us to follow:

(a,) => { };

Then instrument js2-parse-expr with C-u C-M-x, make an edit in the code buffer and step through the source until the cursor is after (setq right . Press i to jump in.

Step through and jump inside every successive call to functions named js2-parse-*. Meaning, js2-parse-assign-expr, then to js2-parse-cond-expr, and so on. When we reach js2-parse-primary-expr, it finishes with reporting a syntax error and creating a js2-error-node. The thing to note here is it “consumes” the current token. The closing paren has just been parsed as an error node.

As we continue with stepping through the code with SPC, we can see the execution bubble up to js2-parse-assign-expr. In there, we see (= tt js2-ARROW) condition match, token stream rewound back to the position before the last node parsed, and then parsed again as function.

Erase and Rewind

What’s that about rewinding?

The ability to save and restore the current stream state was added for the implementation of the arrow functions’ support.

When the parameter list is parsed first, there is no good way to know in advance that it belongs to an arrow function, and it’s not just a parenthesized expression. So the parens are first parsed as an expression, and then if it’s followed by the arrow token, the stream is rewound and the code is parsed as a function.

For all of this to work, any valid parameters list of an arrow function has to be parse-able as a parenthesized expession, too.

At some point you might have been wondering: if the parameter list was not parsed correctly, and it was not followed by the arrow token, how come js2-parse-function-params was called at all? Hopefully the previous section answered that question ².

Solution

One possible direction would be to change js2-parse-primary-expr not to consume the current token. Then the inner js2-parse-assign-expr call wouldn’t match js2-ARROW as next token, js2-parse-expr would find the closing paren fine, and the call to js2-parse-assign-expr up the stack which we never actually stepped through, would find the arrow token in its proper place and parse the arrow function fine.

Unfortunately, that’s kind of dangerous, that approach can lead to infinite looping. Error nodes should consume tokens to ensure that the parser moves forward.

The solution I ended with is a simplified version of Mozilla’s code from Firefox. When a feature is already implemented in there, it’s often handy to refer to its code (even if it looks much busier than the elegant Lisp we are currently editing). The general structure is similar enough, the hierarchy of calls in particular.

The function in question is GeneralParser::expr. We can see that rather than allow just any kind of errors inside, it handles two special cases (trailing comma and triple dot operator), and only when within a context where it’s possible that the curent node is a function parameter list.

The change I checked in is much shorter because we don’t really need to worry about (not) reporting syntax errors: whenever we rewind the token stream to a previous position, we change js2-parsed-errors to its previous value as well (see the end of js2-parse-assign-expr).

Performance impact³ from this change was very modest (if noticeable at all), so it looks like a success.

Thanks for reading.

If you want to follow the scenario step-by-step, you need to check out the version of code that was used at the time, Git commit b891ede. ↩
To recap: what was recognized as the “parameter list” expression wasn’t followed by the arrow token, but the “error node” inside it was. So the arrow function was parsed from that position, and js2-parse-function-params was called then. ↩
To measure it, you can visit some large-ish file and evaluate (benchmark 10 '(js2-reparse t)). Then change the implementation, M-x eval-buffer and run the benchmark again. Do this a few times, to get the feel for the spread. ↩

Electric Indentation in Ruby in Emacs 24.4

2014-01-20T03:14:40+02:00

Intro

Although electric-indent-mode will be on by default in Emacs 24.4, most of the time it is used only in the most straightforward way possible: the major mode adds some characters to electric-indent-chars, and after the user types one of them, the current line gets reindented.

This limits us to electric indentation only after specific characters, not keywords, and without regard to the context.

However, electric-indent-mode allows a more advanced behavior, and for that one needs to set electric-indent-functions.

So far, only two modes bundled with Emacs use it: ruby-mode and perl-mode, and the latter only to disable electric indentation whenever point is not at the end of the line.

Motivation

But when we write a function, we can look at the full symbol before point, see if it’s a keyword, and if it starts at indentation, or if it was a keyword before we typed the last character. We can look at the character after point and see whether we just turned a continuation method call into a straight metod call, which does not need additional indentation.

Examples

| denotes the cursor position.

# before
if foo
  bar
  els|
end

# after
if foo
  bar
else|
end

# before
if foo
end|

# after
if foo
  ends|

# before
foo
|

# after
foo
  .|

# before
foo
  |.bar

# after
foo
t|.bar

The above list is based on my personal preference, so please let me know how it works for you.

Hopefully, this general behavior will also spread to other major modes.

Emacs 24.3’s Killer Feature: Eager Macro-Expansion

2013-04-07T00:03:00+03:00

Looks like this change has gone largely unnoticed, aside from occasional bug reports when it failed and emitted a warning.

Meanwhile, the speed-up it provides for uncompiled code ranges from nice to amazing, depending on the amount and complexity of macros used.

Intro

I first noticed the difference when benchmarking an mmm-mode function that calls syntax-propertize-function from different major modes, including ruby-mode.

ruby-syntax-propertize-function, like most of the similar functions, uses syntax-propertize-rules, a distinctly complex macro. The difference between interpreted and compiled code was orders of magnitude, and it was especially noticeable in mmm-syntax-propertize-function, because the ERB code example I usually use for performance testing has ~200 ERB regions, so that’s the amount of times ruby-syntax-propertize-function was called.

Some numbers

For a more practical example, let’s measure the time js2-mode parser takes to process a large source file. All ~1500 lines of uncompressed Backbone.js.

js2-mode has always been notoriously slow in interpreted mode, due to the heavy use of defstruct facility and other macros from the cl package.

Version	Interpreted (time, s)	Compiled (time, s)
Emacs 24.2	6.4251	0.3025
Emacs 24.3	0.5026	0.2524

So, compiled code became a bit faster. Not critical, but nice.

Interpreted code became a lot faster, losing to the compiled code only by the factor of 2.

This is huge, it means that we can drop the strict recommendation to compile the package when installing manually, for Emacs 24.3 and later. I’m in no hurry to change the doc, but the amount of dissatisfied keyboard jockeys who routinely skip the documentation will go down, at least on this subject.

It’s especially nice for me personally: having to recompile the code after making some changes has always been a pain. Authors of other Elisp packages and users with a lot of code in their init files should also see the benefit.

More details

Not having studied the innards of bytecomp.el in detail, I’ll stick to what we can glean from experiment.

A good way to see what some function really does is fire up the Emacs Lisp REPL (M-x ielm) and there evaluate (symbol-function 'foo), where foo is the function in question.

As an aside, this is also a good way to find out what some macro like define-minor-mode does without studying it in detail: just look at the body of the resulting -mode function.

Take this definition:

(defun foo ()
  (when t
    (dotimes (i 10))))

With Emacs 24.3, you can (sometimes unwittingly) avoid the eager macro-expansion by using eval-last-sexp (C-x C-e) instead of eval-defun (C-M-x) or eval-buffer (no default binding). So we can try it both ways.

Without eager expansion, the body looks very familiar:

ELISP> (symbol-function 'foo)
(lambda nil
  (when t
    (dotimes
        (i 10))))

ELISP> (js2-time (dotimes (i 1000) (foo)))
0.0367

With eager expansion, we can see that it has been reduced to basic forms:

ELISP> (symbol-function 'foo)
(lambda nil
  (if t
      (progn
        (progn
          (let
              ((--dotimes-limit-- 10)
               (i 0))
            (while
                (< i --dotimes-limit--)
              (setq i
                    (1+ i))))))))

ELISP> (js2-time (dotimes (i 1000) (foo)))
0.0086

The numbers fluctuate heavily with repeated invocations, but the second version is always several times faster. The “familiar” version has to expand the macros each time the function body is evaluated, which drags the performance down significantly.