Representation: A Rule Service in Elixir

representation.livemd

A Rule Service in Elixir

Agenda

1. A Rule Engine, What & Why?
2. Formular: Elixir As DSL

A Rule Service, What & Why?

We manage to develop a rule service for other applications (micro services).

• Applications' configurations are mananged in the rule service.
• The changes apply on-the-fly without a need to restart the service.
• Applications only rely on the rule service at starting. After that, even the rule service is down, they keep running

Context

• Zubale is a crowdsourcing platform working for multiple brands.

• In Zubale, we create Tasks for our users.

• Tasks belong to stores, which belong to brands.

• Tasks have delivery windows, the expected duration for delivery to fulfill

• Each task has a visibility duration, named Cycle, during which a user can take the task.

• An example struct:

  %{
    delivery_window: {~U[2022-01-16 08:00:00Z], ~U[2022-01-06 16:00:00Z]},
    cycle: {~U[2022-01-16 08:00:00Z], ~U[2022-01-17 23:59:59]},
    store: %{
      id: 2040,
      brand: %{
        id: "wong"
      }
    }
  }

• Cycle calculation rules differ from different stores and brands.

The problem

How do we deal with the diversity of the cycle rules?

E.g.

• For brand "wong"
  • cycle starts at the beginning of the day of the delivery window
  • ends at the end of the day of the delivery window
• For brand "qiu"
  • ... and all stores except store#2040
    • cycle starts 3 hours before the delivery window
    • ends at the end of the day, plus 48 hours
  • and store#2040
    • cycle starts 4 hours before the delivery window
    • ends at the end of the day, plus 48 hours

In summary:

brand	store	cycle start	cycle end
wong	*	beginning_of_the_day(dw_start)	end_of_the_day(dw_end)
qiu	2040	dw_start - 4h	end_of_the_day(dw_end) + 48h
qiu	*	dw_start - 3h	end_of_the_day(dw_end) + 48h

(dw = delivery_window)

with a helper module:

defmodule TimeHelper do
  import DateTime, except: [add: 3]

  defdelegate add(t, value, unit), to: DateTime

  # let's ignore the timezone here
  def beginning_of_day(%DateTime{} = t),
    do: t |> to_date() |> new(~T[00:00:00]) |> elem(1)

  def end_of_day(%DateTime{} = t),
    do: t |> to_date() |> new(~T[23:59:59.999]) |> elem(1)

  def substract(%DateTime{} = t, value, unit),
    do: add(t, -value, unit)
end

How do we complete this function?

@doc """
Given a quest struct, return the computed cycle.
"""
@spec compute_cycle(map()) :: {DateTime.t(), DateTime.t()}
def compute_cycle(%{} = quest) do
  # what do we write here?
  ...
end

V1: The Hard Code

Our first version is hard-coded. It looks like:

defmodule CycleV1 do
  import TimeHelper

  @four_hour_stores_for_qiu [2040]

  def compute_cyle(quest) do
    case quest do
      %{store: %{brand: "wong"}, delivery_window: {dws, dwe}} ->
        {beginning_of_day(dws), end_of_day(dwe)}

      %{store: %{brand: "qiu", id: store_id}, delivery_window: {dws, dwe}}
      when store_id in @four_hour_stores_for_qiu ->
        {
          dws |> substract(4 * 3600, :second),
          end_of_day(dwe) |> add(48 * 3600, :second)
        }

      %{store: %{brand: "qiu"}, delivery_window: {dws, dwe}} ->
        {
          dws |> substract(3 * 3600, :second),
          end_of_day(dwe) |> add(48 * 3600, :second)
        }
    end
  end
end

It works:

quests = [
  %{
    delivery_window: {~U[2021-12-11 13:00:00Z], ~U[2021-12-11 18:00:00Z]},
    store: %{
      id: 1,
      brand: "wong"
    }
  },
  %{
    delivery_window: {~U[2021-12-11 13:00:00Z], ~U[2021-12-11 18:00:00Z]},
    store: %{
      id: 4,
      brand: "qiu"
    }
  },
  %{
    delivery_window: {~U[2021-12-11 13:00:00Z], ~U[2021-12-11 18:00:00Z]},
    store: %{
      id: 2040,
      brand: "qiu"
    }
  }
]

for quest <- quests, do: CycleV1.compute_cyle(quest)

The problem is that the rules frequently change as we expand to new brands or make adjustments for specific stores. So we need to change the code and deploy the application each time.

Deployments are energy costly and disturbing!

V2: Open-Close & Make It Configurable

What if we separate the hot function as a configuration?

defmodule CycleV2 do
  @formula """
  case quest do
    %{store: %{brand: "wong"}, delivery_window: {dws, dwe}} ->
      {beginning_of_day(dws), end_of_day(dwe)}

    %{store: %{brand: "qiu", id: store_id}, delivery_window: {dws, dwe}}
    when store_id in [2040] ->
      {
        dws |> substract(4 * 3600, :second),
        end_of_day(dwe) |> add(48 * 3600, :second)
      }

    %{store: %{brand: "qiu"}, delivery_window: {dws, dwe}} ->
      {
        dws |> substract(3 * 3600, :second),
        end_of_day(dwe) |> add(48 * 3600, :second)
      }
  end
  """

  def compute_cyle(quest) do
    Code.ensure_compiled!(TimeHelper)

    env = [
      functions: [
        {TimeHelper, TimeHelper.__info__(:functions)},
        {Kernel, Kernel.__info__(:functions)}
      ]
    ]

    {cycle, _binding} =
      Code.eval_string(
        @formula,
        [quest: quest],
        env
      )

    cycle
  end
end

for quest <- quests, do: CycleV2.compute_cyle(quest)

We can move the string formula into a configuration, e.g.

# file: config/config.exs
config :my_app, :cycle_formula, System.fetch_env!("CYCLE_FORMULAR")

then use it in the code:

@formula Application.get_env(:my_app, :cycle_formula)

Now, we can only update the configuration without changing the main module, which is more accessible than v1 because we can set the formula from an environment variable!

So far, so good, but can we make it better?

Yes, for sure!

Compiling AST into an Elixir module

First, we can compile it into an Elixir module to improve the performance (~200x).

Instead of:

Code.eval_string(code, binding, env)

we will turn the code string into a module:

defmodule MyEvalModule do
  def eval(binding) do
    # code
  end
end

then run this module:

MyEvalModule.eval(binding)

The problem is how we turn code a + b into module:

defmodule MyEvalModule do
  def run(binding) do
    a = Keyword.fetch!(binding)
    b = Keyword.fetch!(binding)

    a + b
  end
end

To achieve it, we need to know the variables used in the code. In other words, we need to implement a function used_variables/1, which acts like:

iex> code = """
case quest do
  %{store: %{brand: "wong"}, delivery_window: {dws, dwe}} ->
    {beginning_of_day(dws), end_of_day(dwe)}

  %{store: %{brand: "qiu", id: store_id}, delivery_window: {dws, dwe}}
  when store_id in [2040] ->
    {
      dws |> substract(4 * 3600, :second),
      end_of_day(dwe) |> add(48 * 3600, :second)
    }

  %{store: %{brand: "qiu"}, delivery_window: {dws, dwe}} ->
    {
      dws |> substract(3 * 3600, :second),
      end_of_day(dwe) |> add(48 * 3600, :second)
    }
end
"""
iex> used_variables(code)
[:quest]

used_variables(code_string)

I end up traversing the AST tree and dealing with the scopes manually.

References:

• Elixir Scoping Rules
• Formular.Compiler.extract_vars/1
• Find out unbound variables in the AST
• Comments from José

  I want to add that this is actually hard to do correctly in practice. You would effectively need to implement a large chunk of the compiler.

• Another approach without AST traversing

Summary

We can do it by compiling string into an Elixir module.

Improve Configurations Management

To change the application configuration, we still need to deploy the application.

• It causes restarting the application.
• Only developers can do the updating, which is annoying.

So I made Formular Server to solve it.

V3: Formula Server

[Demo]

Roadmap

• New revision validator
• Authentification & Authorization
• Open sourcing
• Modeling the configuration
  • free developers
  • Process Modeling

Thank you!