Sebastian's Gopherspace (HTTP Gateway)

Introduction
------------------------------------------------------------
 
Paperboy
 
 
Paperboy is a Clojure abstraction for reliably passing
messages to arbitrary brokers. It provides the building
blocks needed to buffer, order, claim, deliver, and
acknowledge messages between an application and a
destination system.
 
For example, Paperboy can provide a persistent buffer
between an application and a messaging system such as MQTT.
More generally, it can sit in front of any destination that
may be temporarily unavailable: the application submits a
message to Paperboy, which can persist and deliver it
independently of the calling code and retry it later when
necessary.
 
Project status: Paperboy is at an early stage of
development. Its API, protocols, data model, and behavior
are still subject to change. The current implementations
primarily exist to develop and validate the architecture and
its concurrent workflows.
 
 
Installation
------------------------------------------------------------
 
This library can be installed from Clojars [1].
 
 
Architecture
------------------------------------------------------------
 
Paperboy consists of three separate components:
 
  1. The Spooler accepts new messages from the application.
  2. The Queue coordinates available, in-flight, and
     successfully delivered messages.
  3. The Broker takes messages from the queue and transmits
     them to the destination system.
 
Architecture
 
 
The components communicate through small Clojure protocols.
Each implementation can therefore be replaced independently.
A persistent spooler does not need to know which concrete
queue or broker is being used. Similarly, a broker does not
need to know whether messages were persisted in memory, a
file, or a database.
 
 
Envelopes and messages
----------------------
 
Paperboy transports each message inside an envelope:
 
  {:path "/sensors/temperature"
   :message {:id "1"
             :payload "21.5"}}
 
An envelope contains:
 
  * :path: the logical destination, such as a topic,
    channel, or routing key;
  * :message: the message itself, containing:
    * :id: a unique identifier used to coordinate delivery
      and deletion;
    * :payload: the content to transmit.
 
 
Message flow
------------
 
Message flow
 
 
The message flow separates accepting a message from
delivering it and from cleaning up its persisted copy.
 
  1. The application submits a path and payload to the
     Spooler using submit!.
  2. The Spooler assigns a unique ID and creates an
     envelope. A persistent implementation stores the
     message before making it available for delivery.
  3. The Spooler passes the envelope to the Queue using
     put!.
  4. The Queue makes the envelope available and notifies a
     waiting Broker.
  5. The Broker waits for work with await-ready! and
     reserves the next envelope with claim!.
  6. Claiming prevents another Broker from processing the
     same delivery concurrently. It does not mean that the
     message has been delivered successfully.
  7. The Broker transmits the envelope to its destination.
 
If transmission succeeds, the Broker calls ack!. The Queue
records the successful delivery and notifies the Spooler
that the corresponding persisted message may be removed. The
Spooler's cleanup worker waits for this notification with
await-removed!, obtains acknowledged message IDs in batches
through drain-removed!, and deletes their persisted copies.
A batch is completed only when the Spooler's removal
function reports success; otherwise, the removal request is
retained for a later attempt.
 
If transmission fails, the Broker must not call ack!. The
delivery remains unacknowledged, and the Broker is
responsible for buffering the in-flight message and retrying
it later. This allows each Broker implementation to apply
the retry, backoff, and destination-specific error-handling
strategy appropriate for its transport.
 
 
Spooler implementations
-----------------------
 
paperboy.spooler.passthrough is a non-persistent Spooler. It
assigns message IDs and forwards envelopes directly to the
Queue.
 
paperboy.spooler.rocksdb persists messages in a RocksDB
database before making them available to the Queue. When it
is created, it restores all messages still present in the
database in ascending ID order. Its cleanup worker removes
messages from RocksDB after the Queue reports that they were
acknowledged or evicted.
 
  (ns persistent-example
    (:require [paperboy.api :as api]
              [paperboy.queue.fifo :as fifo]
              [paperboy.spooler.rocksdb :as rocks]))
 
  (def queue (fifo/fifo))
  (def spooler (rocks/rocksdb "var/paperboy" queue))
 
  (api/start! spooler)
  (api/submit! spooler "/notifications/email" "hello")
 
  ;; Stops cleanup and closes the RocksDB database.
  (.close ^java.io.Closeable spooler)
 
Message IDs are unique among messages retained by a RocksDB
Spooler, but are not globally monotonic across restarts. If
higher IDs have already been removed, they may be assigned
again after the Spooler is reopened. The Queue connected to
this Spooler must contain only envelopes submitted through
that Spooler.
 
 
Queue implementations
---------------------
 
paperboy.queue.fifo offers a simple in-memory FIFO queue. It
keeps all waiting envelopes in insertion order and makes
them available to Brokers one at a time.
 
paperboy.queue.quota offers an in-memory queue with per-path
quotas. It accepts a default quota and optional
route-specific quotas:
 
  (quota/quota {:size 100
                :routes {"/tenants/a/*" {:size 10}
                         "/alerts/?" {:size 20}}})
 
Quota route patterns use the same path segment rules as
envelopes, plus two wildcards:
 
  * ? matches exactly one path segment.
  * * matches the rest of the path.
 
When an envelope is put into the queue, its path is matched
against the quota routes. A match uses the matching route's
:size, but the bucket is still the concrete matched path.
For example, /tenants/a/one and /tenants/a/two both match
/tenants/a/*, but they are stored in separate buckets, each
with size 10. Paths that match no route use the default
:size and the default bucket.
 
When a bucket is full, adding a newer envelope evicts older
waiting envelopes from that bucket. Evicted message IDs
become ready for cleanup through await-removed! and
drain-removed!, just like acknowledged message IDs.
 
 
Broker implementations
----------------------
 
paperboy.broker.stdout is a diagnostic Broker that writes
each claimed message to *out* and acknowledges it after a
successful write.
 
paperboy.broker.mh publishes messages to an MQTT broker
through a Machine Head [2] client. The envelope's :path
becomes the MQTT topic and its :payload becomes the
published payload:
 
  (ns mqtt-example
    (:require [clojurewerkz.machine-head.client :as mh]
              [paperboy.api :as api]
              [paperboy.broker.mh :as mqtt]
              [paperboy.queue.fifo :as pq]
              [paperboy.spooler.passthrough :as pp]))
 
  (def queue (pq/fifo))
  (def spooler (pp/passthrough queue))
  (def client (mh/connect "tcp://localhost:1883"))
  (def broker (mqtt/mqtt queue client))
 
  (api/start! spooler)
  (api/start! broker)
  (api/submit! spooler "/sensors/temperature" "21.5")
 
The MQTT client owns the connection settings,
authentication, and reconnect behavior. Paperboy claims and
publishes one envelope at a time. While the client is
disconnected or publishing fails, it retains the claimed
envelope and retries it before claiming the next one. If
publishing succeeds but acknowledging the envelope fails,
Paperboy publishes it again. Delivery is therefore at least
once, and MQTT consumers must tolerate duplicate messages.
 
During controlled shutdown, stop the Paperboy components and
close the MQTT client:
 
  (api/stop! broker)
  (api/stop! spooler)
  (mh/disconnect-and-close client)
 
 
Example
------------------------------------------------------------
 
The following example wires the three components together
and deliberately submits messages before starting the
Broker. This demonstrates that accepting a message and
delivering it are independent operations.
 
  (ns example
    (:require [paperboy.api :as api]
              ;; The FIFO implements both sides of the Queue
              ;; abstraction:
              ;; Producer for the Spooler and Consumer for
              ;; the Broker.
              [paperboy.queue.fifo :as pq]
 
              ;; Passthrough is a non-persistent Spooler. It
              ;; creates
              ;; envelopes and forwards them directly to the
              ;; Queue.
              [paperboy.spooler.passthrough :as pp]
 
              ;; Stdout is a simple Broker that prints
              ;; claimed messages and
              ;; acknowledges them after they have been
              ;; written successfully.
              [paperboy.broker.stdout :as pb]))
 
  ;; Create the shared Queue. Both the Spooler and Broker
  ;; receive this same
  ;; instance, but interact with it through different
  ;; protocols.
  (def queue (pq/fifo))
 
  ;; Connect the Spooler to the Producer side of the Queue.
  (def spooler (pp/passthrough queue))
 
  ;; Start the Spooler's cleanup worker. It waits for
  ;; acknowledged message IDs
  ;; and removes their persisted copies. Passthrough has no
  ;; external storage,
  ;; so removal always succeeds, while still exercising the
  ;; complete workflow.
  (api/start! spooler)
 
  ;; Submit two messages. The Spooler assigns an ID to each
  ;; payload, creates an
  ;; envelope, and puts it into the Queue. At this point the
  ;; Broker has not been
  ;; started, so the messages remain available in FIFO
  ;; order.
  (api/submit! spooler "/foo" "hello")
  (api/submit! spooler "/bar" "world")
 
  ;; Connect the Stdout Broker to the Consumer side of the
  ;; same Queue.
  (def broker (pb/stdout queue))
 
  ;; Start the Broker worker. It waits for available
  ;; messages, claims them one
  ;; at a time, writes them to stdout, and acknowledges
  ;; successful deliveries.
  ;; The acknowledgements wake the Spooler's cleanup worker.
  (api/start! broker)
 
The example creates background worker threads. In a
long-running application they normally live for the lifetime
of the application. During controlled shutdown, stop both
components explicitly:
 
  (api/stop! broker)
  (api/stop! spooler)
 
 
References
------------------------------------------------------------
[1]: Clojars
[2]: Machine Head
 
 
Links
------------------------------------------------------------
main.zip
GitHub