Home

@wq/outbox

@wq/outbox

@wq/outbox

@wq/outbox is a wq.app module providing an offline-cabable "outbox" of unsynced form entries for submission to a web service. @wq/outbox integrates with @wq/store to handle offline storage, and with @wq/model for managing collections of editable objects.

By default, @wq/outbox does not apply changes to the local @wq/model state until after form submissions are succesfully synced to the server. Accordingly, @wq/app is configured to show unsynced outbox records at the top of model list views and/or in a separate screen. As of wq.app 1.2, it is also possible to configure @wq/outbox to optimistically apply @wq/model changes immediately, with syncing happening in the background.

As of wq.app 1.2, @wq/outbox is based on Redux Offline, and leverages its strategies for detecting network state and retrying failed submissions. Most notably, Redux Offline schedules sync attempts automatically, whereas @wq/outbox in wq.app 1.1 and earlier relied on @wq/app to manage the sync interval.

@wq/outbox can be used to store photos and other files submitted with a form. The files will be stored as Blobs in offline storage until the outbox is synced. See the @wq/app:photos documentation for more information about this feature.

Installation

wq.app for PyPI

@wq/app for npm

python3 -m venv venv      # create virtual env (if needed)
. venv/bin/activate       # activate virtual env
python3 -m pip install wq # install wq framework (wq.app, wq.db, etc.)
# pip install wq.app      # install wq.app only

npm install @wq/app       # install all @wq/app deps including @wq/outbox
# npm install @wq/outbox  # install only @wq/outbox and deps

API

When used with @wq/app, @wq/outbox is available as app.outbox. When using directly, @wq/outbox is typically imported as outbox, though any local variable name can be used. Each outbox instance should be bound to the corresponding @wq/store instance it will use for managing data. The main outbox is pre-bound to the main store instance and should be suitable as-is for most use cases.

wq.app for PyPI

@wq/app for npm

// @wq/app usage
define(['wq/app', './config', ...], function(app, config, ...) {
   config.outbox = ...;
   app.init(config).then(...);
   var outbox = app.outbox;
});

// Direct usage
define(['wq/store', 'wq/outbox', ...], function(ds, outbox, ...) {
    outbox.init(config1);
    var secondStore = ds.getStore('store2');
    var secondOutbox = outbox.getOutbox(secondStore);
    secondOutbox.init(config2);
});

// @wq/app usage
import app from '@wq/app';
import config from './config';
config.outbox = ...;
app.init(config).then(...);
const outbox = app.outbox;

// Direct usage
import outbox, { getOutbox, Outbox } from '@wq/outbox';
import { getStore } from '@wq/store';

outbox.init(config1);  // Main outbox

const secondStore = getStore('store2');
// secondOutbox = new Outbox(secondStore) // May conflict if an outbox is already defined for store2
const secondOutbox = getOutbox(secondStore);  // This will return store2's outbox if it exists, or create it
secondOutbox.init(config2);

Outbox Item

outbox.save(), outbox.unsyncedItems(), and the related APIs described below leverage the concept of an "outbox item", which is a single record describing a pending form submission and metadata.

name	purpose
id	The local unique identifier for the outbox item.
data	The form key-value pairs as passed to outbox.save()
options	Additional parameters that configure how the data should be sent to the server, and potentially how the response should be interpreted. (see outbox.save())
synced	Whether the outbox item has been successfully saved to the server.
result	If applicable, the successful response JSON returned from the server.
error	If applicable, the error returned from the server or from the AJAX call when attempting to save the item. Will be either a string or a JSON object.
newid	Removed in wq.app 1.2. This can be retrieved from result.id instead.

Changed in wq.app 1.2: @wq/outbox no longer uses an internal @wq/model instance for managing unsynced records. Instead, the underlying Redux Offline outbox state is wrapped with a model-like API. If you have code relying on outbox.model.load(), change it to use outbox.loadItems() instead which provides an equivalent structure.

Initialization

outbox.init(config)

outbox.init() configures the outbox with the necessary information to communicate with a web service. The outbox will automatically re-use the service, formatKeyword, defaults, and debug parameters provided to @wq/store. The list of outbox-specific options is described below:

name	purpose
syncMethod	Default HTTP method to use for sending data to the server. The "default" default is POST. This can be overridden on a per-form basis by setting the method option.
applyState	Default setting for when to apply changes to the local @wq/model state. The "default" default is "ON_SUCCESS". This can be overridden on a per-form basis by setting the "applyState" option.
cleanOutbox	Whether to clean up synced outbox items whenever the application starts (default true). Note that Redux Offline usually removes items from the queue as soon as they succeed or fail. @wq/outbox overrides this to keep the items around until the outbox is cleaned or emptied.
maxRetries	The maximum number of times to attempt sending an outbox item before giving up. In wq.app 1.2, the default changed from 3 to 10. Note that Redux Offline automatically increases the interval between consecutive failed sync attempts (whereas wq.app 1.1 and earlier used a fixed interval.)
csrftokenField	The form field name to use when submitting the CSRF token. Note that the token will be set when the form is actually uploaded to the server (and may override the csrf token that was initially submitted to the outbox). The default field name is csrfmiddlewaretoken since that's what Django calls it.
validate(data, item)	Defines a callback that ensures data is valid before saving it to the outbox. The default implementation always returns true
batchService	Specifies that the server supports submitting multiple requests as a batch. Changed in wq.app 1.2: This should be specified as an API path relative to the main service URL. By default, the server's batch API is assumed to be compatible with Django Batch Requests.
batchMinSize	New in wq.app 1.2. Specifies the threshold at which batch mode should be activated. The default is 2, meaning the batchService will only be used when there are at least 2 records in the outbox that are ready to sync. Set this to 1 (or 0) to ensure that batch mode is always used.
batchMaxSize	New in wq.app 1.2. Specifies the maximum number of requests to include in a single batch. Defaults to 50.
parseBatchResult(result)	Removed in wq.app 1.2. This option was used to parse the result of a batch upload. In wq.app 1.2 the result is assumed to match the output structure of Django Batch Requests. If not, you can use an ajax() plugin hook to specify a custom parsing function.
applyResult()	Removed in wq.app 1.2. This option was used to customize whether a form submission is successful. In wq.app 1.2 this can be done with an ajax() plugin hook that throws an error on failure.
updateModels()	Removed in wq.app 1.2. This was used to configure how local models are updated based on the server response. In wq.app 1.2, this is done by dispatching the appropriate Redux actions to update the local state.

Plugin Hooks

@wq/outbox provides support for the following @wq/app plugin hooks.

onsync(item)

New in wq.app 1.2.

Called after an outbox item is succesfully synced (or has an error). See Outbox Item above for the description of the available attributes.

Note that this function will be called each time an item is synced, regardless of how the item was created. If you have custom code that calls outbox.save() directly, you can use outbox.waitForItem(id) instead of this hook.

Outbox Methods

As discussed above, all data being sent to the server (e.g. as a result of a form submission) is queued through an outbox. This section describes the available functions for working with the outbox.

outbox.setCSRFToken(csrftoken)

Updates the CSRF token that will be applied to outbox items when they are synced to the server. This should be updated whenever the user's authentication status changes. @wq/app calls this function automatically.

outbox.save(data, [options])

outbox.save() takes the form data as a simple JavaScript object (see above) and an optional options object, and creates an outbox item. The options object can have one or more of the following set:

name	purpose
url	URL to post to (relative to the base service URL). If unset, it is assumed that the base service URL can handle form submissions itself. @wq/app will set this from the action of the submitted form.
modelConf	The configuration for a corresponding model that should be updated when this item is synced. This is set automatically by @wq/app by resolving the url to a configured model.
method	HTTP method to use when posting the data (PUT, POST, etc.). The default is config.syncMethod (usually POST), but @wq/app will automatically use PUT when updating an existing model instance.
applyState	New in wq.app 1.2. For model-backed forms, applyState determines when to apply form submissions to the local state. The default is config.applyState (usually "ON_SUCCESS"). See Redux Actions below for more info.
id	The outbox id of a previous form submission that hasn't yet been synced. This option makes it possible to allow the user to review and edit outbox items before they are synced to the server. It can be set automatically by @wq/app if data-wq-outbox-id is set on the <form>.
storage	Where to store the form data associated with the outbox record. By default, the data is stored directly in the Redux state that is persisted to offline storage. If the form contains sensitive data (such as username/password), "temporary" should be used instead to ensure the form data is not persisted. For form submissions containing binary data (e.g. Blob), storage should be set to "storage", which preserves the data in a separate form key to avoid performance issues when persisting the Redux state.
preserve	A list of fields to preserve in the existing outbox item. This option can be used with id to avoid overwriting hard-to-set fields like file uploads and GPS coordinates. It can be automatically set by @wq/app if data-wq-preserve is set on the <form>. See the Species Tracker code for an example.

outbox.save() returns a Promise that resolves to the stored outbox item.

Changed in wq.app 1.2: @wq/outbox now uses Redux Offline to handle syncing in the background. Because of this, the returned Promise is always resolved before the record is synced. If you need to wait for the sync result, call outbox.waitForItem() after calling outbox.save()). Relatedly, outbox.save() no longer accepts a third noSend argument. If you would like to save an item to the outbox without triggering an immediate sync attempt, call outbox.pause() before outbox.save().

wq.app for PyPI

@wq/app for npm

$form.submit(function() {
     var data = {};
     $form.serializeArray().forEach(function(field) {
         data[field.name] = field.value;
     });
     outbox.save(data).then(function(item) {
         return outbox.waitForItem(item.id);
     }).then(function(item) {
         if (item.synced) {
             console.log("Item successfully synced!");
         } else {
             console.log(item.error);
         }
     });
});

$form.submit(async () => {
     var data = {};
     $form.serializeArray().forEach(function(field) {
         data[field.name] = field.value;
     });
     const { id } = await outbox.save(data);
     const item = await outbox.waitForItem(id);
     if (item.synced) {
         console.log("Item successfully synced!");
     } else {
         console.log(item.error);
     }
});

Redux Actions

Internally, outbox.save() dispatches a Redux action with appropriate Redux Offline commit, and rollback metadata. The commit action is dispatched after a successful sync, while the rollback action is dispatched after the record is rejected or fails more than maxRetries times. If config.debug is true, all Redux actions will be logged to the console which can help with debugging.

The generated action types vary depending on the modelConf and applyState options supplied to outbox.save(). In particular, action types marked with * below are recognized by the @wq/model reducer.

applyState = ON_SUCCESS

When applyState is set to "ON_SUCCESS", form submissions will not be reflected in the local model state until after the form is successfully synced to the server. This is the default configuration (and was the only option in wq.app 1.1 and earlier).

Form Type	Submit Action	Commit Action	Rollback Action
@wq/model (POST, PUT)	ORM_{model}_SUBMIT	ORM_{model}_UPDATE*	ORM_{model}_ERROR
@wq/model (DELETE)	ORM_{model}_DELETESUBMIT	ORM_{model}_DELETE*	ORM_{model}_DELETEERROR
other configured page (e.g. login)	{page}_SUBMIT	{page}_SUCCESS	{page}_ERROR
unconfigured	FORM_SUBMIT	FORM_SUCCESS	FORM_ERROR

applyState = IMMEDIATE

When applyState is set to "IMMEDIATE", form submissions are reflected in the local model state before they are sent to the server.

Form Type	Submit Action	Commit Action	Rollback Action
@wq/model (POST, PUT)	ORM_{model}_UPDATE*	ORM_{model}_SUCCESS*	ORM_{model}_ERROR
@wq/model (DELETE)	ORM_{model}_DELETE*	ORM_{model}_DELETESUCCESS	ORM_{model}_DELETEERROR

If the record does not already have a server-assigned ID, a temporary local id will be generated. When ORM_{model}_SUCCESS is dispatched, the local record will be updated with the new ID from the server.

Note that if the request fails, there is currently no code to process the Rollback Action (e.g. to revert the local change).

applyState = LOCAL_ONLY

When applyState is set to "LOCAL_ONLY", form submissions are not synced to the server at all. In this case, nothing is stored in the outbox, so the only reason to use outbox.save() is to maintain API consistency with other forms.

Form Type	Submit Action	Commit Action	Rollback Action
@wq/model (POST, PUT)	ORM_{model}_UPDATE*	-	-
@wq/model (DELETE)	ORM_{model}_DELETE*	-	-

outbox.sendItem()

This method was removed in wq.app 1.2. Use outbox.waitForItem() instead.

outbox.sendAll()

This method was removed in wq.app 1.2. Use outbox.retryAll() and/or outbox.waitForAll() instead.

outbox.waitForItem(id)

New in wq.app 1.2. Returns a promise will be resolved when the specified outbox item is synced to the server (or has failed more than maxRetries times). The resolved value will be the outbox item.

outbox.waitForAll()

New in wq.app 1.2. Returns a promise will be resolved when all items in the outbox have been synced to the server (or have failed more than maxRetries times). The resolved value will be empty.

outbox.unsynced([modelConf])

outbox.unsynced() returns a Promise that will resolve to the number of unsynced items in the outbox. If modelConf is set, returns only the number of items that associated with the specified model (see outbox.save()).

outbox.unsyncedItems([modelConf[, withData]])

outbox.unsyncedItems() returns a Promise that resolves to an array containing any items in the outbox that haven't been synced yet. If modelConf is set, unsyncedItems() returns only the items that associated with the specified model. If withData is true, unsyncedItems() will load any data stored separately from the outbox records (e.g. data containing binary Blobs). withData is false by default.

outbox.pendingItems([modelConf[, withData]])

Like unsyncedItems(), but limited to items that haven't been sent at all (or at least haven't failed more than maxRetries times).

outbox.retryItem(id)

New in wq.app 1.2. Clear out any error or success on the specified outbox item, ensuring it will be retried during the next sync attempt.

outbox.retryAll()

New in wq.app 1.2. Clear out any error on all unsynced items, ensuring that they will be retried during the next sync attempt.

outbox.pause()

New in wq.app 1.2. Pause Redux Offline syncing until resume() is called.

outbox.resume()

New in wq.app 1.2. Resume syncing.

outbox.empty()

New in wq.app 1.2. Wipe out all outbox records, including those that have not been synced. Internally, this is accomplished by resetting Redux Offline to its initial state.