PREFACE: This is an early draft of the still-untitled Home Cloud project's architecture. I'm sharing this as a kind of progress-report since I'm not ready to share code yet. Feedback is welcome but keep in mind that my head's still "in the details" so it may be hard to take too many suggestions at this stage. I'm sure lots of things will change as the rubber hits the road.
Since I don't have a name for the project yet, I'm using the temporary codename "Austin."
Austin uses a microservice cloud architecture with the following attributes:
- Peer-to-peer networking
- Strictly-defined protocols
- A small host environment and modular userland programs
- A focus on ease-of-use for developers and end-users
Austin is designed for Home Clouds and must be easy to maintain. While that goal might encourage a monolothic architecture to "guarantee" a baseline experience, doing so would reduce portability, discourage contributions, complicate debugging, and impose too many opinions. Austin uses a microservice architecture instead. As the application and networking stacks require distributed computing, the Austin architecture focuses on coordinating services with rigorous tooling; moving core services into the program-space just allows those services to benefit from that tooling.
The Austin Architecture is comprised of a "host environment" and multiple userland programs which provide services. The host environment in Austin is currently written in NodeJS, while the user programs are currently Deno scripts.1
The host environment executes user programs, routes messages, applies permissions, manages user accounts, and provides debugging/auditing tools. It depends on a core set of user programs, most importantly the ADB (Austin DataBase) service where program and user configuration is stored.
Programs import and export APIs which are described by the Austin Protocol Description Language (APDL). The APDL is a machine-readable format which automatically validates messages and is used to generate code which includes type information. APDL services two purposes: to help ensure correctness and to help programs interoperate. APDL documents are identified using global identifiers (e.g. example.com/my-api) and act as a strict contract between programs. The host environment can dynamically route messages using the APDL IDs.
The Austin DataBase (ADB) service uses APDL documents to describe data records. As with APIs, the data-record descriptions help ensure correctness and interoperability between programs. As the Austin Architecture is designed for decentralized/distributed computing, the APDL documents are key for coordinating compatibility between deployments and between programs.
1 Both of these choices may evolve over time; for instance, the host environment could be rewritten in other languages for performance, and user programs could be replaced with docker or wasm containers to support more programming languages. Additionally, better security primitives such as gvisor or firecracker are needed to secure the user programs.
Austin's core goal is to provide a Home Cloud network. This kind of distributed and decentralized computing requires:
- Interoperation between services and record schemas,
- Multiple networking APIs for communication and for collaborative data structures,
- Tools to detect, debug, and solve failures, as well as tools to audit the security of the system, and
- Convenient solutions to network failures.
The Austin Architecture solves 1 with strictly-defined messaging and data formats which provide validation and code generation. It solves 2 with Hypercore Protocol services. It solves 3 with tooling baked into the host environment. It solves 4 by using CRDTs and eventually-consistent databases.
The Austin Architecture is designed to fit a number of environments and use-cases. The host environment and the core services should adapt to each use-case as needed. Variations could include new hosting environments (mobile phones, data centers / clusters) new technologies other than Hypercore (IPFS, SSB, blockchains) and new user devices (VR/AR). The goal is to minimize changes required to support new use-cases, thereby achieving program portability. Portability is key to software longevity.
Austin's strict protocol descriptions make it possible to describe endpoints by their behaviors. The endpoints are bound by the description (a "contract") as conformance-validation is applied on both ends of the exchange. This system makes it possible for programs to be glued together (composed) at runtime according to their protocol contracts.
Runtime composition greatly enhances the flexibility of the system. Programs can have their behaviors extended by introducing external proxy or terminal endpoints which conform to the expected protocols. For example, a host environment could use a remotely-hosted Hypercore service by replacing the local program with a proxy to the remote. It could also leverage multiple Hypercore services -- both local and remote -- by a proxy program which dynamically selects between the local and remote services. Such a tool could be configured on a per-program basis or globally for the system. Runtime composition is crucial to achieving Austin's goals of portability and ease-of-use.
Ease-of-use applies to developers and end-users. Austin's reliance on strict schemas, baked-in tooling, and small programs is designed to take the pain out of development and administration. Programs should be easy to understand, easy to write, and easy to debug. Configuration should be as minimal and convenient as possible.
The Austin Protocol Description Language (APDL) is a JSON-based schema for describing service APIs and data records.
APDL documents are identified using a hostname/docname scheme, e.g. example.com/my-api. These IDs provide globally-unique identifiers to the documents.
APDL documents must be hosted at the following URI:
https://{hostname}/.well-known/apdl/{docname}.json
Therefore our example of example.com/my-api would be located at:
https://example.com/.well-known/apdl/my-api.json
APDL IDs may include a revision indicator using the form hostname/docname@revision, e.g. example.com/my-api@2. This form is used to describe the minimum-required revision.
The host environment will use cached APDL documents and only download fresh documents under the following conditions:
- No local copy of the document is available, or
- A program has requested a revision which is higher than the locally-cached document's revision.
There are a couple unresolved tasks remaining for the APDL ids and resolution system:
- Programs may wish to use APDL documents which are not exported to other programs, aka "local APDL docs." Local APDLs would be useful for early prototyping or for ADB records which are not meant to be shared with other programs. Local APDLs IDs will require a namespace which is resilient to conflicts and easy for developers to use.
- Developers may wish to use unpublished variations of APDL documents during development. This requires a solution for resolving and mapping to the unpublished documents in the host environment.
- Hosting APDL documents may be onerous for developers. While APDL hosting could be solved by a package-management service, an enticing alternative would be a mechanism for APDLs to be shipped with application code. This would require some form of signature to prove the authenticity of the document's authoring by its ID's hostname.
APDL documents must never break backwards compatibility. This is because Austin applications are not centrally coordinated, and therefore cannot deploy breaking changes safely.
If a breaking change is required, a new APDL ID must be used. For example, if example.com/my-api requires a breaking change, then it should be named e.g. example.com/my-api.v2.
Non-breaking changes are permitted. Every APDL document includes a revision integer which can be used to indicate such a change.
While the "no breaking changes" requirement may seem onerous, there are some simple solutions available. Most APDL schemas are defined using JSON-Schema, which supports a "oneOf" construct to encode multiple different valid structures. This enables an API or data-record APDL document to encode both the "old" and the "new" schemas as valid options.
APDL documents are JSON. They include the following common attributes:
idThe APDL document ID.typeMust be"api"or"adb-record".revisionA number greater than zero indicating which revision this is (optional, defaults to 1).definitionA type-specific schema.templatesA set of template definitions with various uses. Depends on the type; may include record-key generation and UI descriptions of the APIs or records.
For example:
{
"id": "example.com/my-api",
"type": "api",
"revision": 1,
"definition": { /*...*/ },
"templates": { /*...*/ }
}The APDL is still under development and the schema will likely change or expand as the system evolves.
Service APIs (type=api) use a definition which follows the following shapes:
methodsThe set of methods exported by the API{methodName}The name of an individual methodparamsA JSON-Schema defining the expected parameters. If an array is given, the value is treated as the"items"definition of an array with nominimumLengthvalue.- A
nameattribute may be specified on the individual parameters for use in code generation.
- A
responseA JSON-Schema defining the expected return value.
For example:
{
"id": "example.com/my-api",
"type": "api",
"title": "My Example API",
"definition": {
"methods": {
"toUppercase": {
"params": [{"name": "str", "type": "string"}],
"response": {"type": "string"}
},
"concat": {
"params": [{"name": "a", "type": "string"}, {"name": "b", "type": "string"}],
"response": {"type": "string"}
}
}
}
}Service APIs (type=adb-record) use a definition which is a JSON-Schema. For example:
{
"id": "example.com/blogpost",
"type": "adb-record",
"title": "Blog post",
"definition": {
"type": "object",
"required": ["title", "content", "createdAt"],
"properties": {
"title": {
"type": "string"
},
"content": {
"type": "string"
},
"createdAt": {
"type": "string",
"format": "date-time"
}
}
}
}The atx CLI provides a set of tools for working with APDL.
The validate command ensures that an APDL document is correct. This includes a backwards-compatibility check against previously-published documents.
The gen command produces Typescript code for consuming and producing APIs as well as for accessing ADB records. The APDL schemas produce complete type definitions as well as runtime validation which make access convenient.
Generated code is output to the ./gen directory by default. Here is some example usage of the generated code:
// example.com/blogpost (adb-record) usage
import blogposts from './gen/example.com/blogpost.ts'
await blogposts(dbId).list({limit: 5})
await blogposts(dbId).create({title: 'Hello world!', content: '# My first blogpost'})
// example.com/my-api (api) client usage
import myApi from './gen/example.com/my-api.ts'
console.log(await myApi.toUpperCase('hello, world')) // => 'HELLO, WORLD'
console.log(await myApi.concat('left', 'right')) // => 'leftright'
// example.com/mt-api (api) server usage
import MyApiServer from './gen/example.com/my-api.server.ts'
const myApiServer = new MyApiServer({
async toUpperCase (str = '') {
return str.toUpperCase()
},
async concat (a = '', b = '') {
return a + b
}
})
for await (const req of server) {
if (req.url === '/my-api') {
myApiServer.serve(req)
}
}Austin uses JSON-RPC to send messages between services. JSON-RPC was chosen for the following reasons:
- JSON parsing and serialization is optimized in the JS environments which currently implement Austin.
- JSON-RPC is simple to use over many different transports (HTTP, WebSockets, WebRTC, Hyper's swarm sockets).
- JSON-RPC is simple to describe in APDL. (Alternatives such as HTTP/REST are much more complicated to describe.)
The downsides of JSON-RPC are:
- Less compact (this can be alleviated using compression).
- Poor at representing binary data (requires base64 encoding).
- No streams primitive.
The use of JSON-RPC should be re-evaluated as the project evolves.
The most common kind of messaging in the Austin Architecture is "internal," meaning between the host environment and its user programs. (A multi-device host environment may send internal messages over the network, and therefore may not be strictly "local" to a machine.) These messages are delivered using JSON-RPC over HTTP through the host environment's API gateway.
With high-level permissions, a user service may open direct external network connections. The Hypercore service is the most common example of this. By default, user programs are not allowed to access the network directly, and this permission should be granted rarely.
Peer-to-peer messaging over the "Hyperswarm" network is coordinated by the Hypercore service. User programs request peer connections through the Hypercore service's API, which then proxies the messages over the network on their behalf.
HTTP or WebSocket messaging to other devices is coordinated by the host environment. As with p2p sockets, programs must request these connections and have permissions applied on a case-by-case basis.
Because all external messaging is routed through the host environment and core services, we are able to audit, permission, and (when appropriate) dynamically re-route messages.
Program execution is managed by the host environment. Programs are currently run as Deno scripts and are sandboxed using Deno's JS isolate configuration; this may be expanded to other runtimes (wasm, docker) and should use more powerful sandboxes (gvisor, firecracker).
Programs are passed a small set of environment variables and allowed access to two ports, the host environment's port and an assigned port which the program must listen on. No other access to the hosting device is permitted; all external access is accomplished through API cals to the host environment. An authorization token is passed as an environment variable; this token should be included in calls to the host environment using Bearer Authentication.
User program lifecycle is managed by the host environment. Programs are started when a request is sent to them, and killed after an idle period to conserve resources. Programs which export a user interface or public HTTP API are assigned an additional "external port" which points to a proxy managed by the host environment. This "external port proxy" handles the following tasks:
- Starting the program if needed
- Tracking the program access times to detect idleness
- Logging traffic for auditing and debugging
The program's stdout/err are also stored by the host env for debugging.
The API gateway is a proxy through which all programs route their messages. JSON-RPC requests are sent to the endpoint where they are logged, have permissions applied, and then proxied to their proper destinations.
The endpoint is a single HTTP route which takes a number of query parameters relevant to the request meta information.
apiThe target API's APDL ID. (required)appThe target program's ID. (optional)
The host environment maintains a service registry which tracks all program APIs. This registry is used to route API requests automatically when a target app is not specified.
The host environment maintains logs of messaging traffic and notable system events (such as permission grants or user account creations). An additional logging API is provided by the host environment for applications to record their own events.
When possible, structured logs are used. This enables queries against the log to help users diagnose events. A log-viewing GUI is provided to help users debug the system and detect atypical activity.
A future goal is to provide distributed call tracing, though it will likely require instrumentation within user programs.
The Austin Architecture is a multi-user environment. User IDs are automatically generated and opaque. Users may have the "admin" flag set to get special privileges over the system, including the ability to view all logs, install/configure system-wide programs, and read/write all user data.
Programs are either installed for individual users or for the entire system. As a consequence, a program may be installed multiple times (for each user) or once (for all users). Programs may be designed for a single user or multiple user installation, as is appropriate for the program. (For instance, the Hypercore service must be a multi-user program, while a personal website might be a single-user program.)
Permissioning is still under active development. The system must provide permissioning for the host environment's users, the individual programs, and for remote users. This requires a flexible registry for declaring, configuring, requesting, and enforcing permissions.
APIs have permission-sets defined in their APDL document to define permissions for certain method calls. These permissions are enforced by the API gateway. They are granted to the application during install or upgrade. A program notes the permissions it requires in its manifest file.
API calls to programs include metadata which indicate the program and user which originated the call. If additional permissions are required, they can be applied by the program at runtime.
The host environment provides a GUI environment for accessing the system. It takes advantage of APDL documents, program manifests, and core services to provide the following functionality:
- A unified search of user data and programs
- Application management
- A database manager & record editor with automatically-generated forms
- Log viewing and editing
- User and system configuration
The GUI environment is currently shipped with the host environment implementation, though it could conceivably be moved into a separate codebase and swapped out by users.
The following user programs are specified as "core services" to be included in the host environment deployment by default.
The Hypercore service is a program developed by the Hypercore Protocol organization called "Hyperspace." It implements the Hypercore protocol networking and data-storage, and provides APIs for:
- Hypercore reads/writes, storage, and networking
- Hyperswarm sockets
The "hypercore" is a low-level log structure. Higher level data structures have been implemented on top of the log, including Hyperbee (a key-value store) and Hyperdrive (a files system). These higher-level structures are not exported by the Hypercore service's API; instead, they are imported as modules into a dependant program (e.g. in the ADB service) and then given the Hypercore service's log-structure API as their "backend." This design helps keep the Hypercore service as minimal as possible, and gives other programs freedom to evolve their data structures over time.
The "ADB" (Austin DataBase) service is a high-level document database designed to simplify application development. It uses Austin's protocol description language (APDL) to describe data schemas in a globally-interoperable manner, a crucial requirement for decentralized software.
ADB depends on the Hypercore service to store and replicate databases.
The database management service mediates access to the Hypercore and ADB services. It handles the following tasks:
- Instructing Hypercore to swarm and cache databases.
- Managing permissions for applications and users to access databases.
If other database technologies are introduced to the host environment, this service will need to be modified to manage the new database service.
User programs are Deno scripts which bind an HTTP server to their assigned port.
Applications must include an app.json manifest file at the root of their source package. Here is a quick overview of their properties:
- title The title of the application.
- description A short description of what the application is/does.
- author The author of the application.
- license A short string describing how the application is licensed (e.g. "MIT").
- protocols An array of APDL IDs used by the application with an optional minimum revision.
This manifest is likely to expand and change as this system develops.
The following environment variables are passed to the application process:
ASSIGNED_PORT: The port to which the application process' HTTP server should bind.HOST_PORT: The port of the host environment HTTP server, which provides the host APIs.HOST_BEARER_TOKEN: The "Bearer Auth" token which should be passed in the HTTP Authentication header in requests to the host.
These variables may be expanded over time and could potentially include key/values specified by the user.
User programs may currently be installed from the following source URLs:
file://A path-URL which is local to the host environment.https://An HTTP-URL which resolves to a Git repository.
If the program-source is a local file-path, the program is executed directly from that path. Updates are managed externally by whoever manages the host environment device.
If the program-source is a Git repository, the program is installed by cloning the repo to a folder internal to the host environment's config directory. Versions of the program must be indicated as Git tags which match semantic versions. The host environment will periodically check the Git repository for new tags and will suggest application updates to the user accordingly. Updates and processed by checking out the target tag.