Home · Tommy Cheese

OpenCode v2 Extensions: Architecture and Integration

Sun, 13 Sep 2026 00:00:00 +0800

Research date: 2026-09-13. This article describes the general plugin capabilities of OpenCode v2, including plugin types, loading, capability registration, execution hooks, permission boundaries, and lifecycle management, as a reference for plugin development and embedded integration.

The code baseline is the v2 branch of the OpenCode repository under study, at commit 2308db16387c9b59e88d732a93ec9bac54462b03. Source references use repository-relative paths at that fixed commit. The interfaces and behavior described here apply to that version and do not promise compatibility with other OpenCode versions or the legacy plugin API.

This article is a source-code study and capability overview. Code examples illustrate interfaces and calling conventions and have not been independently executed.

Plugin: The Extension Entry Point

What Is a Plugin?

OpenCode extensions connect to existing workflows by registering capabilities and execution hooks. In the current OpenCode v2 implementation, extensions primarily correspond to the Plugin mechanism. Plugins can provide agents, tools, skills, models, and commands, or adjust behavior at specific execution points. The OpenCode core remains responsible for session execution, model requests, tool results, and execution records.

The Plugin Flow: Loading, Capability Registration, and Session Execution

View the Mermaid diagram source

flowchart TD
    A[配置文件、本地插件或 SDK 注册] --> B[插件加载与生命周期管理]
    B --> C[构建助手、工具、技能等能力]
    C --> D[会话选择助手]
    D --> E[准备上下文与工具快照]
    E --> F[请求阶段钩子]
    F --> G[调用模型]
    G --> H{返回内容}
    H -->|工具调用| I[工具执行与钩子]
    I --> J[保存结果与执行事件]
    J --> G
    H -->|最终回答| K[本次执行结束]

Plugin Categories

Internal, external, and SDK plugins differ by their source of integration but ultimately share the same runtime management. All three can register agents, tools, and execution hooks. Their main differences are who supplies the definition, how it enters the host, and whether the host injects additional internal core services. This article covers server-side plugins; TUI plugins for the terminal interface use a separate extension entry point.

Comparison	Internal plugin	External plugin	SDK plugin
Provider	OpenCode source code and distribution	Project or independent plugin package maintainer	Application host embedding OpenCode
Entry point	`PluginInternal`’s `pre` / `post` collection	Directory discovery, configured files, or packages	Embedded host’s `opencode.plugin(definition)`
Loading form	The core imports definitions directly	The module is resolved and its default export is read	The plugin object is passed directly in memory
Available interfaces	Public Context plus host-injected internal services	Public Plugin Context	Public Plugin Context; closures can access dependencies explicitly supplied by the host
Primary configuration	Native configuration, internal services, or built-in defaults	`plugins[].options` → `ctx.options`	Host code and closures; the registration interface has no separate `options` parameter
Update source	Follows the OpenCode code version; some plugins watch configuration themselves	Configuration changes, supported local entry changes, or package configuration changes	The host registers the plugin again, incrementing the internal revision
Instance scope	Activated independently for each Location	Activated according to each Location’s configuration	Definitions are shared within a host; activation is independent for each Location
Use cases	Implementing native agents, tools, providers, and configuration handling	Adding project or business capabilities to an existing OpenCode service	Embedding and managing OpenCode within a JS/TS application

Effect and Promise are two interfaces for authoring plugins. An Effect plugin object using the public interface can either be default-exported and loaded through configuration or passed directly to the embedded SDK for registration. Its core capabilities can be reused, while loading order and configuration delivery differ.

Internal plugins organize some of OpenCode’s own functionality as plugins. They are not special files installed by users in a directory. The core imports them statically and explicitly includes them in the PluginInternal collection. PluginInternal.list() obtains the services needed by the current Location, injects them into internal plugins through Effect.provide(context), and passes the result to the unified loader.

There are two layers of Context here: effect(ctx) still receives the public Plugin Context. Internal plugins obtain additional services such as Config.Service, Permission.Service, Shell.Service, Location.Service through the Effect environment. Their additional capabilities come from explicit host dependency injection, not from a plugin ID beginning with opencode..

Internal plugin example	Collection	Actual responsibility
`opencode.agent`	`pre`	Registering the base definitions of native agents
`opencode.tool.shell`	`pre`	Registering the Shell tool with native execution, permission, and runtime-state services
Native provider, search, and other tool plugins	`pre`	Providing foundational model, search, and tool capabilities
`opencode.config.agent`	`post`	Reading agent configuration and related Markdown files and applying the results to the agent catalog
Provider, skill, policy, and model-variant configuration plugins	`post`	Applying configuration or postprocessing to capabilities contributed earlier

This also explains the relationship between custom agents and internal plugins: users edit agent configuration, while opencode.config.agent reads, watches, and applies it. Each custom agent is an Agent definition and does not require its own plugin. External or SDK plugins can also register Agents, which configuration plugins may subsequently adjust.

Built-in plugins participate in configuration-based enablement and disablement as well. For example, -opencode.config.agent removes the plugin responsible for applying agent configuration from the active set, affecting that configuration capability. Whether to keep a built-in plugin enabled should be decided according to its actual responsibility.

Adding native functionality that depends on private Core services generally requires changing OpenCode’s source and adding the plugin to the built-in collection. New service dependencies also require corresponding changes to service assembly. These changes are built, released, and upgraded with OpenCode. Internal plugins suit engine functionality; ordinary business tools should prefer public interfaces.

Sources: internal plugin collections and service injection (packages/core/src/plugin/internal.ts), native agent plugin (packages/core/src/plugin/agent.ts), agent configuration plugin (packages/core/src/config/plugin/agent.ts), Shell tool plugin (packages/core/src/tool/plugin/shell.ts).

External plugins add capabilities to an existing OpenCode service through files or packages. “External” means that the code is not in the built-in collection; it still runs inside the OpenCode process. The loader supplies the public Plugin Context without the extra Core service injection provided to internal plugins. It can still access files, networks, and other resources permitted by the runtime. This is neither a separate process nor a security sandbox.

External plugins have two discovery paths:

Automatic discovery: scans plugin/ and plugins/ under native configuration directories. The current implementation directly recognizes .ts, .js files and can also resolve package directories and eligible symbolic links. For a package directory, it first checks package.json for a string-valued exports, module, main, then index.ts, index.js. This simple discovery logic should not be treated as support for every complex package-export rule.
Explicit configuration:plugins accepts relative paths, absolute paths, file URLs, and resolvable packages. Relative paths are resolved against the configuration file’s directory. Local paths go through module loading; package targets go through the package resolver, which tries the server subpath or the package root entry.

Automatic discovery results enter the operation list first, followed by explicit configuration. Configuration can therefore disable automatically discovered plugins. Automatic discovery does not directly enumerate .mjs files. Such entry points must be explicitly configured and loaded by the runtime.

{
  "plugins": [
    {
      "package": "./plugins/example.ts",
      "options": {
        "serviceUrl": "https://api.example.com"
      }
    }
  ]
}

The paths above are configuration-format examples. A plugin uses ctx.options to read serviceUrl and must validate required fields, value ranges, and address restrictions itself. Receiving an object does not mean that its business configuration has been validated.

An external module must default-export a plugin object containing id + effect or id + setup:

Authoring style	Initialization entry	Integration and cleanup
Effect	`effect(ctx)`	Executed by the host within the plugin Scope; resources use scoped lifecycles or finalizers
Promise	`setup(ctx)`	The loader uses `fromPromise` to adapt it to an Effect plugin; initialization may return a cleanup function

@opencode-ai/plugin/effect provides the Effect API, while the package root @opencode-ai/plugin provides the Promise API for this branch. The legacy plugin API has a separate v1 entry point; sharing the name “OpenCode plugin” does not imply interface compatibility.

The typical loading chain for an external plugin is:

配置目录 / plugins 配置
  → ConfigPluginSource 生成有序操作与本地文件时间戳
  → PluginSupervisor 解析路径或包、导入模块、校验默认导出
  → 适配 Promise 定义，并注入该来源的 options
  → Plugin.Service 创建 Location 内的插件实例
  → 注册助手、工具、钩子及需要清理的资源

File or configuration changes can trigger regeneration of the plugin set. Updates are limited to the supported discovery and watching scope; arbitrary dependency-file changes do not necessarily refresh immediately. In particular, when an explicitly configured entry is a directory, changes inside that directory are not guaranteed to trigger reloading. Package upgrades should also be managed through explicit deployment or configuration changes.

If module import or export-format validation fails, the loader records a warning and skips that source. Initialization failures after activation begins are handled by unified plugin management. When diagnosing availability, verify separately that the file was discovered, the module resolved, the plugin activated, and the capability registered. A configuration entry alone does not prove that a tool is available.

Sources: discovery and local watching (packages/core/src/config/plugin/source.ts), module loading and configuration injection (packages/core/src/plugin/supervisor.ts), public package entry points (packages/plugin/package.json), Promise adapter (packages/plugin/src/promise/adapter.ts).

SDK plugins are registered directly by an embedded host and suit applications that use OpenCode as an internal execution engine. Here, SDK specifically means the @opencode-ai/sdk-next embedded host in the current branch. OpenCode.create() creates a runtime and an in-memory HTTP routing chain within the application process. That internal chain needs no network listener, though tools, models, and business services can still make their own network requests.

The host submits an Effect plugin object directly through opencode.plugin(definition), bypassing external module discovery, package resolution, and default-export validation. This does not upload code to an already running remote OpenCode service. A regular HTTP client or the ctx.plugin.list() in Plugin Context is not this embedded registration entry point.

The following example shows how an embedded host registers and queries plugins. Dependencies must match the research baseline:

import { AbsolutePath, Location, OpenCode } from "@opencode-ai/sdk-next"
import { Effect } from "effect"

const program = Effect.gen(function* () {
  const opencode = yield* OpenCode.create()

  yield* opencode.plugin({
    id: "example.reviewer",
    effect: (ctx) =>
      ctx.agent.transform((draft) => {
        draft.update("reviewer", (agent) => {
          agent.description = "检查代码并给出修改建议"
          agent.system = "分析代码质量，并说明建议的依据。"
          agent.mode = "primary"
        })
      }).pipe(Effect.asVoid),
  })

  const location = Location.Ref.make({
    directory: AbsolutePath.make(process.cwd()),
  })
  return yield* opencode.plugin.list({ location })
})

const result = await Effect.runPromise(program.pipe(Effect.scoped))
console.log(result.data)

The example agent defines only its purpose and prompt; permissions require separate configuration. When the example finishes, the Scope that owns the host closes. A real embedded application should keep that Scope alive throughout the service lifecycle.

The scope and update behavior of SDK plugins have two layers:

Layer	Managed content	Effective scope
Host registry	`Map<plugin.id, Versioned>`, storing definitions and increasing revision numbers	Shared within one embedded host; each host has its own registry
Location activation set	Plugin instances, Transforms, Hooks, and Scopes within that directory context	Activated and cleaned up independently for each Location

Every registration publishes sdk.plugin.updated. Running Locations regenerate their plugin sets when notified. Locations started later, or restarted after disposal, read the current definitions from the host registry. Registering once at the host level therefore does not mean initialization runs only once. Mutable objects captured in closures may also be shared by multiple Location instances in the same host and must not be assumed to represent a single session’s state.

Registering the same ID again in one SDK registry replaces its definition and creates a new revision. Separate hosts do not overwrite one another. Registration returning means only that the definition was stored and an update published; it does not mean every Location has finished activation. To verify effectiveness, inspect the target Location’s actual plugin and capability lists and wait for activation events when necessary.

The current SDK registry exposes only register and all, with no separate unregister interface. A Location can still disable activation by plugin ID through configuration, but that does not remove the definition from the host registry. Closing the host cleans up runtime resources. A newly created host requires registration again; the in-memory registry is not a persistent installation record.

The SDK registration entry accepts Effect plugins. The external loader’s automatic Promise adaptation does not happen here. To reuse a Promise definition, explicitly use the corresponding fromPromise adapter. The SDK path performs no additional options injection, and the public Context supplies an empty object by default. Hosts typically pass configuration and business clients through factories or closures.

SDK plugins do not automatically receive the Core service environment available to internal plugins. A host may explicitly supply dependencies, but depending directly on private Core services introduces additional version coupling. In the research baseline, sdk-next is still a transitional package marked private: true; this example is not a promise that a stable public npm API is available for installation.

Sources: embedded host entry point (packages/sdk-next/src/opencode.ts), SDK registry (packages/core/src/plugin/sdk.ts), SDK package status (packages/sdk-next/package.json), public plugin host (packages/core/src/plugin/host.ts). Embedded test source (packages/sdk-next/test/embedded.test.ts) covers updates across Locations, reactivation after Location disposal, and isolation between hosts. These cases were reviewed for this article but not independently executed.

The three sources merge into one ordered activation set. Once enabled entries have been determined, the current order is:

内部 pre → SDK 注册插件 → 外部文件 / 包插件 → 内部 post

ConfigPluginSource handles external sources and configuration operations; PluginSupervisor merges them with internal and SDK definitions; Plugin.Service performs the final duplicate-ID checks and manages Scopes, initialization, and replacement. The sources are not three independent execution engines.

This order means that capabilities such as agents contributed by external and SDK plugins can still be adjusted by later configuration plugins. To determine an agent’s final prompt, model, or permissions, inspect the final capability catalog rather than only its initial plugin definition.

Plugin IDs, package names, and entry paths serve different purposes. If configuration loads a file whose exported plugin ID is example.reviewer, disable it with -example.reviewer. Selectors support exact IDs, prefix.*, and *. Configuration operations run in sequence, and later operations may re-enable an existing definition.

Two duplicate-name rules are easy to confuse:

Registering the same ID again in one SDK registry: updates that definition through a supported registry replacement operation.
Different active sources contributing the same ID: causes activation to reject that round’s entire set for duplicate IDs; definitions are not silently overridden according to source priority. Do not submit the same plugin definition through both an external file and SDK registration.

Also, the current Plugin.Service skips activation when the IDs and versions of the entire ordered set are unchanged. When the set changes, it iterates over the new set, cleaning up and reinitializing existing plugins. Updating one source can therefore reinitialize other, unchanged plugins. All three plugin types must release resources correctly and must not treat initialization as a business action that runs only once. A failed replacement attempts to restore the plugin’s previous version, but cannot undo external business side effects.

Sources: source merging and enablement order (packages/core/src/plugin/supervisor.ts), unified activation and replacement (packages/core/src/plugin.ts), configuration and loading-order test source (packages/core/test/config/plugin.test.ts).

Core Plugin Mechanism 1: Transform

Transforms declare capabilities; Hooks participate in an individual execution. When choosing an extension point, first determine which domain the requirement belongs to.

Requirement	Native extension interface
Register, modify, or remove agents	`ctx.agent.transform`
Register tools that can actually execute	`ctx.tool.transform`
Provide skill instructions and resource entry points	`ctx.skill.transform`
Add commands	`ctx.command.transform`
Adjust providers, model catalogs, and default selections	`ctx.catalog.transform`
Provide reference-material sources	`ctx.reference.transform`
Configure authentication and connections	`ctx.integration.transform`
Provide a search backend	`ctx.websearch.transform`
Modify the current request’s context and visible tools	`ctx.session.hook("context", ...)`
Inspect tool inputs or organize results	`ctx.tool.hook(...)`
Adjust the model SDK or model instance	`ctx.aisdk.hook(...)`
Adjust model HTTP requests or responses	`ctx.session.hook("http.request" / "http.response", ...)`
Adjust command creation parameters	`ctx.shell.hook("create.before", ...)`
Observe live events	`ctx.event.subscribe()`
Create, submit to, wait for, or interrupt sessions	`ctx.session.*`

State domains such as Agent, Catalog, Command, Integration, Reference, and Skill retain active Transforms and rebuild their state from the base state in sequence. Unloading a plugin removes its Transforms and regenerates the result. A Transform should edit only the Draft supplied to the current callback, retain no Draft references, and avoid external business side effects during rebuilding. If external data is needed, load and store it first, then trigger the domain’s reload().

The underlying Tool implementation uses Scope-bound registration and request snapshots. Its lifecycle ownership matches the other domains, but that does not mean all transform methods use identical state containers or return types.

Core Plugin Mechanism 2: Hook

A Hook is an extension point provided by the host within an execution flow. Plugins register callbacks at these points. When OpenCode reaches the corresponding stage, it invokes the callbacks with that stage’s context. Plugins can read data, adjust parameters, or process results according to the interface contract, participating in model requests, tool execution, and related flows.

Using a Hook has two stages: registration and invocation. During initialization, a plugin declares its participation through interfaces such as ctx.session.hook(...), ctx.tool.hook(...). The host invokes the callback only when execution reaches that point. Once registered, the callback may run repeatedly whenever the point is reached. The host waits for it to complete, then continues according to the interface contract. For example, tool.execute.before allows a plugin to return Tool.Error to reject the tool execution.

Transforms build the available capabilities, while Hooks adjust their behavior at specific execution points. For example, register a tool with ctx.tool.transform, inspect the input of an individual tool invocation with ctx.tool.hook("execute.before", ...), and adjust the context about to be sent to a model with ctx.session.hook("context", ...). Hooks participate in the current call chain, while ctx.event.subscribe() subscribes to events to observe execution.

Runtime Hooks execute serially in registration order. Later Hooks see earlier modifications. If several plugins modify the same field, inspect the final loading order. Built-in plugins are divided into leading and trailing collections, so external plugins cannot simply be assumed to override everything last.

Hook	Content that can be changed or observed	Considerations
`session.context`	System prompts, messages, and current-request tool definitions	Agent and model identifiers identify the context; changing messages affects the current request and does not append persistent history
`tool.execute.before`	Tool input	Can return `Tool.Error` to prevent execution
`tool.execute.after`	Successful result or tool error	Useful for organizing results and adding information
`session.http.request/response`	Model HTTP requests and responses	May expose credentials and complete input; control what is logged
`aisdk.sdk/language`	SDK and actual model instance	Suitable for provider adaptation
`shell.create.before`	Command, directory, timeout, shell, and environment variables	String-based rules alone do not provide operating-system isolation

Among the current public Hook types, only tool.execute.before declares a recoverable Tool.Error failure channel. Other Hooks should not be treated as general middleware for throwing arbitrary business errors.

Sources: Plugin Context (packages/plugin/src/effect/plugin.ts), state rebuilding (packages/core/src/state.ts), Hook registration and execution (packages/core/src/plugin/hooks.ts), tool registration and snapshots (packages/core/src/tool.ts).

Tool Examples and Lifecycle

Example 1: Define a Tool with a Plugin Transform

A tool contains a model-visible definition and a host-executed function. After the model generates a tool name and input, OpenCode processes the call using the tool capabilities captured for that request. The function receives the host-provided sessionID, agent, messageID and invocation id. Long-running operations can report progress through context.progress().

The following standalone tool example uses the Effect API:

import { Plugin } from "@opencode-ai/plugin/effect"
import { Effect, Schema } from "effect"

export default Plugin.define({
  id: "example.echo",
  effect: Effect.fn(function* (ctx) {
    yield* ctx.tool.transform((tools) => {
      tools.add({
        name: "echo",
        description: "返回收到的文字",
        input: Schema.Struct({ text: Schema.String }),
        output: Schema.Struct({ text: Schema.String }),
        options: { codemode: false },
        execute: ({ text }) =>
          Effect.succeed({
            output: { text },
            content: text,
          }),
      })
    })
  }),
})

Distinguish result fields by their purpose: output is a structured value for programmatic use when an output Schema has been declared; content is supplied to the model and becomes part of session content; metadata carries bounded additional state. Returning output without declaring an output Schema is an error in the current runtime.

The source reveals two validation details that interface names alone do not convey:

tool.execute.before runs before Schema decoding inside the tool function. A Hook must therefore treat the input as an unknown structure; decoding happens afterward on the potentially modified input.
Effect Schema and supported Standard Schemas validate inputs at runtime. The plain JSON Schema branch passes input through directly. The plain JSON Schema output branch also checks only that the result is a JSON value, not that it satisfies each declared constraint. Business tools must not equate supplying a JSON Schema to the model with server-side parameter validation.

In the current tool pipeline, failure of the pre-execution Hook immediately aborts subsequent processing. Do not assume that execute.after always runs for such rejections. An audit design must cover rejection paths instead of observing only events after successful execution.

Expected, recoverable tool failures can be mapped to Tool.Error. Cancellation, unexpected programming defects, and successful results must retain distinct meanings; do not swallow them all and return ordinary success text.

Sources: tool types (packages/schema/src/tool.ts), tool execution wrapper (packages/core/src/tool.ts), actual parameter validation (packages/core/src/tool/runtime.ts).

Example 2: Compose Tool Calls with Code Mode

Code Mode determines how some tools are presented and composed. The current branch exposes tools with codemode: false directly to the model. Other tools may enter the Code Mode catalog, where execute runs a small piece of code to combine calls and process results. Actual tool calls made within Code Mode still return to the host’s tool execution path.

This suits sequential queries, filtering, and aggregation. The Code Mode runtime restricts direct file access, imports, and similar operations, but these restrictions do not mean that the entire OpenCode service or external plugins run in an operating-system sandbox. The echo example above sets codemode: false to demonstrate direct exposure of a tool to the model.

A model request captures a snapshot of tool registrations when it is prepared. Hot-updating a plugin does not cause an already prepared request to use a completely new tool catalog; subsequent requests prepare capabilities again. Plugin-contributed execution functions should also avoid depending on unowned background resources that may already have been cleaned up.

Sources: tool classification and snapshots (packages/core/src/tool.ts), Code Mode(packages/core/src/codemode/tool.ts), current-request tool availability checks (packages/core/src/session/model-request.ts).

Plugin Locations

Plugin instances are managed per Location; persistent business state requires separate storage. A Location is an execution context made up of a directory and an optional native Workspace identifier. One OpenCode process can host multiple Locations simultaneously, and the same plugin may activate independently in each.

Each plugin instance owns a Scope containing its Transforms, Hooks, tool registrations, and correctly bound resources. Closing the Scope cleans up those registrations. Plugin-owned timers, network subscriptions, and file watchers also need cleanup bindings. Promise plugins may return a cleanup function from setup, while Effect plugins use the corresponding scoped resources and finalizers.

首次加载 → 创建 Scope → 注册能力与钩子 → 服务会话
文件或配置变化 → 替换插件 → 清理旧 Scope → 激活新版本
新版本激活失败 → 尝试恢复旧版本 → 恢复失败则停用

Local plugin entry files and configuration sources are watched for changes. If an explicitly configured entry is a directory, changes to files inside it do not receive the same automatic hot-update guarantee. Loading, updates, and stopping also depend on actual watcher behavior and runtime state. A claim of hot-update support does not replace verification through the native capability catalog.

Plugin recovery restores code and registrations only; it cannot undo side effects already caused in files, databases, or remote services. Scheduled tasks, approval state, retry counts, and idempotency keys should be maintained in reliable storage. In-memory session state must at least be isolated by sessionID. Module-level global variables may also be shared across plugin instances and require particular care.

Sources: plugin Scopes and recovery (packages/core/src/plugin.ts), plugin file watching (packages/core/src/config/plugin/source.ts).

Plugin Best Practices

Prefer Existing Configuration and Public Interfaces:

Use Agent configuration when changing only prompts, purpose, or step limits.
Use a Skill to provide methods and knowledge.
Register a Tool to add executable operations.
Use the corresponding Hook to adjust behavior at an execution point.
When a plugin needs persistent business state or external services, explicitly design storage, authentication, transactions, and idempotency handling. Do not rely on plugin memory to provide these guarantees.

A single-responsibility plugin may consist of one TS extension file. Split capabilities into multiple plugin IDs only when they need independent enablement, versions, or failure policies. Ordinary external plugins should depend on public plugin, client, and schema interfaces and avoid private Core implementations. Distributions should specify compatible host and dependency versions, module entry points, and build methods.

When upgrading a Plugin, check at least the following behaviors instead of merely confirming that its module can be imported:

Agents, tools, and skills appear in the actual capability catalog, and their contributions disappear after unloading.
Multiple-plugin ordering, configuration overrides, and final state after failed hot updates behave as expected.
Tool input, output, cancellation, and execution rejection retain the correct semantics.
Session model and agent selections actually take effect, and subagents receive the intended permissions and context.
Custom tools and the services they call correctly check session ownership, business authorization, state transitions, and duplicate requests.
Persistent state can be restored after a service restart, and missing live events are handled correctly.
Tool results and events accurately represent progress, completion, failure, and cancellation.

Rereading Clean Architecture (6): Boundaries

Mon, 29 Jul 2024 06:13:15 +0800

What Are Boundaries?

Software architecture is the art of drawing boundaries.

Boundaries divide software into elements and constrain dependencies across their two sides. Drawing boundaries early in a project allows decisions to be deferred, ensuring that those decisions will not later interfere with the system’s core business logic. Boundaries also reduce or eliminate unnecessary architectural coupling. Coupling—especially coupling caused by premature, immature decisions about frameworks, databases, and other details that should have been postponed—is what consumes the most human effort in a system.

How should we draw boundaries? Two basic principles help:

Place boundaries between unrelated concerns, such as the GUI and business logic. Input and output through a GUI are incidental to the business logic, which should support different GUI implementations.
Draw boundaries along the axes of change in a system. Components on opposite sides should change for different reasons and at different rates. The principles that guide these axes are SRP at the module or class level and CCP at the component level.

Having established why boundaries matter and where they belong, we can consider the basic process of drawing them:

First divide the system into components. Some contain core business logic; others are plugins that provide necessary functionality unrelated to that core.
Modify the source code so that these noncore components depend on the system’s core business logic components.

Drawing boundaries is a concrete application of dependency inversion and stable abstractions. Dependency arrows should point from low-level concrete implementations toward high-level abstractions.

Anatomy of Boundaries

A system’s architecture is defined by its software components and the boundaries between them

Boundaries take several forms. This chapter examines them through calls that cross boundaries.

At runtime, crossing a boundary means a function on one side calls a function on the other and passes data to it. Designing these calls properly requires careful control of source-code dependencies. Without that control, changing one module’s source may force other modules to change or be recompiled and redeployed. Clear boundaries help reduce this effect.

Cross-boundary calls occur at several levels: source code, deployment, services, and physical boundaries.

Crossing Boundaries at the Source-Code Level

Source-code boundaries appear in monolithic architectures. Such architectures generally need some form of dynamic polymorphism to manage internal dependencies.

The simplest cross-boundary call occurs when a low-level client calls a high-level service function. Runtime and compile-time dependencies then point in the same direction, from the low-level component to the high-level component.

(Original diagram unavailable)

When a client in a high-level component needs to call a service in a low-level component, dynamic polymorphism is needed to invert the dependency. The Service interface in the diagram is a form of SPI.

(Original diagram unavailable)

Crossing Boundaries at the Deployment Level

At the deployment level, deployable units are packaged into a convenient file format. Apart from this distinction, components decoupled at deployment boundaries are nearly identical to a monolithic structure.

Deployment boundaries involve physical boundaries because separately deployed units must communicate across them. Common physical boundaries include dynamically linked libraries, threads, and local processes.

Crossing Boundaries at the Service Level

Services form the strongest architectural boundaries. A service is a process. When defining architectural boundaries at this level, minimize communication wherever possible, and ensure that it can tolerate high latency. Otherwise, the same rules used for local processes apply: lower-level services should act as plugins for higher-level services.

Rereading Clean Architecture (5): Software Architecture

Sun, 28 Jul 2024 19:23:23 +0800

It is time to turn to software architecture. Having covered modules, classes, and components, this article focuses on the basic definition of architecture.

Before We Begin

🌟 A software architect must be a programmer and must continue doing hands-on development. Without personally experiencing the difficulties caused by a system’s design, architects lose touch with the pain of poor design and gradually lose their sense of direction.

What Is Software Architecture?

Architectural work is fundamentally about deciding how to divide a system into components and arrange their relationships and communication—in other words, defining boundaries.

Architectural design generally has two purposes:

To make components easier to develop, deploy, run, and maintain.
To make all this work easier by keeping as many options open as possible for as long as possible;

The quality of a system’s architecture is not closely related to whether its behavior works correctly. Many systems with poor architecture work perfectly well. The trouble usually appears during development, deployment, and subsequent enhancements. This supports the author’s view that architectural value can outweigh behavioral value.

The goals of architectural design are:

Core goal: organize architecture around use cases, isolating them from surrounding concerns by postponing decisions and preserving options.
Primary goal: support the full software lifecycle. Good architecture makes a system understandable, changeable, maintainable, and easy to deploy.
Ultimate goal: maximize programmer productivity while minimizing the system’s total operating cost.

Architectural design focuses on separating policies and regrouping them according to how they change, thereby establishing boundaries. Applicable principles include SRP, CCP, CSP, SDP, SAP, and others.

The Responsibilities of Architecture

Architecture supports development, deployment, operation, and maintenance.

Development: architecture should facilitate software development, so different teams may need different architectural designs. This reflects Conway’s law to some extent.

Why does Conway’s law imply that architecture reflects a team’s structure? This video explains:Conway’s Law: Why Does Your Architecture Reflect Your Team Structure? — Bilibili

Deployment: architecture should make deployment easy, ideally with one click.

Operation:

As mentioned earlier, architecture has less influence on runtime efficiency than on the other concerns. The author attributes this largely to the fact that additional hardware can compensate for operational shortcomings, while architectural design requires more expensive human effort.
Beyond efficiency, architecture should reveal the system’s runtime requirements. A well-designed architecture makes operation apparent to developers. Use cases, features, and required behaviors should be visible as first-class elements, simplifying understanding. The chapter “Screaming Architecture” explores this idea in detail.

Maintenance: maintenance is usually the most expensive part of a system’s lifecycle. Its costs fall into two categories: spelunking and risk.

Spelunking: the effort of exploring an existing system to find the best location and method for adding a feature or fixing a problem.
Risk: the possibility that those changes introduce new problems, along with the cost of managing that possibility.

Keeping Options Open

Recall the two kinds of value: behavioral and architectural. Increasing architectural value means making software more flexible, and flexibility comes from keeping more architectural options open.

The elements of a software system fall into two categories: policy and details. Policy contains the business rules and procedures that provide the system’s real value. Details enable users, other systems, and programmers to interact with policy without defining it. I/O devices and databases are examples. Choices about these details should remain open for as long as possible.

The architect’s goal is to create a system in which policy is the fundamental element and details are separated from it, allowing decisions about details to be deferred. The later a decision is made, the more information is available to make it well. A good architect works to maximize the number of options that remain open.

The author uses device independence to illustrate the value of deferring details. Modern operating systems support many I/O devices, but early computers relied largely on punched paper tape. Programmers naturally coupled tape-reading and tape-writing code to the system. Magnetic tape required new code, then optical discs required more. To avoid repeated implementation and poor adaptability, developers introduced device independence: devices were abstracted behind functions that the operating system called, and developers supplied the concrete implementations. I/O became a plugin to the operating system. This was an early form of OCP: welcome additions 👏 and resist modifications 🥊.

Preserving Independence

Good architecture provides sufficient independence.

Decoupling helps preserve that independence.

Decoupling can be horizontal or vertical.

Horizontal decoupling, or decoupling by layer: divide a system into layers such as the UI and database.
Vertical decoupling, or decoupling by use case: also divide the layers into vertical slices by use case. For example, separate the UI for adding an order from the UI for deleting an order.

Decoupling according to reasons for change allows new use cases to be added without affecting existing ones.

Decoupling can also happen at different levels: source code, deployment, or services.

Source-code level: control dependencies between source modules. Components interact through function calls. This is a monolithic structure.
Deployment level: control dependencies between deployment units such as JAR files. Components may communicate across threads—not necessarily networks—through sockets or shared memory.
Service level: reduce intercomponent dependencies to data structures and communicate through network packets, as microservices do with RPC or REST.

No strict rule makes one level of decoupling universally better. The best choice may change as a project matures. A well-designed architecture should allow a system to begin as a monolith deployed in one file, grow into independently deployable units or services, and return to a monolith if circumstances change. It should also protect most source code from these transitions. For the system as a whole, decoupling mode should itself remain an option. Large deployments may use one mode and small deployments another.

With decoupling established, let us consider its effects on independence.

Operational Independence

If use cases with different concerns are properly isolated, high-throughput and low-throughput use cases naturally separate. Separating the UI and database from business logic allows them to run on different servers. Applications requiring greater bandwidth can also run multiple instances across servers.

Development Independence

When a system is properly decoupled by layers and use cases, its architecture supports multiple teams regardless of whether they are organized by features, components, layers, or another division of work.

Deployment Independence

With effective decoupling, layer implementations and use cases can even be hot-swapped while a system runs. Adding a use case may require only adding new JAR files or starting new services, leaving everything else unaffected.

What Is Code Duplication?

There are two kinds of duplication:

True duplication: repetition that should be eliminated.
False duplication: code that looks similar but follows different evolutionary paths, with different rates and reasons for change. CRP calls for code with different usage patterns to live in different components. Ignoring this can make false duplication look like code that should be merged. Some apparent duplication is necessary—remember the salary-calculation example?

(End of Part 5)

An Introduction to Domain-Driven Design

Fri, 26 Jul 2024 19:49:45 +0800

Overview of Domain-Driven Design

Domain-Driven Design (DDD) is a model-driven approach that captures domain knowledge in a domain model and uses that model to build more maintainable software.

DDD divides the design process into strategic and tactical design. Strategic design addresses domains, subdomains, and bounded contexts, while tactical design addresses entities, value objects, domain events, and related concepts. Their relationship is shown below:

Concepts in Strategic Design

Domain

A domain is the problem area that a system addresses. Product information management, for example, can be a system’s domain.

Subdomain

A domain can be divided into subdomains based on differences in the language used:

Core domain: the subdomain that determines a product’s competitive advantage. It contains the most important, distinctive, and central business functionality.
Generic subdomain: general-purpose functionality shared by multiple subdomains, such as authentication and authorization. These systems are not constrained by company-specific characteristics and need little customization.
Supporting subdomain: a subdomain containing neither core nor generic functionality. It is specific to the business but is not broadly reusable, such as a data dictionary for reference codes.

Bounded Context

A bounded context defines a boundary within which a particular model is used to solve a particular problem.

A bounded context is a subdomain or a collection of subdomains, Ensure that a bounded context supports a complete business process, Ensure that the domains involved in that process belong to the same bounded context. Bounded contexts provide a basis for dividing microservices, with each bounded context corresponding to a microservice.

Why do we need bounded contexts? Because context can make coding difficult. Consider an example:

In recruitment, “platform” might describe the school or organization someone comes from. In diving, it means the structure from which an athlete dives. In rail transport, it means the area where passengers board trains. Imagine that a diver, an HR specialist, and a train attendant meet without knowing one another’s professions. The HR specialist asks the diver, “Which platform are you from?”…

All three use the same word, “platform,” but without an agreed context they may interpret it differently. The solution is to separate the diving, recruitment, and transport domains and define terminology within each. Discussion within one domain then avoids as much ambiguity as possible. This is the agreement that a context establishes, and it explains why it is useful to define the domains involved in a business process within a bounded context: to reduce ambiguity.

Software design encounters the same problem. Suppose you design a large product covering diving, recruitment, and passenger transport without separating these services. You use a platform data structure for diving platforms, another platform for recruitment organizations, and yet another platform for railway platforms. When the three business areas inevitably meet, disaster follows: the screen fills with almost identically named variables that mean different things. What can we do?

By defining the precise meaning of a term within a particular domain, we can use that term freely within it. This is the essential role of a bounded context.

Concepts in Tactical Design

Value Objects and Entities

Let us begin with value objects and entities.

A value object is identified by its attribute values. If all the values inside two value objects are equal, we consider the objects equal. The attributes of a value object are therefore immutable.
An entity is a business object with a unique identity, state, and a lifecycle. Its attributes may change. Two entities with identical attributes are not necessarily the same entity; only identical identities, such as IDs, make them the same.

An example helps distinguish value objects from entities:

Suppose we have a white value object Color white{R:255, G:255, B:255} and a tire entity tire{Air:,Size:}:

Mutability: every attribute of white is immutable, because changing any of them means the object no longer represents white. A tire’s pressure, size, and other values may change, because it remains a tire even when these parameters change.
Equality: the immutability of a value object gives it its equality rule. Two white objects must have the same values, so two value objects are equal when all their internal values are equal. An entity does not have this property because its attributes are mutable.

Entities generally take one of four forms:

Bloodless model: the model contains only data definitions and getter/setter methods. Business and application logic reside in the service layer. Such a class is called a POJO in Java.
Anemic model: the model contains some business logic, but excludes logic that depends on the persistence layer. Persistence-dependent business logic resides in the service layer.
Rich domain model: the model contains all business logic, including logic that depends on the persistence layer.
Overloaded domain model: application logic unrelated to the business, such as authorization and transactions, is also placed in the domain model.

Repository (Repo)

A repository provides storage operations for entities. Its operations should be as low-level as possible, have short names, and remain limited in number. A MyBatis Mapper in Java can be viewed as one implementation of a repository.

Aggregates and Aggregate Roots

An aggregate is a larger unit of encapsulation that groups entities and value objects sharing a lifecycle and inseparable business meaning. Only the aggregate root exposes references outside the aggregate. Aggregates also express cohesion. Aggregate roots may call one another, and their names are generally nouns.

Aggregate roots provide encapsulation in the following ways:

All operations on an aggregate must go through its root; external code must not directly manipulate its elements. For example, purple paintwork + tires + a steel frame + … = a car. When driving, we do not independently operate the tires, steering wheel, or paintwork as separate objects. We operate the car as an aggregate root, indirectly affecting its other entities and value objects.
An aggregate defines a boundary within which every component must remain valid with respect to the business logic.
An aggregate must be operated on within an atomic transaction, or inconsistencies may arise. It is the unit of operation: retrieving it from the repository, modifying it, and putting it back form one atomic operation. For example, if a car aggregate contains wheel entities and a steel, aluminum, or carbon frame entity, removing a wheel for maintenance without reinstalling it cannot count as a completed operation.

Domain Events

Changes to an entity’s attributes produce domain events. A domain event represents something a domain expert considers important: a noteworthy occurrence within the domain. It usually indicates a change in a domain object’s state.Domain events carry messages and trigger further actions in a system, making them an important way to decouple domain models. We often use a message queue to deliver domain events, so that every subscribing subdomain can perform its own response.

A message bus is another implementation option 👋.

For example, a change in a tire entity’s pressure may emit a leaked event. This conveys that the tire has leaked and triggers other responses, such as changes in the power system or steering feel. Domain events are generally named in the past tense to indicate that they have already happened and cannot be undone.

Domain Service

Some actions in a domain appear not to belong to any object. They represent important domain behavior that cannot be ignored or simply attached to an entity or value object. When such behavior is identified, the recommended approach is to declare a service for it: a domain service.

A domain service is not the same as a service in microservices. Services and aggregate roots seem similar because both may operate on multiple entities, but their purposes differ: an aggregate root groups entities, while a service synchronizes the states of multiple entities—for example, adding an item entity to a list entity (such as delivering mail to an inbox). Service abstractions are therefore generally named with verbs.

Domain Modeling in DDD

The usual steps in DDD domain modeling are:

Identify initial subdomains and bounded contexts from the requirements, along with the relationships between contexts.
Analyze each context to identify entities and value objects, and decide which model style each entity needs: bloodless, anemic, rich, or overloaded.
Associate entities and value objects, organize them into aggregates, and define aggregate boundaries and roots.
Design repositories for aggregate roots and consider how entities and value objects should be created.
Implement the domain model in the project, evaluate its suitability in practice, and use the results to identify shortcomings and refactor.

This is DDD’s two-stage design approach: strategic design first, followed by tactical design.

In practice, I suggest using bloodless entities, whose methods consist only of setters and getters, or anemic entities containing simple logic without database operations, such as attribute validation.

This is one approach to DDD modeling: top-down design. A bottom-up approach is also possible, starting with domain objects such as entities and value objects before identifying subdomains and bounded contexts.

(End of section)

Rereading Clean Architecture (4): Component Principles

Sat, 06 Jul 2024 17:27:18 +0800

Rereading Clean Architecture (Part 4): Component Design Principles

Part 3 discussed the design of modules and classes. This article goes a step further and explores component design.

A component is a deployment unit: the smallest part of a software system that can be deployed independently. Components can also be developed independently. Component-based plugin architectures have become a familiar way to build software.

Component Cohesion

Component cohesion tells us which modules and classes should be grouped into components. Three main principles apply:

Reuse/Release Equivalence Principle.
Common Closure Principle.
Common Reuse Principle.

Reuse/Release Equivalence Principle (REP)

REP states that the unit of reuse should be the same as the unit of release.

REP approaches the problem from the perspective of code reuse. Code sharing a common theme and function can be grouped into a component, and reuse should take place at that component boundary.

Put simply, a packaged and released component may receive a version number or unique identifier. We reuse code by importing an existing library: the unit imported is the packaged component, and the version identifies that release.

Common Closure Principle (CCP)

CCP states that classes changing together for the same reason should belong to the same component, while classes changing at different times for different reasons should belong to different components.

CCP takes the perspective of maintenance. Grouping code that changes for the same reason reduces the effort required for release, verification, and deployment.

As discussed earlier, CCP is the component-level version of SRP. Both can be summarized as follows:

Gather together things that change for the same reason and at the same time. Separate things that change for different reasons and at different times.

For SRP, “things” means functions and other elements that make up a class or module.
For CCP, “things” means the classes and modules that make up a component.

Common Reuse Principle (CRP)

CRP states that users of a component should not be forced to depend on things they do not need.

CRP calls for code with different usage patterns and frequencies to be placed in separate components. It guides component boundaries and generalizes the Interface Segregation Principle (ISP).

As a generalized form of interface segregation, CRP and ISP can both be summarized as:

Do not depend on things you do not need.

For ISP, “things” means functions or classes exposing unnecessary methods.
At the component level, “things” means classes or modules containing unnecessary functions.

Reflection

How do REP, CCP, and CRP relate to one another?

They are actually in competition.

REP and CCP are inclusive principles: they tell us which classes and modules belong together. CRP is exclusive: it tells us which classes and modules should not be grouped together.

If an architect considers only REP and CCP, components will include many parts their users do not need, causing too many unnecessary releases.
If an architect considers only CRP and CCP, reuse will become very difficult.
If an architect considers only REP and CRP, changing some classes or modules will inevitably require changes in many related modules.

The architect must balance these three principles. Component composition should evolve as the project’s priorities shift between development and reuse, so its direction must be assessed dynamically.

Component Coupling

Component coupling tells us how to arrange relationships between components. It includes three principles:

Acyclic Dependencies Principle (ADP).
Stable Dependencies Principle (SDP).
Stable Abstractions Principle (SAP).

Acyclic Dependencies Principle (ADP)

ADP states that the component dependency graph must contain no cycles.

Version management addresses “the morning-after syndrome,” but requires ADP. First, consider the author’s description of the syndrome:

You spend an entire day getting some code to work, only to arrive the next morning and discover that it has mysteriously stopped working. Most likely, someone else changed a component your project depends on.

There are generally two ways to address this:

Weekly builds.
Version management.

Weekly builds: everyone works in their own repository, then the project is built and integration conflicts are resolved at a fixed time each week, such as Friday.

The limitations are clear:

As the project grows, integration becomes increasingly difficult to complete on time.
The whole project becomes harder to build and test, feedback cycles lengthen, and development quality declines.

This is why version management is needed.

Version management: when a component releases a new version, each dependent team can decide whether to adopt it immediately.

Version management does not permit cycles in the component dependency graph. It must follow ADP, or the morning-after syndrome becomes unavoidable. Why?

A dependency cycle effectively combines all its components into one larger component. Their versions must remain coordinated for other components to depend on them successfully. Cycles also complicate testing: although mocking is common, repeatedly mocking components throughout a cycle is an awkward approach.

How can circular dependencies be removed? There are two options:

Introduce interfaces through dependency inversion.
Create a new component.

Using DIP to break a cycle is straightforward. The following example from work shows how a new component can achieve the same result.

A team is designing a system with DDD. A Dog entity depends on Dog Repo to perform Save operations, so the Entity module depends on Repo. Unfortunately, Dog PO is also defined inside Entity, while Repo.Save needs that persistence object for conversion and storage. Entity and Repo now depend on one another. Go rejects mutually dependent packages and reports an import cycle. How should this be resolved?

Create a separate PO module and move the PO out of Entity. Move the relevant PO-dependent functions from Repo into it as appropriate, removing the cycle. Dependency injection is another option. In Spring Boot, @AutoWired can address this kind of wiring problem; conceptually, a dependency container introduces another component through which the dependency cycle is resolved.

There are many similar examples. Recall the library version conflicts that can occur when installing Python packages with Anaconda…

As an aside, I initially assumed that component structure should map directly to system features: one component per feature, with the dependency graph mirroring the functional decomposition. That suggested the component dependency graph could be produced at the very beginning of system design.

The author explicitly argues that top-down design of the complete component structure is impossible. The book explains:

[!NOTE]

The component graph must change and grow with the software system; it cannot be perfectly designed at the outset. Component dependencies do not map one-to-one to features. Instead, they form a map of the application’s buildability and maintainability.

As more modules are designed and implemented, the need to manage their dependencies emerges. We want changes to affect as little of the project as possible, so SRP and CCP help group classes that frequently change together.

One important purpose of a component graph is to guide the isolation of frequent changes. Components that change often should not disturb components intended to remain stable.

As the application grows, reusable components also become more important, bringing CRP into the picture. Finally, when cycles appear, applying ADP causes the dependency structure to shift and expand.

Stable Dependencies Principle (SDP)

SDP states that dependencies must point toward greater stability. In general, the supporting foundations of an architecture should be more stable.

A component expected to change frequently should not be depended on by a component that is difficult to change. Otherwise, the volatile component also becomes hard to change. This is a challenge in software development: a component carefully designed for easy modification can lose that flexibility because someone adds a dependency on it. Following SDP helps avoid this.

Component stability can be assessed with a metric. The author defines the instability metric as: $$ I=\frac{Fan-out}{Fan-in+Fan-out} $$ Fan-in is the number of incoming dependencies and Fan-out the number of outgoing dependencies. Fewer outgoing dependencies imply greater stability: with zero outgoing dependencies, no dependency can force the component to change.

SDP requires each component’s I metric to be greater than that of the components it depends on: the components it points to must be more stable.

SDP does not require every component to be stable. The purpose of designing the component architecture is to decide which should be stable and which should be unstable. An architecture in which everything is stable lacks flexibility and therefore architectural value.

Stable Abstractions Principle (SAP)

SAP states that a component’s level of abstraction should match its stability.

High-level policies should reside in stable components, but this can make those policies difficult to change. Fortunately, OCP tells us that stable components can remain open to extension. Abstract classes provide a way to achieve stability while still allowing extension and modification.

Abstract classes form a buffer between interfaces and concrete classes.

SAP connects stability with abstraction. A stable component should also be abstract so that stability does not prevent extension. An unstable component should contain concrete implementation code so that its instability can be accommodated through easy changes. A component intended to be stable should therefore consist of interfaces and abstract classes that allow future extension.

Like stability, abstractness can be measured with a metric: $$ A=\frac{N_c}{N_a} $$ Here, Nc denotes the number of classes in the component, and Na the number of abstract classes and interfaces. A ranges from 0 to 1: 0 means no abstract classes, while 1 means only abstract classes.

Reflection 🤔

How are SDP and SAP related?

First, consider SDP and SAP in relation to DIP

In effect, SDP + SAP = component-level DIP. SDP requires dependencies to point toward greater stability, while SAP implies that stability requires abstraction. Dependencies should therefore point toward greater abstraction.

How should this be understood?

DIP preserves flexibility at the class level. SDP and SAP together preserve flexibility while providing component stability. Their architectural effect is the same: concrete implementations depend on abstract classes or interfaces. In this sense, SDP + SAP is component-level DIP.

There is a difference, however. At the class level there is no middle ground: a class is either abstract or it is not. SDP and SAP operate at the component level, where partial abstraction and partial stability are possible.

Second, consider their shared goals

SDP directs dependencies toward stability. SAP places high-level policy in abstract components, helping separate policy from details and enabling plugin-based development. Their goals are aligned.

Notice that plugin-based development appears again. Where did we encounter it before? In dependency inversion. This further illustrates why SAP + SDP resembles component-level DIP.

The Main Sequence

Plot component instability I against abstractness A. The line a = -i + 1 is called the main sequence.

The diagram contains three regions:

Zone of pain: near (0,0), components are highly stable and highly concrete, making them difficult to change. Database schemas are an example. Utility libraries also fall into this zone: although their I metric is 1 because they depend on many components and are therefore unstable, changing them can break large amounts of code.
Zone of uselessness: near (1,1), components are highly abstract but have no dependents. Such abstractions often serve no practical purpose. Design problems in this region may reflect historical leftovers, such as old code that was never removed.
Main sequence: components along this line balance the two measures. The ideal positions are its endpoints, (0,1) and (1,0). A good architect aims to move most components toward these positions.

How can a component be evaluated overall? Calculate its distance from the main sequence to quantify how closely the design fits it. This distance is D: 0 means directly on the main sequence, while 1 means as far from it as possible. $$ D=|A+I-1| $$ The amount by which D exceeds zero can then guide component refactoring and redesign.

D has further uses. For example, the mean and variance of D across all components provide statistical measures for evaluating a system design:

In a well-designed system, both the mean and variance of D should be close to zero.
Variance can help establish a threshold for identifying unusual components that deserve attention.
Tracking the variance of D over time also shows how architectural stability and abstraction evolve.

(End of Part 4)

Rereading Clean Architecture (3): SOLID Design Principles

Fri, 05 Jul 2024 14:11:30 +0800

Rereading Clean Architecture (Part 3): The SOLID Principles

There are three main goals when building software modules:

Make software tolerant of change.
Make software easier to understand.
Build components that can be reused across multiple software systems.

The SOLID principles primarily tell us how to organize data and functions into classes and how to connect those classes into programs. They guide module design; other design principles apply at the architectural level.

SOLID refers to the Single Responsibility Principle, SRP; the Open–Closed Principle, OCP; the Liskov Substitution Principle, LSP; the Interface Segregation Principle, ISP; and Dependency Inversion, DIP.

Single Responsibility Principle (SRP)

SRP states that each software module should be responsible for a single function: there should be only one reason for a module to change. Each software module should be responsible to one kind of actor.

At the component level, SRP is known as the Common Closure Principle (CCP).

What happens when a module is designed without following SRP? Consider this example.

Suppose the finance, human resources, and engineering departments all depend on a payroll and working-hours program with three functions:

CalculateSalary: calculate salary.
CalculateTime: calculate working hours.
Save: save information.

The three departments use the program without incident until one day the finance department changes its salary calculation rules. Its maintainer updates CalculateSalary. Then something unexpected happens: salaries for human resources and engineering change as well…

Open–Closed Principle (OCP)

“Welcome additions; resist modifications!”

OCP states that a software system should allow its behavior to be changed by adding code, rather than only by modifying existing code. Well-designed software should be easy to extend while resisting modification.

OCP applies to components as well as classes and modules. It originated in the concept of device independence. Specifically, dependency inversion lets us design I/O devices as plugins, making it easy to add new devices without changing existing ones.

OCP is implemented by dividing a system into components and organizing their dependencies hierarchically so that changes in lower-level components do not affect higher-level components.

Liskov Substitution Principle (LSP)

LSP states that if interchangeable components are used to build a software system, they must follow the same contract so that one can substitute for another.

LSP is fundamentally a principle of substitutability: if, for every object o1 of type S, there is an object o2 of type T such that a program P operating on T behaves identically when o1 replaces o2, then S is a subtype of T.

LSP can and should be applied at the architectural level. Once substitutability is violated, a system architecture must introduce extensive, complicated mechanisms to compensate.

Interface Segregation Principle (ISP)

ISP states that unnecessary dependencies should be avoided. At any level of software design, depending on something that is not needed can introduce unexpected trouble.

Dependency Inversion Principle (DIP)

DIP states that code implementing high-level policy should not depend on code implementing low-level details. Instead, code implementing those details should depend on the high-level policy.

To understand dependency inversion, distinguish control flow from source-code dependency flow. Suppose control flows from component A to component B. In a direct implementation, A imports B, so the source-code dependency also points from A to B. System behavior determines control flow, which in turn dictates source-code dependencies, leaving the architecture little freedom. Introducing an interface changes this: B implements the interface, and A depends on the interface instead of importing B. Across that boundary, the source-code dependency is reversed relative to control flow. This is dependency inversion.

Dependency inversion gives architectural design freedom. A flexible system should depend more on abstractions—such as interfaces and abstract classes—than on concrete implementations.

DIP yields several coding guidelines:

Use abstract interfaces and avoid depending on volatile concrete implementation classes wherever possible.
Do not derive classes from concrete implementation classes.
Do not override functions that contain concrete implementations.
Avoid referring in code to the names of concrete implementations or other things likely to change.

(End of Part 3)

Rereading Clean Architecture (2): Programming Paradigms

Fri, 05 Jul 2024 07:25:33 +0800

Rereading Clean Architecture (Part 2)

A programming paradigm is a way of writing programs. It tells us which code structures to use and when to use them.

Three programming paradigms have emerged: structured programming, object-oriented programming, and functional programming.

The author argues that these approaches do not add weapons to an architect’s arsenal. Architects and programmers already have plenty of weapons; the three paradigms instead impose constraints on their use. That is why they are called paradigms.

Structured Programming

Structured programming restricts and disciplines the direct transfer of program control, especially the unrestricted use of goto statements.

Decomposing Programs

Dijkstra hoped to reason about programs using mathematical proofs: a program would become a Euclidean structure, with new programs assembled from proven structures so that the correctness of the whole could be derived.

He also found that large numbers of goto statements make programs difficult to decompose, while their functions can be expressed entirely with branches and loops.

Structured programming therefore shows that programs can be decomposed. A large problem can be divided into a set of high-level functions, each of which can be divided further into lower-level functions, recursively. More importantly, every resulting function can itself be written using structured programming.

However, programming through formal proof never became mainstream. The scientific method is used more often in practice.

The Scientific Method

Scientific theories and laws can be falsified, but cannot be proved true. Similarly, testing can disprove a program’s correctness but cannot prove it: tests can reveal bugs, but cannot establish that none exist. Using the scientific method, structured programming encourages us to recursively break a program into small, provable functions and then write tests that attempt to show those functions are wrong. If these tests fail to falsify them, we can treat the functions as sufficiently correct and infer the correctness of the overall program.

Object-Oriented Programming

Object-oriented programming restricts and disciplines the indirect transfer of program control.

Specifically, object-oriented programming uses polymorphism to constrain function pointers. This can be understood as limiting the targets a function pointer may reference. For example, polymorphism in Java commonly operates between objects of classes related through inheritance.

The author also regards polymorphism as the defining feature of OOP. It enables dependency inversion:

Dependency inversion introduces interfaces to give us complete control over source-code dependencies without being constrained by the system’s flow of control.

Dependency inversion enables plugin-based development. Inverting the dependencies between business logic and the web UI or database decouples the core business logic from both, turning the UI and database into plugins for that logic.

Object-oriented programming is the ability to control source-code dependencies through polymorphism. This enables architects to build plugin architectures that separate high-level policy components from low-level implementation components. Low-level components can be compiled as plugins and developed and deployed independently of the high-level components.

Functional Programming

Functional programming restricts and disciplines assignment.

In other words, variables in functional programming languages are immutable.

The author makes the following argument, quoted from the book:

“All race conditions, deadlocks, and concurrent update problems arise from mutable variables. If variables never change, races and concurrent updates cannot occur. If lock state is immutable, deadlocks cannot occur either.”

Immutability is feasible if we ignore limits on storage capacity and processing speed. In practice, however, we must account for those limits. To make immutability practical to a useful extent, we need segregation of mutability.

Segregation of Mutability

A common approach is to split an application, or its internal services, into mutable and immutable components. Immutable components perform tasks through pure functions without changing any state. They communicate with one or more nonfunctional, mutable components to modify variable state.

Git provides an example. It records file additions, modifications, and deletions by moving references, while the stored file contents are not actually modified or deleted; new data is added and existing data is retrieved. This is an example of immutability.

Transaction management in MySQL and transactional memory also draw on this idea.

Summary

The three programming paradigms are closely related to software architecture:

Polymorphism is how we cross architectural boundaries.
Functional programming is how we discipline and constrain where data is stored and who can access it.
Structured programming provides the foundation for implementing individual modules.

The three paradigms align with the three major concerns of software architecture: component independence (object-oriented programming), data management (functional programming), and functionality (structured programming).

Further Reflection

All three programming paradigms impose new constraints on programmers. Each limits a way of writing code; none adds a new capability. In other words, programming paradigms tell us what not to do.

(End of Part 2)

Rereading Clean Architecture (1)

Wed, 03 Jul 2024 23:10:20 +0800

Preface

Why read it again? I had already read Clean Architecture once, but without enough project experience and trial and error, I felt that my understanding remained shallow. During my work, I finally had the opportunity to take part in several projects and implement real requirements. Better still, the team used Clean Architecture—more specifically, event-driven development with DDD and Clean Architecture. I will discuss DDD in detail later. Rereading this book 📚 proved very rewarding, so in this series I will describe Clean Architecture from my own perspective. More precisely, these are reading notes that I hope to share with fellow developers.

The book is available here: Clean Architecture — Robert C. Martin — WeRead (qq.com)

This series follows the chapters of the book and gradually explores the ideas alongside the author. Because this is a rereading, I will inevitably refer to ideas from other parts of the book. If you encounter something unfamiliar or difficult to understand, it will probably be covered in a later chapter.

Let us begin.

The Meaning of Design and Architecture

Design and architecture are not fundamentally different. Low-level design details and high-level architectural information together define a software system.

The ultimate goal of software architecture is to meet the requirements for building and maintaining a system with the least human effort. Cost can therefore serve as a measure of the quality of an architectural design.

A system built hastily and without design can become a big ball of mud. In such a system, code quality and improvements to design structure have been neglected throughout development. Careful architectural design can help prevent this outcome and reduce costs.

Software development has two key lessons:

To go fast, first learn to go steadily.
Overconfidence can cause a redesign to fall into the same traps as the original project.

Two Dimensions of Value

A software system offers two kinds of value:

Behavioral value: making a machine operate in a specified way to create or increase profit for its users.
Architectural value: keeping software flexible, so that the cost of changing it depends on the scope of a requirement rather than its shape.

How should we understand the scope and shape of a requirement? In my view, scope means the broad extent of the requirement and the domain to which it belongs, while shape means its specific details.

Which dimension matters more? The author considers architectural value more important than behavioral value, because:

If a program works but cannot be changed, it will stop working when requirements change, and we will be unable to modify it to keep it useful. Its value will then become zero.
If a program does not currently work but is easy to change, fixing it and continuing to adapt it as requirements evolve should both be straightforward. Such a program can continue to create value.

A common mistake shared by business and development teams is failing to distinguish genuinely urgent and important features from urgent but unimportant ones. As a result, important architectural concerns give way to unimportant behavioral features.

Balancing the importance of system architecture against the urgency of features is the responsibility of software developers themselves.

Author’s note: Similar problems arise even between development teams. For example, some front-end teams assume that the back end merely needs to expose a set of interfaces and that this is easy, overlooking the importance of good design.

Keep Fighting for Good Software Architecture ✊

If architectural value is ignored, a system becomes increasingly difficult to maintain until eventually it can no longer be changed at all. If that happens, the development team has failed to push back sufficiently against requirements and has not fulfilled its responsibility.

How to Load PyTorch Model Parameters into MindSpore

Fri, 01 Sep 2023 22:53:58 +0530

Problem Overview

In day-to-day model development and training, many existing open-source projects and paper implementations use PyTorch for model design, development, training, and inference. When we need to develop models with MindSpore, two problems arise:

The model is implemented in PyTorch.
Parameters saved after training a PyTorch model cannot be loaded directly by a MindSpore model.

The first problem can be addressed using the official MindSpore documentation: Typical Differences from PyTorch and PyTorch–MindSpore API Mapping to migrate the model.

For parameter conversion, MindConverter is no longer supported in the latest MindSpore version discussed here. We can therefore convert parameters manually, transforming PyTorch model parameters into a format MindSpore can recognize before loading them.

Solution

I will not repeat the model code migration process here.

The main steps for parameter conversion are:

Load the PyTorch model with PyTorch and obtain its parameters, prams_torch.
Load the MindSpore model with MindSpore and obtain its parameters, prams_ms.
Match PyTorch parameter names to MindSpore parameter names one by one where corresponding parameters exist.
Build a torch_2_ms key mapping and use it to place PyTorch parameter values under the corresponding MindSpore parameter names.
Load the parameters with MindSpore.

Case Study

Different models contain different modules and parameter types. Here, one network illustrates the basic conversion approach; the same reasoning applies to other models.

EfficientNet is a paper published by Google in 2019. See the paper for the detailed network architecture. Here we use EfficientNet+FC as an example of a model with a fully connected layer to explore parameter conversion.

Load the PyTorch Model and Obtain prams_torch

import torch
from test.efficientnet_pytorch.model import EfficientNet as EN_pytorch
import pandas as pd

pytorch_model = EN_pytorch.from_name(cfg['model'], override_params={'num_classes': 3})
pytorch_model.cuda()
pytorch_weights_dict = pytorch_model.state_dict()
param_torch = pytorch_weights_dict.keys()
param_torch_lst = pd.DataFrame(param_torch)
param_torch_lst.to_csv('param_torch.csv')

After this step, the PyTorch model parameters have been saved to param_torch.csv. Inspect the data:

	keys
0	_conv_stem.weight
1	_bn0.weight
2	_bn0.bias
3	_bn0.running_mean
4	_bn0.running_var
5	_bn0.num_batches_tracked
6	_blocks.0._depthwise_conv.weight
7	_blocks.0._bn1.weight
8	_blocks.0._bn1.bias
9	_blocks.0._bn1.running_mean
10	_blocks.0._bn1.running_var

Load the MindSpore Model and Obtain prams_ms

import mindspore as ms
from test.efficientnet_mindspore.model import EfficientNet as EN_ms
import pandas as pd

mindspore_model = EN_ms.from_name(cfg['model'], override_params={'num_classes': 3})
prams_ms = mindspore_model.parameters_dict().keys()
prams_ms_lst = pd.DataFrame(prams_ms)
prams_ms_lst.to_csv('prams_ms.csv')

After this step, the MindSpore model parameters have been saved to prams_ms.csv. Inspect the data:

	keys
0	_conv_stem.weight
1	_bn0.moving_mean
2	_bn0.moving_variance
3	_bn0.gamma
4	_bn0.beta
5	0._depthwise_conv.weight
6	0._bn1.moving_mean
7	0._bn1.moving_variance
8	0._bn1.gamma
9	0._bn1.beta
10	0._se_reduce.weight

Match PyTorch Parameter Names to MindSpore Parameter Names

We now have parameter key tables for MindSpore and PyTorch, provided in the attachments. Comparing their naming conventions reveals consistent patterns, including:

Batch Normalization:
- Weights: weight|bias → gamma|beta.
- Moving mean and variance: running_mean|running_var → moving_mean|moving_variance.
Custom blocks: PyTorch uses the _blocks. prefix.
Other differences.

Key Mapping Table

We can use these patterns to write a Python script that converts key names and generates a mapping table:

Pytorch	mindspore
_conv_stem.weight	_conv_stem.weight
_bn0.weight	_bn0.gamma
_bn0.bias	_bn0.beta
_bn0.running_mean	_bn0.moving_mean
_bn0.running_var	_bn0.moving_variance
_blocks.0._depthwise_conv.weight	0._depthwise_conv.weight
_blocks.0._bn1.weight	0._bn1.gamma
_blocks.0._bn1.bias	0._bn1.beta
_blocks.0._bn1.running_mean	0._bn1.moving_mean
_blocks.0._bn1.running_var	0._bn1.moving_variance
_blocks.0._se_reduce.weight	0._se_reduce.weight

Next, retrieve each weight value from the PyTorch weight dictionary using the corresponding Pytorch_key in the mapping file, wrap it with mindspore.Parameter, and assign it to the corresponding mindspore.key:

for i in ms_param_lst.values:
    ms_key = i
    pt_key = param_mapping[ms_key]
    pt_val = pt_values_dict[pt_key]
    if not isinstance(pt_val, np.ndarray):
        pt_val = pt_val.cpu().numpy()
    ms_val = Parameter(pt_val, ms_key)
    print(ms_val)
    ms_values_dict[ms_key] = ms_val

Load the Parameters with MindSpore

load_param_into_net(mindspore_model, ms_values_dict)

The parameters should now be accepted by MindSpore.

What’s more

When storing parameter values, pay attention to differences in parameter precision between PyTorch and MindSpore.

(End)

Beyond the Gradient: The Hessian Matrix

Fri, 01 Sep 2023 22:53:58 +0530

This article explores the Hessian matrix, a powerful mathematical tool for studying gradient descent. Before discussing the Hessian, we first need the basic concepts of gradients and the Jacobian matrix.

⭐ This article assumes familiarity with gradient descent and basic numerical analysis and linear algebra. Original article

Gradients and the Jacobian Matrix

Gradient descent requires derivative information at the current point of a function. For a function with multiple input directions, its gradient is the vector of partial derivatives in those directions.

The discussion above assumes a single output. When the function’s output is also a vector, we must take the gradient of each output element with respect to the inputs and stack them together. The resulting matrix is the Jacobian matrix.

For example:

If a function $f$ takes three inputs $x1、x2、x3$ and produces one output $y$, its gradient is:

$$ \begin{equation} Grad = [\frac{\partial y}{\partial x_1}, \frac{\partial y}{\partial x_2}, \frac{\partial y}{\partial x_3}] \end{equation} $$

If a function $f2$ takes three inputs $x1、x2、x3$ and produces three outputs $y1、y2、y3$, its Jacobian is:

$$ \begin{equation} Jacobian = \begin{bmatrix} \frac{\partial y_1}{\partial x_1} & \frac{\partial y_1}{\partial x_2}&\frac{\partial y_1}{\partial x_3} \ \frac{\partial y_2}{\partial x_1} & \frac{\partial y_2}{\partial x_2}&\frac{\partial y_2}{\partial x_3} \ \frac{\partial y_3}{\partial x_1} & \frac{\partial y_3}{\partial x_2}&\frac{\partial y_3}{\partial x_3} \end{bmatrix} \end{equation} $$

Second derivatives describe the curvature of a function in a particular direction $d$. This information helps anticipate the behavior of gradient descent. Along direction $d$:

If the second derivative is positive, the first derivative increases along $d$, and the function value decreases more slowly.
If the second derivative is negative, the first derivative decreases along $d$, and the function value decreases more quickly.
If the second derivative is zero, the first derivative remains constant along $d$, and the function value decreases at a constant rate.

⭐ Gradient descent reduces a loss function, so we analyze how derivatives change within a small local segment of a decreasing function. The decreasing side of a quadratic function is often used as an approximation, as in a second-order Taylor expansion or Newton’s method.

The Hessian Matrix

Like the Jacobian, the Hessian matrix contains information about the function’s second derivatives: $$ Hessian = \begin{bmatrix} \frac{\partial^2y}{\partial x_1\partial x_1} & \frac{\partial^2y}{\partial x_1\partial x_2}&\frac{\partial^2y}{\partial x_1\partial x_3} \ \frac{\partial^2y}{\partial x_2\partial x_1} & \frac{\partial^2y}{\partial x_2\partial x_2}&\frac{\partial^2y}{\partial x_2\partial x_3} \ \frac{\partial^2y}{\partial x_3\partial x_1} & \frac{\partial^2y}{\partial x_3\partial x_2}&\frac{\partial^2y}{\partial x_3\partial x_3} \end{bmatrix} $$ Because mixed second derivatives can be interchanged, namely $\frac{\partial^2y}{\partial x_1\partial x_2}=\frac{\partial^2y}{\partial x_2\partial x_1}$, the Hessian is a symmetric matrix. For a symmetric matrix, we can use eigendecomposition to study the relationship between eigenvalues and second derivatives and obtain a directional second derivative efficiently.

For a particular direction d, the second directional derivative can be written as $d^THd$. Therefore:

🔗 Second Directional Derivatives and Properties of the Hessian Matrix — CSDN

If d is an eigenvector of H corresponding to eigenvalue λ:

Since d is an eigenvector corresponding to λ, by definition: $$ Hd = \lambda d\ \Rightarrow d^THd=d^T\lambda d = \lambda d^Td=\lambda \ \ \ 对称矩阵d^T = d^- $$

The eigenvalue λ corresponding to that eigenvector is therefore the second derivative in that direction.
For another direction d, let $e_i$ be an eigenvector of $H$ with eigenvalue $\lambda_i$. From the result above, $$ \lambda_i=e_i^THe_i $$ Any direction $d=\sum_i^mt_ie_i$ is a linear combination of eigenvectors, where m is the number of eigenvalues and $t_i$ is the weight of the $i$th eigenvector. Thus: $$ d^THd=(\sum_i^mt_ie_i)^TH(\sum_i^mt_ie_i)=\sum_i^mt_ie_i^THt_ie_i=\sum_i^mt_i^2\lambda_i $$ The second derivative in an arbitrary direction that is not an eigenvector is therefore a weighted sum of all eigenvalues. In particular, this weighted sum describes an ellipsoid. With two eigenvalues, it is an ellipse, with the equation: $$ y=\frac{\lambda_1}{\frac{1}{t_1^2}}+\frac{\lambda_2}{\frac{1}{t_2^2}} $$

The figure shows that the maximum second derivative is determined by the largest eigenvalue, along the major semiaxis, and the minimum by the smallest eigenvalue, along the minor semiaxis.

Applications of the Hessian Matrix

With the definition of the Hessian established, we can use its properties to analyze optimization methods: identifying local maxima, local minima, and saddle points; choosing learning rates; and assessing how ill-conditioning affects gradient descent. We can also use the Hessian to implement Newton’s method as an optimization algorithm.

(End of section)

Explore

Sat, 25 Jun 2022 18:35:46 +0530