Document subscription formats and ALT runtime integrationresearch

- Add the Roentgen domain glossary
- Record research on 3x-ui subscriptions, runtime integration, and defaults

1 files changed, 161 insertions, 0 deletions

diff --git a/docs/research/3x-ui-subscriptions.md b/docs/research/3x-ui-subscriptions.md
new file mode 100644
index 0000000..d7d7d7a
--- /dev/null
+++ b/docs/research/3x-ui-subscriptions.md
@@ -0,0 +1,161 @@
+# What does a 3x-ui subscription contain?
+
+Research decision ticket for Roentgen. Verified on 2026-10-29. Planning only.
+
+## Answer and scope
+
+A 3x-ui subscription URL identifies a subscriber and returns an ordered collection of connection choices. Its ordinary response is a newline-separated share-link list, usually wrapped in Base64. Depending on the configured endpoint and request, it can instead return complete Xray client configurations, a Mihomo YAML profile, an HTML page, or status JSON. It does not define a universal stable identifier for each exported choice. See the [subscription documentation][docs], [HTTP controller][controller], and [export service][service].
+
+Roentgen's agreed scope is a Linux CLI/TUI that controls the host on which it runs, including when invoked over SSH. Users add or update a subscription URL. The first configuration is initially selected if one exists, but import never enables the proxy. Proxy on/off is explicit. Enabled operation captures host traffic through TUN and persists through a service. Custom routing comes later. These are product requirements supplied for this investigation, not properties of 3x-ui.
+
+Recommendation: make the raw link endpoint the initial interoperability contract. Accept plain and Base64 link lists, preserve their order, and report unsupported entries. Treat full Xray JSON as a separate import format with an explicit normalization policy. Do not execute an imported full configuration as the host's runtime configuration.
+
+## Evidence baseline and limits
+
+The latest official 3x-ui release checked was [v3.11.0][panel-release], commit `c621ba9014a6aa21fd7c9ecf45655e44e24ebf13`. It was also the repository HEAD retrieved during this investigation. The panel release names Xray v26.10.21. The latest official Xray release checked was [v26.10.29][xray-release], and the source checkout was `0afec66085a85191e9dd2a4a6646c7c9a94688d1`. Source citations below pin those commits. Older deployed panels and forks can differ.
+
+Method: browsed official documentation and release pages, then read the official repositories' controller, exporters, ordering helpers, default template, external-link handling, and Xray configuration loaders. No live subscription URL or credentials were provided. Findings describe source behavior, not an end-to-end network test against a deployed panel. No application code, tracker, glossary, or runtime configuration changed.
+
+## Verified response formats and negotiation
+
+| Response | How the panel selects it | Body and caveats |
+| --- | --- | --- |
+| Raw links | Administrator-configured `subPath` plus subscriber ID | Each generated link ends with a newline. `subEncrypt=true` applies standard Base64 to the whole UTF-8 list. Otherwise the list is plain text. Both use a text response. |
+| Xray JSON | Configured `subJsonPath`, when enabled | One full configuration object when exactly one exists and `subJsonAlwaysArray=false`; otherwise an array. The normal endpoint sends `text/plain`; `?view=raw` sends `application/json` and an attachment filename. |
+| Automatically detected JSON | Raw endpoint with enabled JSON auto-detection and matching administrator-configured User-Agent regex | Always an array, with `application/json`. |
+| Mihomo YAML | Configured `subClashPath`, when enabled; also `/mihomo/` unless that alias conflicts | Full YAML profile, not Xray JSON or a bare node array. Sends `application/yaml`. |
+| Legacy Clash YAML | `/clash-legacy/`, when enabled and not conflicting | Compatibility-filtered profile. Returns HTTP 422 if no compatible proxy exists. |
+| HTML | `Accept` containing `text/html`, `?html=1`, or `?view=html` | Subscription information page. Browser output is not evidence that the URL has returned machine-readable configuration. |
+| Information JSON | Raw endpoint with `?format=info` | Status/page data without the `links` field. This is not an Xray configuration. |
+
+These branches, content types, route enablement, and alias collision checks are explicit in the [controller][controller]. JSON cardinality and empty-body handling are in [GetJson][json]. The configured raw and JSON paths are independent settings; a client cannot reliably discover one by appending `/json` to the other. [URL construction][urls] also allows configured public URIs and prefixes.
+
+Raw endpoint precedence is information query, HTML request, hardware-ID enforcement, Mihomo User-Agent detection, JSON User-Agent detection, then links. `Accept: application/json` alone does not select JSON. Both auto-detection modes default to false; JSON itself defaults to disabled. There is no general `?format=json` switch in these handlers. JSON and YAML `?view=raw` bypass the HTML branch. See [controller][controller] and [setting defaults][settings].
+
+Recommended request policy:
+
+- Use GET with a stable, honest `Roentgen/<version>` User-Agent and an `Accept` value that excludes HTML, such as `text/plain, application/json;q=0.9, */*;q=0.1`.
+- Retain the supplied URL and query. Detect known HTML/status selectors and explain why they cannot import; do not silently rewrite unrelated query parameters.
+- Detect format from a bounded body as well as Content-Type. Text/plain is valid for both raw links and JSON. Validate the parsed structure so an HTML login page or information JSON cannot become an empty success.
+- Never impersonate another client to trigger auto-detection. Requesting an explicitly supplied JSON URL is more predictable than depending on an administrator's User-Agent regex.
+
+## Verified protocols and encodings
+
+The raw export dispatch handles the following forms. These are observed panel outputs, not a promise that every protocol will belong in Roentgen's first release. See the [link generators][service] and [raw dispatch][dispatch].
+
+| Exported choice | Raw representation | Xray implication |
+| --- | --- | --- |
+| VLESS | `vless://<id>@host:port?...#remark` | Xray `vless`; credentials, encryption, flow, transport, and TLS/REALITY all matter. |
+| VMess | `vmess://` followed by Base64 JSON | Decode the inner JSON separately from the subscription's outer Base64. Fields include `add`, `port`, `id`, `scy`, `net`, `tls`, and `ps`. |
+| Trojan | `trojan://<password>@host:port?...#remark` | Xray `trojan`, including security and transport settings. |
+| Shadowsocks | `ss://...` | SIP002 uses Base64url of `method:password` for userinfo. Shadowsocks 2022 instead uses percent-encoded method/password userinfo. |
+| Hysteria | `hysteria2://...`, or `hysteria://...` for version 1 | Xray's outbound protocol name is `hysteria`; URI version and associated transport settings need translation. |
+| WireGuard | `wireguard://<private-key>@host:port?...#remark` | Includes public key, client addresses, and optional preshared key, MTU, DNS, and keepalive. |
+| AmneziaWG | `vpn://` plus unpadded Base64url configuration text | Panel's JSON exporter explicitly skips it; Xray's inspected outbound loader has no AmneziaWG protocol. |
+| TUIC | `tuic://<uuid>:<password>@host:port?...#remark` | Panel's JSON exporter explicitly skips it; Xray's inspected outbound loader has no TUIC protocol. |
+| MTProto | `tg://proxy?server=...&port=...&secret=...` | A Telegram proxy link, not an all-traffic Xray outbound. |
+| Information entry | `socks://127.0.0.1:1080#...` | Optional synthetic usage/expiry entry. It can appear first or replace every real entry. |
+
+The optional information entry is especially relevant to "select first". It can be a syntactically valid proxy URI while representing only usage, expiry, or quota information. JSON has an equivalent dummy SOCKS configuration. Arbitrary remarks make detection by display text unreliable. See [information-node generation][info] and [JSON generation][json].
+
+Current source has additional differences between the formats. The inbound query admits MASQUE, and the JSON generator supports it, but raw `GetLink` has no MASQUE case. MTProto also has no explicit JSON generation case; the generic code can still append the default direct/block outbounds. Therefore validate that a JSON choice has an actual usable proxy outbound; do not infer equivalence between raw and JSON counts. These are source observations requiring fixtures against a deployed release before making a compatibility guarantee. See [inbound query][query], [dispatch][dispatch], [JSON switch][json-switch], and [Xray loader][xray-loader].
+
+A subscriber can also receive administrator-added external links or links fetched by the panel from an external subscription. The raw exporter appends these after local exports. The JSON exporter parses supported external links into proxy outbounds and skips those it cannot parse. The protocol list is therefore not a closed raw-input schema. [External configuration handling][external]
+
+Recommended decoding policy: strip a UTF-8 BOM if present, accept LF or CRLF and blank lines, and try a recognizable plain link list before one outer Base64 decode. Accept padded/unpadded standard Base64 and Base64url as a compatibility extension, with bounded decoded size. The inspected panel itself emits standard Base64 for the outer wrapper. Parse VMess inner Base64 and protocol-specific userinfo only at their own layer. Percent-decode individual URI components once, preserve IPv6 brackets and credential punctuation, and retain unknown fields for diagnostics. Never repeatedly Base64-decode arbitrary content until something parses.
+
+## Ordering and identity on refresh
+
+Verified ordering is more complex than database creation order. Local inbounds start in ascending `sub_sort_index`, then `id`. Hosts-first mode reorders endpoint variants through the Hosts settings. One inbound/client can expand into multiple addresses. External entries follow their own client/sort/index order and append after local entries. JSON also interleaves configured balancer profiles. Optional information entries can precede everything. Preserve the final received order rather than rebuilding a presumed order. See [query][query], [host ordering][host-order], [raw assembly][service], [external entries][external], and [JSON ordering][json].
+
+The exported URI and JSON object expose connection credentials and settings, not the database inbound/host/client IDs used for sorting. A subscriber ID identifies the subscription, not one configuration. Remarks are templates and can change with usage or expiry. REALITY generation randomly chooses an SNI and short ID from configured lists for each fetch. Host address expansion can also introduce changes. Consequently, array index, remark, raw URI hash, full JSON hash, and credential UUID alone are each insufficient as a universal stable node identity. This is an inference from the [link generators][service], [REALITY generation][reality], [remark expansion][remarks], and [host expansion][host-sub].
+
+Recommended matching policy, with its limits:
+
+1. Assign each imported occurrence a local opaque ID and keep a separate content digest. Scope matches to the local subscription record, which survives URL edits.
+2. Match an unchanged normalized connection definition first. Exclude display remarks, usage metadata, and serialization order from that comparison. Normalize equivalent URI/JSON field shapes deliberately.
+3. For remaining entries, allow a weaker match only when a candidate is unique on both sides. A candidate key can include protocol, endpoint host/port, authentication identity, transport kind/path/service, security kind, and server public key. Changes in randomized REALITY SNI/short ID need special treatment; ignore them only for identity matching, never for the connection settings actually applied.
+4. If matching is ambiguous, credentials or endpoint rotate, or a selected choice disappears, expose a selection-needs-attention state. Do not guess based on a name or ordinal. Keep old records long enough to explain the change.
+
+This policy cannot guarantee identity across arbitrary provider edits. Duplicate endpoints and credentials can represent distinct choices; an operator can rotate every observable field. Supporting perfect continuity would require an upstream stable exported identifier or an explicit user mapping. Conservative failure is a product choice, not a 3x-ui guarantee.
+
+Initial selection and refresh are separate operations. Apply the agreed first-configuration default only on initial successful import. Refresh must preserve the local selection when it can match it and must never transition proxy state from off to on. Whether "first configuration" means the first received entry or the first supported non-information entry remains an explicit product decision.
+
+## Redirects, failures, and metadata
+
+The subscription handlers return 404 when no subscription rows match, 500 for internal errors, and 422 for a legacy Clash profile with no compatible proxy. Hardware-ID enforcement can also return 404, with `X-Hwid-*` response headers. It reads `X-HWID`, User-Agent, and device headers. A GET can record a subscription fetch and may consume/register a hardware-ID slot under the panel's policy. HEAD is routed through the same handlers, so do not assume it is a side-effect-free preflight. [HTTP handlers][controller]
+
+Successful output does not prove a usable proxy exists. Disabled/expired choices, excluded inbounds, closed host bindings, unsupported conversions, synthetic information entries, or inactive external subscriptions can leave empty or unusable results. JSON can return HTTP 200 with an empty body and subscription metadata for some inactive/closed cases. [Raw assembly][service] and [JSON empty handling][json]
+
+The inspected handlers do not define a redirect-based discovery protocol. Redirects may come from the router, reverse proxy, HTTP-to-HTTPS deployment, or a moved URL. HTTP semantics define `Location`, relative resolution, redirect codes, and 304 handling; no 3x-ui-specific redirect guarantee follows. [RFC 9110, redirection][http-redirect]
+
+Recommended transport policy: accept HTTP and HTTPS URLs, verify TLS, bound time, body size, decoded size and redirect count, and detect loops. Keep the original configured URL separately from the final fetched URL. Follow redirects under an explicit policy, reject HTTPS-to-HTTP downgrade by default, and do not forward authorization or device-identifying headers across origins without an established policy. Treat subscription URLs and response bodies as secrets because they carry reusable credentials. Report status/host and redacted diagnostics, not the full URL token or body.
+
+Recommended refresh semantics: fetch, decode, validate, and reconcile a candidate snapshot before an atomic replacement. A timeout, TLS error, non-success status, HTML response, malformed body, unsupported format, or all-unsupported result preserves the last good snapshot and selection. Represent a verified empty result separately from failure; decide whether it clears entries before implementation. A 304 is usable only if Roentgen has a corresponding cached representation and validator. The panel sets no-cache/no-store headers, so do not assume validators or delta updates. [Cache headers][cache]
+
+Metadata belongs beside the list, not in identity or activation logic. `Subscription-Userinfo` contains `upload`, `download`, `total`, and `expire`; the panel converts expiry milliseconds to Unix seconds and can adjust month-boundary presentation. `Profile-Update-Interval` suggests refresh hours. `Profile-Title` and `Announce` use a `base64:` prefix. `Support-Url` and `Profile-Web-Page-Url` are optional. Routing and client-specific policy headers may also appear. Treat metadata as optional display data and refresh advice. Do not let `Routing`, `Routing-Enable`, or `Hide-Settings` change Roentgen's local policy. [Metadata headers][headers] and [userinfo serialization][userinfo]
+
+## Xray fields and the host-owned runtime
+
+A complete subscription JSON object can contain local SOCKS/HTTP listeners, DNS, logging, policy, routing, stats, extra outbounds, mux, and balancer/observatory configuration. The shipped template listens on loopback ports 10808 and 10809; it does not establish Roentgen's host TUN policy. Provider settings can replace DNS and routing. [Default template][template] and [JSON template assembly][json]
+
+| Connection data to retain | Required interpretation |
+| --- | --- |
+| `outbounds[].protocol`, `settings` | Protocol-specific endpoint and authentication. Current VLESS/VMess support flat `address`, `port`, `id` and historical `vnext[].users[]` forms. Trojan/Shadowsocks can use server collections. Preserve cipher, encryption, flow, passwords and keys. |
+| `streamSettings.network` and transport settings | RAW/TCP, WebSocket path and Host, gRPC service/authority, HTTPUpgrade, XHTTP mode/extra/xmux, KCP and other supported transport options are connection behavior. |
+| `streamSettings.security`, `tlsSettings`, `realitySettings` | SNI, fingerprint, ALPN, certificate verification/pins, REALITY public key, short ID, spiderX and post-quantum verification settings must survive normalization. |
+| `mux`, `finalmask` and connection dependencies | These can be necessary. Vision flow imposes mux restrictions; XHTTP can carry its own multiplexing. Unknown extensions need a declared support result rather than silent deletion. |
+| `tag`, routing references and multiple proxy outbounds | A generated single-node profile commonly uses `proxy`, `direct`, and `block`. Balancer profiles can contain several proxy outbounds and dependencies. Tags are runtime references, not subscription node IDs. |
+
+Sources: [panel outbound generation][json-switch], [Xray VLESS loader][vless], [VMess loader][vmess], [outbound documentation][outbound-docs], and [transport documentation][transport-docs]. Current source confirms `rawSettings` and the `tcpSettings` alias. [Transport loader][transport-source]
+
+Recommendation: Roentgen owns the runtime envelope, including TUN, DNS strategy, host routing, loop prevention, management access, service lifecycle and off-state cleanup. Import supported connection definitions into that envelope. Reject complex JSON profiles that cannot be normalized safely rather than silently flattening a balancer or applying provider listeners/routing. Validate the generated candidate with the pinned Xray version before activation. Version compatibility needs an explicit supported matrix.
+
+Current Xray has a native `tun` inbound on Linux. Its configuration includes interface name, MTU, gateway prefixes, automatic system routes and outbound interface selection. The current Linux source also has an opt-in DNS takeover path using `resolvectl` and `autoSystemDnsToGateway`; the `dns` array itself is documented as Windows-only. Linux route installation and cleanup exist in source. Native TUN support therefore does not establish a universal DNS configuration or prove SSH-safe activation. [TUN documentation][tun-docs], [TUN configuration loader][tun-source], and [Linux implementation][tun-linux]
+
+Recommendations for the stated host scope: proxy on must apply a tested host-owned capture policy for IPv4, IPv6 and DNS, and preserve the SSH management connection and core egress. Proxy off must remove Roentgen's capture state. Installing or starting the persistent control service must not imply proxy on after an import. Whether the user's explicit on state survives reboot, and how to recover after a failed route change over SSH, need separate lifecycle decisions. No TUN or service implementation is proposed by this report.
+
+## Open decisions for the parent
+
+1. Format contract. Raw plain/Base64 first is recommended. Decide whether the first release also accepts full Xray JSON. YAML conversion is a separate capability; an Xray binary cannot consume Mihomo YAML directly.
+2. Supported connection matrix. Decide protocols, transports, security extensions and minimum Xray version. Recommend an explicit unsupported status for every entry outside it, including TUIC, AmneziaWG and Telegram MTProto when Xray is the only engine.
+3. First-entry semantics. Confirm how the initial-first requirement handles information nodes and unsupported entries. Recommendation: display received order, initially select the first usable connection, and explain skipped entries. This refines the supplied requirement and needs a product decision.
+4. Refresh continuity. Approve conservative matching, what happens to a running connection when its definition changes, and behavior when selection disappears. Recommendation: do not select a replacement automatically; never enable through refresh. Keeping an already active old snapshot versus turning off is unresolved.
+5. Empty and partial updates. Decide whether a valid empty response clears entries and whether a mixed malformed/valid list partially imports. Recommendation: preserve the last good snapshot on malformed updates and show unsupported entries separately from malformed entries.
+6. Redirect and HWID policy. Decide cross-origin redirects, plain HTTP acceptance, and whether Roentgen supplies a persistent host-scoped hardware ID when required. HWID enrollment has server-side effects and is not a generic mandatory 3x-ui requirement.
+7. Runtime ownership. Decide whether JSON balancers and provider mux/obfuscation dependencies are supported initially. Custom provider routing remains deferred under the supplied scope.
+8. Service semantics. Decide reboot restoration of explicit on/off state, DNS integration, SSH protection and rollback. These are required host-control decisions that a subscription payload does not resolve.
+
+Before implementation, capture sanitized fixtures for plain and outer-Base64 lists, nested VMess, JSON object/array, information entries, randomized REALITY refreshes, reordered/duplicate entries, external unsupported links, valid empty output, HTML/status JSON, HWID denial and redirect/error responses. Validate supported generated configurations with the chosen Xray release. This is proposed validation work, not tests run during this research.
+
+[docs]: https://docs.sanaei.dev/docs/config/subscription/
+[panel-release]: https://github.com/MHSanaei/3x-ui/releases/tag/v3.11.0
+[xray-release]: https://github.com/XTLS/Xray-core/releases/tag/v26.10.29
+[controller]: https://github.com/MHSanaei/3x-ui/blob/c621ba9014a6aa21fd7c9ecf45655e44e24ebf13/internal/sub/controller.go
+[service]: https://github.com/MHSanaei/3x-ui/blob/c621ba9014a6aa21fd7c9ecf45655e44e24ebf13/internal/sub/service.go
+[json]: https://github.com/MHSanaei/3x-ui/blob/c621ba9014a6aa21fd7c9ecf45655e44e24ebf13/internal/sub/json_service.go
+[urls]: https://github.com/MHSanaei/3x-ui/blob/c621ba9014a6aa21fd7c9ecf45655e44e24ebf13/internal/sub/service.go#L3323-L3384
+[settings]: https://github.com/MHSanaei/3x-ui/blob/c621ba9014a6aa21fd7c9ecf45655e44e24ebf13/internal/web/service/setting.go
+[dispatch]: https://github.com/MHSanaei/3x-ui/blob/c621ba9014a6aa21fd7c9ecf45655e44e24ebf13/internal/sub/service.go#L919-L944
+[info]: https://github.com/MHSanaei/3x-ui/blob/c621ba9014a6aa21fd7c9ecf45655e44e24ebf13/internal/sub/service.go#L429-L608
+[query]: https://github.com/MHSanaei/3x-ui/blob/c621ba9014a6aa21fd7c9ecf45655e44e24ebf13/internal/sub/service.go#L750-L780
+[json-switch]: https://github.com/MHSanaei/3x-ui/blob/c621ba9014a6aa21fd7c9ecf45655e44e24ebf13/internal/sub/json_service.go#L627-L1025
+[external]: https://github.com/MHSanaei/3x-ui/blob/c621ba9014a6aa21fd7c9ecf45655e44e24ebf13/internal/sub/external_config.go
+[host-order]: https://github.com/MHSanaei/3x-ui/blob/c621ba9014a6aa21fd7c9ecf45655e44e24ebf13/internal/sub/host_order.go
+[reality]: https://github.com/MHSanaei/3x-ui/blob/c621ba9014a6aa21fd7c9ecf45655e44e24ebf13/internal/sub/json_service.go#L886-L912
+[remarks]: https://github.com/MHSanaei/3x-ui/blob/c621ba9014a6aa21fd7c9ecf45655e44e24ebf13/internal/sub/remark_vars.go
+[host-sub]: https://github.com/MHSanaei/3x-ui/blob/c621ba9014a6aa21fd7c9ecf45655e44e24ebf13/internal/sub/host_sub.go
+[http-redirect]: https://www.rfc-editor.org/rfc/rfc9110.html#section-15.4
+[cache]: https://github.com/MHSanaei/3x-ui/blob/c621ba9014a6aa21fd7c9ecf45655e44e24ebf13/internal/sub/controller.go#L764-L768
+[headers]: https://github.com/MHSanaei/3x-ui/blob/c621ba9014a6aa21fd7c9ecf45655e44e24ebf13/internal/sub/controller.go#L945-L1015
+[userinfo]: https://github.com/MHSanaei/3x-ui/blob/c621ba9014a6aa21fd7c9ecf45655e44e24ebf13/internal/sub/calendar_expire.go
+[template]: https://github.com/MHSanaei/3x-ui/blob/c621ba9014a6aa21fd7c9ecf45655e44e24ebf13/internal/sub/default.json
+[xray-loader]: https://github.com/XTLS/Xray-core/blob/0afec66085a85191e9dd2a4a6646c7c9a94688d1/infra/conf/xray.go
+[vless]: https://github.com/XTLS/Xray-core/blob/0afec66085a85191e9dd2a4a6646c7c9a94688d1/infra/conf/vless.go
+[vmess]: https://github.com/XTLS/Xray-core/blob/0afec66085a85191e9dd2a4a6646c7c9a94688d1/infra/conf/vmess.go
+[outbound-docs]: https://xtls.github.io/en/config/outbound.html
+[transport-docs]: https://xtls.github.io/en/config/transport.html
+[transport-source]: https://github.com/XTLS/Xray-core/blob/0afec66085a85191e9dd2a4a6646c7c9a94688d1/infra/conf/transport_internet.go
+[tun-docs]: https://xtls.github.io/en/config/inbounds/tun.html
+[tun-source]: https://github.com/XTLS/Xray-core/blob/0afec66085a85191e9dd2a4a6646c7c9a94688d1/infra/conf/tun.go
+[tun-linux]: https://github.com/XTLS/Xray-core/blob/0afec66085a85191e9dd2a4a6646c7c9a94688d1/proxy/tun/tun_linux.go