STEVEN LIN
01 / 23 OPENING

RADIXARK FINAL INTERVIEW · STEVEN LIN

Is anything wrong?

Building the answer
from signals to action at scale.

PAST EXPERIENCE 35m FUTURE + WHY RADIXARK 10m Q&A 15m
BUILDING THE ANSWER

Building the Answer

Answering the question requires all three capabilities, connected end to end.

ONE QUESTIONWhat happened · where · impact · evidence · change · owner · safest next step

Infrastructure Overview: Scale and Hierarchy

200K+ switchescouple million serversmany regions · many data centers
ILLUSTRATIVE · NOT TO SCALE
Is this server healthy?
FRAME 01One server is already a system.

CPU, memory, GPUs, interconnect, NICs, storage, power, cooling, and independent health signals.

MY ROLE · Build the software platform that observes, models, and troubleshoots this global infrastructure.

Stage 1 Example — The Syslog Problem

A distributed, bursty stream with no shared language.

SYNTHETIC RAW INPUTDifferent vendors, firmware versions, and operational events
CISCO · LINK%LINK-3-UPDOWN: Interface Ethernet1/23 changed state to down RUIJIE · LINKIF_DOWN: GigabitEthernet 0/24 link state changed to DOWN ARISTA · LINKLINEPROTO-5-UPDOWN: Ethernet12 line protocol is down HUAWEI · LINKIFNET/4/LINK_STATE: 10GE1/0/24 changed to DOWN H3C · LINKIFNET/3/PHY_UPDOWN: Ten-GigabitEthernet1/0/8 is DOWN FIRMWARE · NEWPhysical port et-0/1/8 changed oper-state: inactive ROUTING · BGPBGP-5-ADJCHANGE: peer 10.8.1.4 changed Established → Idle POWER · PSUPSU/3/FAIL: power unit 2 output voltage below operating range COOLING · FANFANTRAY-2-FAULT: fan 3 speed stalled below safe threshold PROCESS · EXITPROC_MGR: bgpd terminated unexpectedly with exit code 137
First five vendors · same operational probleminterface.link_down
PROBLEM STATEMENT · STORAGE-BASED SCALE ESTIMATE

How do we reliably collect distributed logs from a global fleet, recognize useful meaning across changing syntax, and make the result reusable?

200K+
switches producing distributed operational logs
≈1 TB/day
stored log payload across the fleet
≈3–5B/day
events, assuming roughly 200–300 bytes each
≈40–60K/s
fleet-wide average event rate
10×+
≈400–600K+/s implied burst range; syntax also keeps changing
RAW · DISTRIBUTED · BURSTY · HETEROGENEOUSMEANINGFUL SIGNALS THAT OTHER SYSTEMS CAN USE

A Stable Collection Boundary

BEFORESSH into each management servergrep remembered keywordshacky time-window searchNo global view · easy to miss unknown patterns
01STABLE COLLECTION PLANEShared, difficult-to-change hosts do the smallest reliable amount of work.
DEVICE FLEET200K+ switches · global sources
SINGLE COPY
ANYCAST IPnearest regional ingress
FAN OUT
REGIONAL MANAGEMENT SERVERS
LogstashRotationTailer
DURABLE
RAW KAFKAfirst durable shared copy
02EVOLVABLE PROCESSING PLANEFast-changing intelligence evolves without destabilizing collection.
PROCESSING CONSUMER UNITrecognize · extract · normalize · enrichExpanded on the next page →
STRUCTURED KAFKAshared event contract
ElasticsearchClickHouse
SEARCH + ANALYTICSindexed events · time-series analytics
QUERY SERVICESAPIs · search · downstream systems
Keep the single-copy path simple. Add intelligence only after the durable boundary.

A Vendor-Agnostic Recognition Layer

Two mappings turn raw text into a shared operational contract.

RAW TEXT01 · PROBLEM CLASS + REQUIRED FIELDS02 · CANONICAL ENTITY IDSTRUCTURED EVENT
INPUT · FROM THE RAW STREAMCisco%LINK-3-UPDOWN · Ethernet1/23 · downHuaweiIFNET/4/LINK_STATE · 10GE1/0/24 · DOWNFirmwareoper-state · et-0/1/8 · inactiveRoutingBGP-5-ADJCHANGE · peer 10.8.1.4 · IdleCoolingFANTRAY-2-FAULT · fan 3 · stalledProcessbgpd · terminated · exit 137
01
Recognize the operational classProduction-style regex rules span vendors and extract a required contract.
TAGILLUSTRATIVE REGEXMUST EXTRACT
interface.link_down/(?:LINK-\d+-UPDOWN|IF_DOWN|LINK_STATE|PHY_UPDOWN).*?(?<if_name>(?:Ethernet|Gi|10GE|Ten-GigabitEthernet|et-)\S+).*?(?<status>down|inactive)/iif_name · status
routing.bgp_state/BGP-\d+-ADJCHANGE.*?peer\s+(?<peer_ip>\d{1,3}(?:\.\d{1,3}){3}).*?(?<status>Idle|Active|Established)/ipeer_ip · status
power.psu_failed/(?:PSU|PWR).*?(?<psu_id>\d+).*?(?<status>fail|fault|low|absent)/ipsu_id · status
cooling.fan_failed/(?:FAN|FANTRAY).*?(?<fan_id>\d+).*?(?<status>fault|stalled|absent)/ifan_id · status
process.exited/(?:PROC_MGR|PROCESS).*?(?<process>[a-z0-9_.-]+).*?(?:exit|terminated).*?(?<exit_code>\d+)/iprocess · exit_code
02
Normalize vendor-specific entity namesA small mapping service converts extracted aliases into one canonical ID format.
Ethernet1/23interface:sw-1023:1/2310GE1/0/24interface:sw-2048:1/0/24xe-0/0/12interface:sw-0771:0/0/12
AFTER · SHARED CONTRACT{
  "event_type": "interface.link_down",
  "device_id": "switch-1023",
  "entity_id": "interface:sw-1023:1/23",
  "raw_interface": "Ethernet1/23",
  "status": "down",
  "source_vendor": "vendor-a"
}
Normalize both the problem and the entity. Downstream systems should depend on neither vendor syntax nor vendor naming.

Build a Learning System, Not a One-Time Parser

USABLE RECOGNITION RATEvalidated class + required metadata extracted÷all incoming eventsInput distribution keeps changing; improvement must be continuous.
REDESIGN
AFTER · CONTINUOUS RECOGNITION SYSTEMMinimize unknowns before they become incidents.
PROACTIVE REVIEW LOOPCluster Review Service
CLUSTER QUEUE#27 · 18.4% of unknown…port [TOKEN] oper-state inactive…interface [TOKEN] carrier lost3–4 representative samples
ENGINEER REVIEW
BOOTSTRAP · JUPYTER~70–80%
REVIEW SERVICE95%+
CONTINUOUS LOOP99%+
END STATE · Unknown tail moves toward ~0.1% scale—small enough to alert, cluster, and absorb with cross-layer context.
Metrics create direction. Feedback creates momentum.

Beyond Syslog: The Sensing Layer

SENSE · COMPLETE
EVENT-DRIVENSYSLOG

Device and server operational events

POLLINGSNMP / TELEMETRY

CPU · memory · temperature · optics · traffic · errors

ACTIVE MEASUREMENTPINGMESH

Reachability · loss · latency distributions

Appendix A1
ACTIVE SIGNALPINGER

Additional reachability and latency observations

COLLECTORSKAFKA / DATA BUSSTRUCTURED SIGNALS

I worked on several collection pipelines; Syslog was the highlighted deep dive.

Next: How do we make all these signals meaningful together?ENTER INTEGRATE →

We Had Signals. We Still Could Not See the System.

OTHER MAJOR PROJECTS · INTEGRATEI built the alert platform from 0 → 1.

Better detection and aggregation reduced noise before TopSight connected the meaning.

INGESTDEDUPLICATEAGGREGATEROUTE
DYNAMIC BASELINEATTENTION-BASED DEVIATION
ALERT PLATFORM0 → 1 lifecycle and routingALERT AGGREGATIONGroup repeated symptomsDYNAMIC THRESHOLDLearn heterogeneous baselines
REMAINING PROBLEMWhat does this anomaly mean together?

Every signal described the world differently.

PINGMESHendpoint scopesINTERFACEdevice namesSYSLOGraw identifiersSERVER HEALTHasset IDsK8S PLACEMENTnode aliasesMAINTENANCEcalendar scopeOWNERteam taxonomyDEPLOYMENTservice versions
Which dashboard? · Which physical object? · Network, server, deployment, or maintenance?
Unusual does not mean understood.

Model the World Before Correlating the Data

Topology is the base; overlapping operational scopes add meaning.

SYNTHETIC · LEFT HALF OF THE SHARED TOPSIGHT GRAPH
Connectivity explains dependency. Membership explains shared impact and operational context.

TopSight: One Graph, Multiple Operational Views

SYNTHETIC · HOVER ANY LINK FOR ILLUSTRATIVE TRAFFIC
Service endpointServerToRLeaf / spine fabricDegraded entity or linkHover a line: highlight + capacity + traffic history.

Ask TopSight a Question

ILLUSTRATIVE INVESTIGATION · SAME GRAPH, QUESTION-SPECIFIC SCOPE
TopSight turned an unbounded search into a bounded, evidence-rich search space.

TopSight: From Infrastructure Data to Operational Action

One entity model for topology, live state, containment, and controlled operations.

INTEGRATE · COMPLETE
INFRASTRUCTURE DATA
Physical network + inventoryVPC + virtual networkServer + GPU dataMaintenance + operational scopesManagement + external ingress
Cross-team sources arrive with different names, lifecycles, and ownership.
NORMALIZED ENTITY MODELIdentity · alias · name · location

Contains · member of · connected to

TOPSIGHT GRAPH LAYERRedis KV + custom in-memory graph

On-demand lazy load · Python → Go · million-scale

ILLUSTRATIVE API SURFACESemantic capabilities—not literal endpoint names
get_neighbors(entity_id, depth=1)Relevant nodes and edges around one entityInput: switch / server / interface / service
INTERACTIVE TOPOLOGY UIExpand the selected neighborhood

Lay out irregular physical, virtual, management, and external-ingress relationships.

AUTHORIZED QUICK OPERATIONSSelected device → explicit operator authorization
SSHshow versionuptimeCPU / memory
CONTAINMENT Pod · Mini-Pod · maintenance · topology areaGPU TRAINING SCOPE Trail ID · job → GPU → server → networkPATH SEMANTICS Candidate physical paths—not exact packet flow
The frontend did not draw a clean tree. It rendered the irregular system operators actually had to reason about.

Visualization Is Not Automation

TopSight bounded the search, but humans still interpreted evidence and coordinated the response.

01 · ALERT-DRIVENVolume remained operationally impossible for humans
TENS OF THOUSANDSbase alerts + repeated signals / day
DEDUPLICATE · CORRELATE · PRIORITIZE
SEVERAL HUNDREDmeaningful highest-priority investigations / day
COMMON EXAMPLE · INTERFACE FLAPPING
Syslog UP ↔ DOWNfrequency / time windowsuspected NIC / module degradationreplacement workflow
02 · HUMAN-REPORTEDWeak symptoms arrived in inconsistent language
DOZENS → 100+human on-call escalations on busy days

Service SREs · application teams · support teams

“The network seems slower.”“This service is intermittently unreachable.”“Latency changed after the deployment.”
Representative wording · incomplete scope · low initial information
TWO ENTRY POINTS · THE SAME REPEATED MANUAL LOOPALERT OR REPORTestablish initial scopeINTERPRET SYMPTOMtranslate weak evidenceSELECT TOOLSchoose the next checkLOAD CONTEXTtopology · change · ownerCOMPARE EVIDENCEtest competing hypothesesCOORDINATE SAFE ACTIONroute · repair · validateSKILLED ENGINEER STILL IN THE LOOP
TopSight made the evidence available. Repeated investigation and coordination still needed automation.

RCA: A Finite-State Workflow Platform

A Python framework for explicit, reusable, operator-controlled investigations and recovery.

PYTHON + DECORATORS@state("drain_traffic")
def drain(context): …
Illustrative syntax · each state is one Python function
SHARED LIBRARIESTopology · telemetry · traffic control · notification · ticketing
CO-AUTHORED WORKFLOWSNetwork engineers + SREs encode their operational logic
TRIGGERAlertEngineer requestLLM request · laterinterface_flapping RCA + recoverytypical elapsed operation · 30–60 min
01Drain trafficisolate affected component
02Monitor remaining trafficprotect residual capacity
03Notify on-site teamreplacement request
04Replace NIC / modulehuman operation
05 · GATEConfirm completionwait for acknowledgement
06 · GATEHealth checktelemetry + reachability
07Undrain trafficcontrolled restoration
08Final validationclose or escalate
READYObservation and human wait states are accelerated in this demo.
OTHER WORKFLOW SCOPESPoint-to-pointClusterSwitchToR domainKubernetes serviceEXPLICIT · BOUNDED · REPEATABLE · AUDITABLEBuilt from mid–late 2023 · dozens of scenario-specific workflows
The trigger could vary. The workflow sequence, wait states, and safety gates remained deterministic.

Troubleshooting Agent: The Control Boundary Moved

From selecting complete workflows to composing atomic capabilities.

INPUTQuestion / Alert
CONTROL OWNERWORKFLOW ENGINEFixed sequence

Rasa or a bounded intent layer selects one complete RCA workflow.

Intent → Complete RCA Workflow → Fixed Sequence
WORKFLOW ENGINESHARED CONTROLTROUBLESHOOTING AGENTAuthorization remains deterministic
More model capability reduced hard-coded sequencing—and increased the need for better tools, context, and safety boundaries.

The Model Was Not the Bottleneck

The agent stopped where grounded data and executable coverage stopped.

MULTI-TENANT · OPENCLAW-BASED · STANDARD MCP
01 · HIDDEN KNOWLEDGEKnown to people, not exposed as an API

Owner, previous on-call discussion, escalation path, and recent change were often implicit or buried.

OWNER?WHAT CHANGED?WHO WAS ON CALL?HOW DO WE ESCALATE?
GROUNDED CONTEXTWhat can be trusted?
Topology + signalsDeployment + maintenanceOwner + on-call historyCMDB ≠ deploy DB ≠ chat
OPENCLAW-BASEDMULTI-USER AGENTShared Skills · Standard MCP
RETRIEVE → REASON → TEST → UPDATE
BOUNDED HANDLERSWhat can actually run?
Topology · metrics · logsReplay · compare · isolateTicket · maintenance · deploymentMISSING INTERFACE → SUGGESTION ONLY
TEAM AGENT AUSER MANUALLY CARRIES CONTEXTTEAM AGENT BTool and agent boundaries often followed organizational ownership.
OPERATIONAL AGENT VALUEReasoning × Grounded Context × Bounded Handler CoverageInvestigate broadly · high-risk authority remains outside the model
The boundary I reached was the boundary of the data and actions I could access—not the boundary of model capability.

Three Design Principles I Would Carry Forward

PAST EXPERIENCE · 35m
PRINCIPLE 01

Complexity should be paid once.

Vendor-specific complexity belongs below a stable contract. Abstraction scales; feedback compounds.

Vendor-specific logsShared operational event model
Different acceleratorsSGLang serving abstraction
GLOBAL SCHEDULERUNIFIED DEPLOYMENT CONTRACT
PROVIDER APROVIDER BCUSTOMER CLOUDBYOC
These are not network-only lessons. They are architectural principles for AI infrastructure.

Is anything wrong?

Now we can answer.

QUERY · 10:42 UTC · CHECKOUT SRE“Is checkout-service healthy right now?”Learned preference · availability and customer impact first
01 · SENSETraffic + Syslog

P99 +12 ms · no packet loss
2 ToR link transitions

02 · INTEGRATETopology + Change

Maintenance · Mini-Pod 07
No service deployment

03 · REASON & ACTService Impact

6 / 240 endpoints shifted
Monitor · no action now

ILLUSTRATIVE ANSWER · HIGH CONFIDENCE

Checkout Service is healthy overall. Scheduled Mini-Pod 07 maintenance briefly reduced capacity for 6 of 240 endpoints, but traffic shifted as designed. Availability remains 99.98%; no packet loss or overlapping service deployment is visible. Continue monitoring until 11:00 UTC—no remediation is needed now.

One role-aware answer, grounded in query time, scope, live evidence, topology, and change history.

AI Infrastructure Is a Different Scaling System

Market research changed how I see workload physics, stack value, and model serving.

≈2.5 MIN
WHAT THE WORKLOAD REQUIRESAI infrastructure does not scale like general-purpose cloud.

Accelerator density, memory bandwidth, interconnect, model state, and tail latency must be optimized as one system.

The model, runtime, scheduler, topology, and memory policy cannot be designed independently.
THREE SIGNALS FROM THE RESEARCH
WORKLOAD PHYSICSCPU elasticity → accelerator systems

Dense compute · topology · synchronized work · KV state

MARKET MOTIONCapacity providers move into software

NVIDIA → Run:ai · CoreWeave → W&B → OpenPipe

STRATEGIC CONTROL POINTServing determines usable intelligence

Context · precision · scheduling · cache · reliability · cost

The opportunity is not only more GPUs. It is the software that turns model weights into dependable, economical intelligence.

Better Models Raise the Infrastructure Bar

The system must serve, remember, and learn—then do it again.

≈2.5 MIN
BETTER MODELSMORE AGENT WORK · LONGER CONTEXT · MORE MODEL CALLSMORE SYSTEMS CO-DESIGN
01 · SERVING CO-DESIGNThe same weights can produce a very different service.
WHAT BECOMES DURABLE

Context policy, precision, scheduling, cache locality, isolation, observability, and recovery.

CLAIM BOUNDARY

Serving changes effective quality and economics; it does not change the benchmark identity of the weights.

MY FILTERAI INFRASTRUCTUREINFERENCE SOFTWAREEARLY STARTUPTRAINING CAPABILITYFIT · HW/SW OPTIMIZATION + DISTRIBUTED DATA + END-TO-END OWNERSHIP
I want to build where inference and training co-design a continuously improving system.

Why RadixArk

A real serving foundation, a path into post-training, and an early systems problem worth shaping.

≈3 MIN
PUBLIC FOUNDATIONS
INFERENCESGLang

Open, high-performance serving with real technical and community depth.

POST-TRAININGMiles

RL and post-training on the other side of the improvement loop.

COMPANY STAGEEarly · open · infrastructure-first

Foundational interfaces and operating systems are still shapeable.

WHAT CONNECTS THEMOBSERVABLE, RELIABLE
PRODUCTION SYSTEMS
WHAT I BRING
SYSTEM DESIGNStable abstraction boundaries

Complexity belongs below contracts that can survive scale and change.

OPERATIONSCross-layer context and control planes

Identity, topology, signals, changes, owners, diagnosis, and safe action.

EARLY-STAGE EXECUTION0→1 ownership with systems taste

My edge is choosing what should remain durable—not typing faster than an agent.

TECHNICAL THESIS I WANT TO TESTAs agent context grows, KV and model-native state may become a first-class data layer.
NOW · reuse + tiered placementNEXT · composition + intelligent servicesRESEARCH · state programming
A research direction—not a claim about RadixArk’s current roadmap.
RadixArk sits at the intersection of the future I want to build: open serving, post-training, and the systems that connect them.

Q&A

Happy to go deeper into the architecture, trade-offs, impact, or how these lessons map to AI serving.

Thank you.

Pingmesh Architecture and Aggregation

← BACK TO SENSING
CONTROLLERtarget / ping lists
HOST + VM AGENTSpoint-to-point probes
KAFKA / DATA BUSraw observations
STREAM AGGREGATIONwindows + percentiles
SCOPE HEALTHserver → region / service
AGGREGATION SCOPE
ABCDABCD
HIGH-FREQUENCYmany probes per pair / minuteAGGREGATEDlarge raw sample sets per health pointCADENCE~20 seconds per statistical health pointSynthetic / illustrative matrix