Before the battery does — watching, and safely steering, a fleet of DC power systems over SNMP

A telecom site is, at bottom, a promise: that the radios keep breathing and the routers keep their lights on no matter what the grid outside is doing. Keeping that promise is the quiet, thankless job of a rectifier shelf — an Eltek plant that takes whatever the mains throws at it, renders it down to a steady −48 V, and trickle-charges a wall of batteries whose entire reason for existing is to be ready for the moment the power dies. Most days it just sits there, unglamorous and uncomplaining. But when one of these plants begins to drift, you want to hear about it long before the battery has to find out for you — at 3 a.m., the hard way, somewhere in the middle of an outage.

This is the system I built to stand watch over a whole fleet of them. And then, because watching is never quite enough, to reach in and steer them.

From a frozen report to a living system

The thing it replaced was honest but inert: a lone script that stamped out a couple of static HTML pages. It told you yesterday's story, beautifully frozen, and asked you to be content with that. I wasn't. I wanted the present tense — live state, a memory of everything that had changed, and the one capability that turns a wall of dials into an actual instrument: the power to act on a controller without ever logging into it by hand.

And the fleet refuses to be tidy. Two controller families share the network — the old, dependable Smartpack-1 and the newer, cleverer Smartpack-R / eNexus — each with its own MIB, its own dialect, its own small treacheries. So the heart of the system had to stay studiously vendor-agnostic, and it had to let an operator teach it a brand-new controller without a single line of fresh code.

Discovery → classify → poll

Three quiet modules do the breathing:

• Discovery leans on Zabbix — the network is already catalogued there — lifts the host list, and remembers it.
• Classify interrogates each host in turn. It puts the same identifying question — an identity OID — to every known controller profile, and the first profile to get a straight answer claims the host as its own. Whatever stays mute is exiled to an Unknown bin, to be puzzled over later.
• Poll walks every classified system every fifteen minutes, reading exactly the OIDs and table columns its profile prescribes, and paints the dashboard anew.

The elegant part — the part I'm quietly proud of — is that classify and poll know nothing, in themselves, about Eltek at all. They are profile-driven. A profile is simply a dictionary of OIDs living in the database: the identity question, the scalars to read (input voltage, temperature, the float and boost setpoints…), the tables to walk (one row per rectifier slot). Teaching the system a new vendor — a Huawei plant, say — isn't an engineering project; it's an afternoon in the UI: defining a profile, prodding a sample device with a live Probe button to watch which answers come back, and promoting the first match to a real, monitored system. No deploy. No code review. Just knowledge, typed in.

An inventory that refuses to cry wolf

Every poll brings home two harvests: the live metrics, and a component inventory — every rectifier module and control unit, named by serial number, slot, hardware and firmware. Lay two consecutive polls side by side and the differences surface on their own: installed, removed, changed — each one wired to an email that lands on the desk of whoever owns that site.

But the hard thing about diffing an inventory was never the diff. It is the discipline of not lying. SNMP over a tired network drops packets the way a bad phone line swallows syllables; one lost poll must never be allowed to haul someone out of bed to mourn a rectifier that never went anywhere. Two instincts keep the system honest:

• A sanity check: if the headcount insists "eight rectifiers" but the rectifier table comes back empty-handed, that isn't a massacre — it's a half-finished SNMP conversation. The system shrugs, marks that whole category untrusted for this pass, and refuses to raise a single false removed.
• A grace period: a component must go missing across three consecutive polls — three quarters of an hour — before the system will believe it has truly gone. A momentary blink stays a blink, and stays silent.

And the very first time a freshly-onboarded site checks in, the system bites its tongue — bootstrap suppression — so that wheeling a hundred new sites onto the floor at once doesn't bury the mailing list under a hundred breathless "new component!" telegrams.

From reading to writing — and learning to fear it

Monitoring is the gentle half of the story. The half with teeth is letting an operator change a controller — float and boost voltage, charge-current ceiling, alarm thresholds — straight from a browser, over SNMP, without ever opening a shell on a live battery plant.

Writing to power systems is exactly the place to be a little afraid, and to fold that fear into the code. Every single set walks the same unhurried, deliberate path:

edit a parameter
   │
   ├─ validate against the MIB     (type · enum · range → refuse the impossible)
   ├─ battery-critical? ───────────→ demand an explicit, deliberate confirm
   ├─ SET
   ├─ read the value BACK          (a mismatch is a FAILURE, never a success)
   └─ write it to the audit log    (who · what · old → new) · revert in one click

The read-back is the keystone — the step everything else leans on. A polite "OK" from an SNMP SET means almost nothing; the controller can nod and quietly ignore you. So the system turns around and asks the device what it now believes, and only then dares to call the change done. Anything out of range is turned away at the door, before it can ever touch the wire. The genuinely dangerous values — the voltages that stand between a battery charging gently and a battery boiling — will not budge without a human deliberately saying yes, I mean it. And every move can be walked back to exactly where it stood, with one click and no ceremony.

The catalogue of writable OIDs isn't lovingly hand-typed either — it is distilled straight from the vendor MIBs, because the MIB already knows each OID's type, its range, its access. The guardrails are drawn from the source of truth, not from my memory of it.

The battery test

One control deserves to stand alone, because it is the one that can actually hurt. A battery test deliberately drains the string to take the true measure of its remaining capacity — invaluable, and also the very last thing you want to set off on a whim while the mains is already trembling. So it is a single button that stops to warn you first, then keeps a close, anxious watch — re-checking that system live, once a minute, for as long as the test runs. When the controller reaches its floor — its minimum end-voltage — or throws an alarm, it ends the test of its own accord, and the button quietly resets itself. The interface never has to guess what is happening; it simply asks the controller, and the controller tells it the truth.

Reaching a device without knowing its password

The last piece closes the gap between the tracker and the controller's own little web interface. Every device has one — but scattering device passwords across an operations team scales badly and audits worse. So instead, an authenticated user clicks Web UI and simply finds themselves inside the controller's native interface, reverse-proxied behind the very login they are already holding. The proxy stands guard with the tracker's own session and slips the stored credential in on the user's behalf, so the operator never once lays eyes on it. And the per-device plumbing behind all this is generated from the inventory itself, never hand-tended: the fleet shifts, the configuration rewrites itself, tests itself, reloads — or quietly rolls back the moment anything looks wrong. The secrets sleep encrypted at rest; the plaintext never touches the database.

What it is really made of

Underneath, it is unremarkable on purpose: FastAPI and APScheduler, PostgreSQL, SNMP, a server-rendered Jinja interface, the whole thing folded into Docker. The framework was never the story. The story is a single conviction, threaded through every corner of it: let the device be the source of truth, and always, always verify against it. Classify a host by what it actually answers. Pronounce a component dead only when it is genuinely, three-times-over gone. Call a parameter changed only once the controller itself has read it back to you. Monitoring will happily tell you how the world looks; the real craft — the part worth losing sleep over — is reaching into that world and changing it without ever making things worse.