Skip to content

sgns::neoswarm::core::InferenceEngine

Abstract interface for all inference backends.

#include <inference_engine.hpp>

Inherited by sgns::neoswarm::core::MNNInferenceEngine

Public Functions

Name
virtual ~InferenceEngine() =default
virtual outcome::result< void > LoadModel(const std::string & model_path) =0
Load a model from disk (MNN .mnn file or similar).
virtual outcome::result< InferenceResponse > Infer(const Task & task) =0
Synchronous inference — returns the full generated output.
virtual outcome::result< void > StreamInfer(const Task & task, std::function< void(const std::string &token)> callback) =0
Streaming inference — calls callback for each generated token.
virtual bool IsLoaded() const =0
virtual std::string BackendName() const =0

Public Functions Documentation

function ~InferenceEngine

virtual ~InferenceEngine() =default

function LoadModel

virtual outcome::result< void > LoadModel(
    const std::string & model_path
) =0

Load a model from disk (MNN .mnn file or similar).

Parameters:

  • model_path Path to the model file.

Return: outcome::success or ModelLoadFailed.

Reimplemented by: sgns::neoswarm::core::MNNInferenceEngine::LoadModel

function Infer

virtual outcome::result< InferenceResponse > Infer(
    const Task & task
) =0

Synchronous inference — returns the full generated output.

Parameters:

  • task Inference task with prompt and generation parameters.

Return: InferenceResponse or InferenceFailed.

Reimplemented by: sgns::neoswarm::core::MNNInferenceEngine::Infer

function StreamInfer

virtual outcome::result< void > StreamInfer(
    const Task & task,
    std::function< void(const std::string &token)> callback
) =0

Streaming inference — calls callback for each generated token.

Parameters:

  • task Inference task.
  • callback Called with each token string as it is generated.

Return: outcome::success or InferenceFailed.

Reimplemented by: sgns::neoswarm::core::MNNInferenceEngine::StreamInfer

function IsLoaded

virtual bool IsLoaded() const =0

Return: True if a model has been loaded.

Reimplemented by: sgns::neoswarm::core::MNNInferenceEngine::IsLoaded

function BackendName

virtual std::string BackendName() const =0

Return: Human-readable backend name (e.g. "MNN/Vulkan", "MNN/CPU").

Reimplemented by: sgns::neoswarm::core::MNNInferenceEngine::BackendName


Updated on 2026-10-09 at 03:59:42 +0000