sgns::neoswarm::core::InferenceEngine¶
Abstract interface for all inference backends.
#include <inference_engine.hpp>
Inherited by sgns::neoswarm::core::MNNInferenceEngine
Public Functions¶
| Name | |
|---|---|
| virtual | ~InferenceEngine() =default |
| virtual outcome::result< void > | LoadModel(const std::string & model_path) =0 Load a model from disk (MNN .mnn file or similar). |
| virtual outcome::result< InferenceResponse > | Infer(const Task & task) =0 Synchronous inference — returns the full generated output. |
| virtual outcome::result< void > | StreamInfer(const Task & task, std::function< void(const std::string &token)> callback) =0 Streaming inference — calls callback for each generated token. |
| virtual bool | IsLoaded() const =0 |
| virtual std::string | BackendName() const =0 |
Public Functions Documentation¶
function ~InferenceEngine¶
function LoadModel¶
Load a model from disk (MNN .mnn file or similar).
Parameters:
- model_path Path to the model file.
Return: outcome::success or ModelLoadFailed.
Reimplemented by: sgns::neoswarm::core::MNNInferenceEngine::LoadModel
function Infer¶
Synchronous inference — returns the full generated output.
Parameters:
- task Inference task with prompt and generation parameters.
Return: InferenceResponse or InferenceFailed.
Reimplemented by: sgns::neoswarm::core::MNNInferenceEngine::Infer
function StreamInfer¶
virtual outcome::result< void > StreamInfer(
const Task & task,
std::function< void(const std::string &token)> callback
) =0
Streaming inference — calls callback for each generated token.
Parameters:
- task Inference task.
- callback Called with each token string as it is generated.
Return: outcome::success or InferenceFailed.
Reimplemented by: sgns::neoswarm::core::MNNInferenceEngine::StreamInfer
function IsLoaded¶
Return: True if a model has been loaded.
Reimplemented by: sgns::neoswarm::core::MNNInferenceEngine::IsLoaded
function BackendName¶
Return: Human-readable backend name (e.g. "MNN/Vulkan", "MNN/CPU").
Reimplemented by: sgns::neoswarm::core::MNNInferenceEngine::BackendName
Updated on 2026-10-09 at 03:59:42 +0000