Protocol Buffer (protobuf)

core.proto

Ack

An acknowledgement from a replica to the leader that it has caught up to that point in the log.

Field Type Label Description
replica uint32 Ordinal of replica the request comes from
replay bool Whether the replica is requesting a replay starting from acked offset

DeleteRequest

Represents a delete request

Field Type Label Description
ids string repeated Ids of items to delete

FetchRequest

Represents a fetch request

Field Type Label Description
ids string repeated Ids of vectors to fetch
vectors NdArray repeated Result vectors corresponding to ids

IndexRequest

Represents an index request

Field Type Label Description
ids string repeated ids of items to be inserted
data NdArray vectors of items to be inserted

InfoRequest

Represents an info request

Field Type Label Description
index_size uint64 Size of a stateful function (Index)

LogEntry

Field Type Label Description
offset uint64 WAL offset of this entry
entry Request The request itself as it was received
ack Ack Or an acknowledgement from a follower of receiving this entry

NdArray

Represents the a numpy ndarray

Field Type Label Description
buffer bytes the actual array data, in bytes
shape uint32 repeated the shape (dimensions) of the array
dtype string the data type of the array
compressed bool whether lz4 compression is used on buffer

QueryRequest

Represents a query request

Field Type Label Description
top_k uint32 Number of results to return for each query
include_data bool Whether to include the vectors and raw data in response as well as ids
data NdArray The batch of vectors to query
matches ScoredResults repeated The corresponding matches returned for each query in the batch

Request

Represents an execution plane Request

Field Type Label Description
request_id uint64 unique id of the request
timeout uint32 timeout in second until this message is dropped
path string path in DAG (defined ahead of time) to follow
version string request schema version
routes Request.Route repeated a list of routes this message goes through
status Status status info, e.g. error
query QueryRequest A query request object
index IndexRequest An upsert items request object
delete DeleteRequest A delete items request object
info InfoRequest An object for a get info request
fetch FetchRequest Payload for a request to fetch vectors by id
namespace string Namespace to perform operation in (if data is divided by namespace, default "")
client_id uint32 Integer ID that represents the client connection. Assigned by gateway.
client_offset uint32 Offset of request within client connection. Assigned by gateway.
shard_num uint32 Ordinal of the last shard visited of a stateful function (or 0).

Request.Route

Represents the route paths of this message

Field Type Label Description
function string the name of the Service
function_id int32 the id of the Service
start_time google.protobuf.Timestamp receiving time
end_time google.protobuf.Timestamp sending (out) time

ScoredResults

Field Type Label Description
ids string repeated String ids of the item results of a query
scores float repeated Corresponding pairwise scores between the query to each item
data NdArray Corresponding data for all of these items, if requested

ServiceControlRequest

Request from the controller to individual service for monitoring, updating, and terminating running services.

Field Type Label Description
function string Name of the function.
function_id uint64 Unique id of the function (if any).
status Status Status of the function.
service string Name of the service the function belongs to.

Status

Field Type Label Description
code Status.StatusCode status code
description string error description of the very first exception
details Status.Details repeated the details of the error
msg_sent uint64 number of messages sent by the function
msg_recv uint64 number of messages received by the function
avg_time Status.AvgTimeEntry repeated average time spent by messages in function
size uint64 size of a function's dataset if stateful

Status.AvgTimeEntry

Field Type Label Description
key string
value uint32

Status.Details

Field Type Label Description
function string the name of that problematic service
function_id string the id of that problematic service
exception string the class name of the exception
traceback string the reason of the exception
time google.protobuf.Timestamp the timestamp when error occurs

Status.StatusCode

Name Number Description
SUCCESS 0 success
READY 1 ready to use
ERROR 2 error
ERROR_DUPLICATE 3 already a existing service

RPCClient

core gRPC service.

Method Name Request Type Response Type Description
Call Request stream Request stream Pass in a Request stream and completed Requests will be returned

Scalar Value Types

.proto Type Notes C++ Java Python Go C# PHP Ruby
double double double float float64 double float Float
float float float float float32 float float Float
int32 Uses variable-length encoding. Inefficient for encoding negative numbers – if your field is likely to have negative values, use sint32 instead. int32 int int int32 int integer Bignum or Fixnum (as required)
int64 Uses variable-length encoding. Inefficient for encoding negative numbers – if your field is likely to have negative values, use sint64 instead. int64 long int/long int64 long integer/string Bignum
uint32 Uses variable-length encoding. uint32 int int/long uint32 uint integer Bignum or Fixnum (as required)
uint64 Uses variable-length encoding. uint64 long int/long uint64 ulong integer/string Bignum or Fixnum (as required)
sint32 Uses variable-length encoding. Signed int value. These more efficiently encode negative numbers than regular int32s. int32 int int int32 int integer Bignum or Fixnum (as required)
sint64 Uses variable-length encoding. Signed int value. These more efficiently encode negative numbers than regular int64s. int64 long int/long int64 long integer/string Bignum
fixed32 Always four bytes. More efficient than uint32 if values are often greater than 2^28. uint32 int int uint32 uint integer Bignum or Fixnum (as required)
fixed64 Always eight bytes. More efficient than uint64 if values are often greater than 2^56. uint64 long int/long uint64 ulong integer/string Bignum
sfixed32 Always four bytes. int32 int int int32 int integer Bignum or Fixnum (as required)
sfixed64 Always eight bytes. int64 long int/long int64 long integer/string Bignum
bool bool boolean boolean bool bool boolean TrueClass/FalseClass
string A string must always contain UTF-8 encoded or 7-bit ASCII text. string String str/unicode string string string String (UTF-8)
bytes May contain any arbitrary sequence of bytes. string ByteString str []byte ByteString string String (ASCII-8BIT)