`

Protobuf for node

 
阅读更多

Protobuf for Node adds idiomatic protocol buffer handling to node.JS.

It is actually two things in one: firstly, you can marshal protocol messages to and from Node byte buffers and send them over the wire in pure JS.

Secondly, you can use protocol messages as a native interface from JS to C++ add-ons in the same process. It takes away the details of the V8 and the eio threadpool APIs and makes native extensions to Node much easier to implement.

How to use

To read/write protocol buffers, protobuf for node needs the parsed representation of your protocol buffer schema (which is in protobuf format itself). The protobuf compiler spits out this format:

$ vi feeds.proto
package feeds;
message Feed {
  optional string title = 1;
  message Entry {
    optional string title = 1;
  }
  repeated Entry entry = 2;
}
:wq
$ $PROTO/bin/protoc --descriptor_set_out=feeds.desc --include_imports feeds.proto

Using this file (read into a Buffer) you can create a 'Schema' object:

 var fs = require('fs');
 var Schema = require('protobuf_for_node').Schema;
 var schema = new Schema(fs.readFileSync('feeds.desc'));

A Schema object maps fully qualified protocol buffer names to type objects that know how to marshal JS objects to and from Buffers:

 var Feed = schema['feeds.Feed'];
 var aFeed = Feed.parse(aBuffer);
 var serialized = Feed.serialize(aFeed);

When marshalling, the serializer accepts any JavaScript object (but only picks the properties defined in the schema):

 var serialized = Feed.serialize({ title: 'Title', ignored: 42 });
 var aFeed = Feed.parse(serialized);  => { title: 'Title' }

Note that property names are the camel-cased version of the field names in the protocol buffer description, like for the Java version (http://code.google.com/apis/protocolbuffers/docs/reference/java-generated.html#fields). E.g. "optional string title_string = 1" will translate into a "titleString" JS property.

Note how we're using uppercase for the type objects like for constructor functions. That's because they are: when parsing, objects are constructed using the respective constructor for the message type. This means that you are able to attach methods:

  Feed.prototype.numEntries = function() {
    return this.entry.length;
  };
  var aFeed = Feed.parse(Feed.serialize({ entry: [{}, {}] }));
  aFeed.numEntries()  =>  2

Native Interface

Protocol buffers aren't only great for data interchange between processes - you can also use them to send data between code written in JS and C++ within the same process. Protobuf for node makes it simple to implement a native add-on without having to touch the V8 api at all: you implement a protobuf service instead. It's three easy steps:

1. Define the add-on service interface:

// An example service to query pwd(3).
package pwd;

// Empty (=no arg) request message.
message EntriesRequest {
}

message Entry {
  optional string name = 1;
  optional int32 uid = 2;
  optional int32 gid = 3;
  optional string home = 4;
  optional string shell = 5;
}
message EntriesResponse {
  repeated Entry entry = 1;
}

service Pwd {
  rpc GetEntries(EntriesRequest) returns (EntriesResponse);
}

... and generate the C++ code for it:

proto/bin/protoc --cpp_out=. service.proto

2. Implement and export the service in an add-on:

extern "C" void init(v8::Handle<v8::Object> target) {
  // Look Ma - no V8 api required!

  // Simple synchronous implementation.
  protobuf_for_node::ExportService(
      target, "pwd",
      new (class : public pwd::Pwd {
        virtual void GetEntries(google::protobuf::RpcController*,
                                const pwd::EntriesRequest* request,
                                pwd::EntriesResponse* response,
                                google::protobuf::Closure* done) {
          struct passwd* pwd;
          while ((pwd = getpwent())) {
            pwd::Entry* e = response->add_entry();
            e->set_name(pwd->pw_name);
            e->set_uid(pwd->pw_uid);
            e->set_gid(pwd->pw_gid);
            e->set_home(pwd->pw_dir);
            e->set_shell(pwd->pw_shell);
          }
          setpwent();
          done->Run();
        }
      }));

3. Use it from JS:

// prints the user database
puts(JSON.stringify(require('pwd').pwd.GetEntries({}), null, 2));

If your service is CPU intensive, you should call it with an extra callback argument: the invocation is automatically placed on the eio thread pool and does not block node:

// prints the user database
require('pwd').pwd.GetEntries({}, function(response) {
  puts(JSON.stringify(response), null, 2);
});

Your service is free to finish and call the "done" closure asynchronously.

Note that marshalling between JS and the request and response protos necessarily happens on the JS thread. Passing tons of data will block just as much using asynchronous invocation.

Speed

The Protobuf for Node add-on relies on the protobuf C++ runtime but it does not require any generation, compilation or linkage of generated C++ code. It works reflectively and is thus able to deal with arbitrary schemas. This means however, that it is not as fast as with generated code. Simple measurements show that unmarshalling is about between 20% and 50% faster than V8's native JSON support.

Calling C++ services is faster since it avoids marshalling to bytes but transfers data directly between the JS objects and the protocol messages. Also, in this case you compile the generated service interface into your code, so reflection is faster.

Building and running

You need to have node.js 0.4 and protobuf 2.4.0a installed. protobuf_for_node depends on the respective versions of v8 and eio shipped by node. Use branch node2 for node0.2/protobuf 2.3.0. If you want to use a later protobuf version, you may need to re-generate the example protoservice codecd example; protoc --cpp_out=. protoservice.proto.

To build protobuf for node, do:

PROTOBUF=<protobuf prefix> /path/to/node/bin/node-waf configure clean build

This will build a) the protobuf_for_node library (build/default/libprotobuf_for_node_lib.so). It is linked to by your service-exporting add-ons and by the b) protobuf_for_node add-on (build/default/protobuf_for_node.node).

You need a) and the protobuf library in your DYLD_LIBRARY_PATH and b) in your NODE_PATH if you require('protobuf_for_node').

Running the examples

The examples should be run from within the protobuf-for-node directory, e.g. "node example/feeds.js".

Limitations

There are mismatches between the protocol buffer and javascript type systems. For example, the JS Number datatype does not offer enough precision to represent proto's uint64 datatypes.

There is no specification for the type conversions performed during JS->proto conversion. There may be surprises when you marshal, say, a JS object into an int64 proto field. That being said, the behavior is more or less in line with the ECMA conversions like ToNumber et al.

The marshalling code does not preserve unknown fields nor extensions.

分享到:
评论

相关推荐

    protobuf for vs2015

    protobuf for vs2015 VS2015的protobuf插件

    protobuf for unity 在unity中使用protobuf工程示例

    protobuf for unity 在unity中使用protobuf工程示例,数据的序列化和反序列化工程示例

    protobuf for windows php

    标题中的"protobuf for windows php"指的是在Windows操作系统上使用PHP语言与Protocol Buffers(简称protobuf)进行交互的技术。Protocol Buffers是Google推出的一种数据序列化协议,它允许开发者定义数据结构,然后...

    protobuf for java 插件

    protobuf for Java 插件是Google开发的一种数据序列化协议(Protocol Buffers)在Java平台上的实现,用于高效、紧凑地存储和传输结构化数据。它提供了编译器,可以将定义的数据结构转换为Java类,使得开发者能方便地...

    protobuf-for-with-python

    标题“protobuf-for-with-python”指的是将Protocol Buffers(通常简称为Protobuf)与Python编程语言结合使用的主题。Protocol Buffers是Google开发的一种数据序列化协议,它允许开发者定义数据结构,然后生成能够在...

    protobuf3.20.1 for android

    标题中的"protobuf3.20.1 for android"指的是Google开发的一种高效的数据序列化协议——Protocol Buffers(protobuf)的3.20.1版本,专门为Android平台的arm64-v8a架构优化。Protocol Buffers是一种语言无关、平台...

    compile_protobuf_protobuf:compile_protobuf_

    在标签"protobuf:compile protobuf"中,"protobuf"是指protobuf库本身,而"compile protobuf"是指执行protobuf编译器`protoc`来生成目标语言的代码。这个过程通常分为以下几个步骤: 1. **定义消息类型**:在.proto...

    protobuf中文学习文档

    **protobuf中文学习文档** **一、protobuf简介** Protocol Buffers(简称protobuf)是Google开发的一种数据序列化协议,用于高效地存储和传输结构化数据。它通过定义消息格式,将数据结构转换为二进制流,相比XML...

    protoc-3.0.0-win32.zip 和 protobuf3-for-unity

    压缩包包含两个文件,protoc-3.0.0-win32.zip 和 protobuf3-for-unity。编译成功 protobuf3-for-unity-3.0.0\src\Google.Protobuf\bin\Release\net35目录下会有一个Google.Protobuf.dll,这个dll就是我们要用在unity...

    protobufprotobufprotobufprotobuf

    protobuf,全称Protocol Buffers,是Google开发的一种数据序列化协议。它允许开发者定义数据结构,然后将结构化的数据序列化,可用于数据存储、通信协议等方面。Protocol Buffers的设计目标是提供一种更高效、更灵活...

    protobuf3.2.0

    protobuf-3.2.0-cp27-cp27mu-manylinux1_x86_64.whl,protobuf for python2.7

    protobuf source v3.21.12

    描述中提到的"for research and learning"表明这个源代码包不仅适用于实际项目,还非常适合学习protobuf的工作原理和实现。通过阅读源代码,开发者可以了解如何设计和实现一个高效的序列化库,这对于提升编程技能和...

    protobuf-3.5-src

    **protobuf-3.5-src** 是一个包含 Google Protocol Buffers(简称protobuf)版本3.5.0源代码的压缩包。Protocol Buffers 是一种高效、灵活且自动化的数据序列化机制,由 Google 开发,广泛应用于分布式系统和跨平台...

    protobuf 3.11版本,静态编译

    protobuf是Protocol Buffers的简称,它是Google开发的一种数据序列化协议,用于结构化数据的序列化,类似于XML、JSON,但更小、更快、更简单。protobuf 3.11版本是一个稳定且广泛使用的版本,提供了许多改进和新特性...

    google protobuf 最新源代码

    google protobuf 最新源代码google protobuf 最新源代码google protobuf 最新源代码google protobuf 最新源代码google protobuf 最新源代码google protobuf 最新源代码google protobuf 最新源代码google protobuf ...

    Protobuf-master包

    **protobuf-master** 是一个包含 Protocol Buffers(简称 Protobuf)源代码和资源的项目,主要服务于 Android、Java 开发者。Protocol Buffers 是 Google 推出的一种数据序列化协议,它提供了一种高效、灵活且自动化...

    protobuf-3.0.0最新版下载

    标题 "protobuf-3.0.0最新版下载" 指的是Protocol Buffers(简称protobuf)的3.0.0版本,这是一个Google开发的开源数据序列化协议。它提供了高效、灵活且跨语言的数据交换格式,广泛应用于网络通信、数据存储等领域...

    protobuf-js-3.17.0

    protobuf-js-3.17.0 是一个针对JavaScript环境...通过protobuf-3.17.0编译器,我们可以将.proto定义转换为JavaScript代码,以便在浏览器或Node.js中处理protobuf数据。这一版本特别强调稳定性,是开发者的理想选择。

    protobuf2.6.1

    protobuf2.6.1是一个流行的开源数据序列化协议,由Google开发,用于高效地存储和传输结构化数据。Protocol Buffers(简称protobuf)是比XML、JSON等格式更小巧、更快、更简单的一种序列化机制。它允许你定义数据结构...

    protobufDemo.rar

    protobufDemo protobuf例程protobufDemo protobuf例程protobufDemo protobuf例程protobufDemo protobuf例程protobufDemo protobuf例程protobufDemo protobuf例程protobufDemo protobuf例程

Global site tag (gtag.js) - Google Analytics