ThriftPy2¶
ThriftPy2 is a pure python implementation of Apache Thrift in a pythonic way.
The official thrift python lib is not pythonic at all, it needs a complicated process of installation, and the generated sdk is very ugly. Everytime the thrift file changed you have to re-generate the sdk which causes more pain in development.
ThriftPy2 helps that, it’s compatible with Apache Thrift so you no longer need to install ‘thrift’ package, it can import thrift file on the fly so you no longer need to re-generate the sdk again and again and again.
Github: https://github.com/Thriftpy/thriftpy2
Code Demo¶
ThriftPy2 make it super easy to write server/client code with thrift. Let’s checkout this simple pingpong service demo.
We need a ‘pingpong.thrift’ file:
service PingPong {
string ping(),
}
Then we can make a server:
import thriftpy2
pingpong_thrift = thriftpy2.load("pingpong.thrift", module_name="pingpong_thrift")
from thriftpy2.rpc import make_server
class Dispatcher(object):
def ping(self):
return "pong"
server = make_server(pingpong_thrift.PingPong, Dispatcher(), '127.0.0.1', 6000)
server.serve()
And a client:
import thriftpy2
pingpong_thrift = thriftpy2.load("pingpong.thrift", module_name="pingpong_thrift")
from thriftpy2.rpc import make_client
client = make_client(pingpong_thrift.PingPong, '127.0.0.1', 6000)
client.ping()
See, it’s that easy!
You can refer to ‘examples’ and ‘tests’ directory in source code for more usage examples.
Features¶
Currently ThriftPy2 have these features (also advantages over the upstream python lib):
Supports Python 2.7, Python 3.4+, PyPy and PyPy3.
Pure python implementation. No longer need to compile & install the ‘thrift’ package. All you need is thriftpy2 and thrift file.
Compatible with Apache Thrift. You can use ThriftPy2 together with the official implementation servers and clients, such as a upstream server with a thriftpy2 client or the opposite.
Currently implemented protocols and transports:
- binary protocol (python and cython)
- compact protocol (python and cython)
- json protocol
- buffered transport (python & cython)
- framed transport
- tornado server and client (with tornado 4.0)
Can directly load thrift file as module, the sdk code will be generated on the fly.
For example,
pingpong_thrift = thriftpy2.load("pingpong.thrift", module_name="pingpong_thrift")
will load ‘pingpong.thrift’ as ‘pingpong_thrift’ module.Or, when import hook enabled by
thriftpy2.install_import_hook()
, you can directly useimport pingpong_thrift
to import the ‘pingpong.thrift’ file as module, you may also usefrom pingpong_thrift import PingService
to import specific object from the thrift module.Easy RPC server/client setup.
Installation¶
Install with pip.
$ pip install thriftpy2
You may also install cython first to build cython extension locally.
$ pip install cython thriftpy2
Usage Notice¶
Cython Binary Protocol¶
The Cython accelerating binary protocol is enabled by default for CPython if it’s available, but disabled for PyPy.
To force use pure python version of binary protocol, you must import them from the direct module.
from thriftpy2.protocol.binary import TBinaryProtocolFactory
from thriftpy2.transport.buffered import TBufferedTransportFactory
from thriftpy2.transport.framed import TFramedTransportFactory
Better Module¶
To load thrift file as better module, provide a module_name in load.
The direct loaded TObjects can’t be pickled.
>>> ab = thriftpy2.load("addressbook.thrift")
>>> pickle.dumps(ab.AddressBook())
PicklingError: Can't pickle <class 'addressbook.AddressBook'>
TObjects can be pickled when load with module_name provided.
>>> ab = thriftpy2.load("addressbook.thrift", "addressbook_thrift")
>>> pickle.dumps(ab.AddressBook())
b'\x80\x03caddressbook_thrift\nAddressBook\nq\x00)\x81q\x01}q\x02X\x06\x00\x00\x00peopleq\x03Nsb.'
You can also use from … import … style after a standard module load.
>>> ab = thriftpy2.load("addressbook.thrift", "addressbook_thrift")
>>> from addressbook_thrift import *
Benchmarks¶
Some benchmark results:
# apache thrift py binary
binary protocol struct benchmark for 100000 times:
encode -> 3.74061203003
decode -> 5.02829790115
# apache thrift c binary
accelerated protocol struct benchmark for 100000 times:
encode -> 0.398949146271
decode -> 0.536000013351
# thriftpy2 & pypy2.3
binary protocol struct benchmark for 100000 times:
encode -> 0.413738965988
decode -> 0.605606079102
# thriftpy2 & py3.4
binary protocol struct benchmark for 100000 times:
encode -> 3.291545867919922
decode -> 4.337666034698486
# thriftpy2 & py3.4 + cython
cybin protocol struct benchmark for 100000 times:
encode -> 0.5828649997711182
decode -> 0.8259570598602295
Checkout the benchmark/benchmark.rst for detailed benchmark scripts and scores.
Contribute¶
- Fork the repo and make changes.
- Write a test which shows a bug was fixed or the feature works as expected.
- Make sure travis-ci test succeed.
- Send pull request.
Changelog¶
Version 0.4.1¶
Released on February 18, 2019.
Version 0.3.9¶
Released on August 26, 2016.
- add support for timeout and ssl in make_server / make_client helper funcs, via #204, #205 and #229.
- add support for thrift_file path in http protocol, via #225.
- preserve traceback when re-raise undeclared exception, via #206.
- performance improvement by dynamically compile spec’d __init__ functions, via #210 and #227.
- performance improvement by refine cython encoding/decoding, via #211 and #212_.
- bugfix for type error in cast_byte parser and improve include dirs function, via #214
- bugfix for parse error when field begin with true/false keyword, via #215 and #218.
- bugfix for is_open not return false when socket closed after open, via #230.
Version 0.3.8¶
Released on May 3, 2016.
Version 0.3.7¶
Released on Mar 24, 2016.
- bugfix for a possible unicode decode error in cybin.
- use a better unhashable implementation for payload.
Version 0.3.6¶
Released on Mar 24, 2016.
Version 0.3.5¶
Released on Feb 16, 2016.
- fix another set_timeout backward compat issue introduced in last version.
- make thrift container struct unhashable, via #184.
Version 0.3.4¶
Released on Feb 3, 2016.
- fix backward compat issue introduced in last version, add back set_timeout api in socket.
Version 0.3.3¶
Released on Jan 21, 2016.
Version 0.3.2¶
Released on Oct 12, 2015.
- add __thrift_meta__ attribute to loaded module, via #138.
- add type validation before write data to transport, via #149 and #150.
- add load_fp api to load thrift from file like object, via #154.
- add support for recursive struct definition, via #155.
- add support for integer boolean constants, via #161.
- simplify the read_i08() bool result cast, via #162.
- performance improvements on payload init() func, via #163.
- bugfix for parsing of duplicate field name or id, now will raise error when duplicates detected, via #139.
- bugfix for server side transport not connected error when closing socket, via #143.
- bugfix for a typo error in default_spec generation, via #145.
- bugfix for i16 byte swap bug in OS X, via #148.
Version 0.3.1¶
Released on May 29, 2015.
Version 0.3.0¶
Released on April 15, 2015.
Non-Backward Compatible changes:
- migrate multiplexed protocol implementation to the same with upstream, via #117.
0.2.x¶
Version 0.2.1¶
Released on April 15, 2015.
- add an experimental tracking feature in thriftpy.contrib, via #96.
- add limitation on thrift reserved keyword for compatible with upstream, via #115.
- bugfix EOF grammar error, via #103.
- bugfix for mis-mach transport in client caused server crash, via #119.
- bugfix for typedef on included thrift files, via #121.
Version 0.2.0¶
Released on March 3, 2015.
- support for default enum values that reference the original enum, via #69.
- support for require keyword, via #72.
- support for allow use and definition of types in the same file, via #77.
- support for multiplexing for services, via #88.
- support for cython accelerated memory transport and framed transport, via #93
- bugfix for transport clean in read_struct in cybin, via #70.
- bugfix for large reading size in framed transport, via #73.
- bugfix for cython build failed in older CentOS, via #92.
- bugfix for thrift file version mis-match caused message corrupt in read_struct, via #95.
Non-Backward Compatible changes:
- refined new parser, the parser now behaves very similar to Apache Thrift, and supports a lot more features than the old one, via #80. Refer to the pull request for more detailed changes.
- refined transport, all transports have cython accelerated version. The cython version of protocol and transport are enabled by default now.
0.1.x¶
Version 0.1.15¶
Released on December 12, 2014.
- add MIT LICENSE file as requested.
- tests refines with tox and pytest fixtures.
- support for a mostly cythonized version of framed transport, via #66.
- bugfix for unix socket param in rpc.
- bugfix for receiving 0-length strings & framed transport, via #63.
- bugfix for json protocol unicode decode error, via #65.
- bugfix for operator __ne__ implementation error, via #68.
Version 0.1.14¶
Released on November 8, 2014.
- support for python2.6.
- support for testing by tox.
- support for oneway keyword, via #49.
- bugfix for wrong type args, via #48.
- bugfix for thrift file include keyword, via #53.
- bugfix for skip method not found in protocol, via #55.
- bugfix for set type support, via #59.
- bugfix for ‘api’ arg name collision in client.
Version 0.1.13¶
Released on September 24, 2014.
Version 0.1.12¶
Released on September 18, 2014.
- bugfix for lack of skip func in cython binary protocol, via #43.
Version 0.1.11¶
Released on September 16, 2014.
- bugfix for init func generator for TStruct.
- bugfix for set constants in parser, via #39.
- add support for “includes” and service “extends”, via #37.
- add close() to servers, via #38.
- implement non-strict mode for binary protocol, via #40.
- removed cython ext in pypy, and add pypy3 support.
- some args updates: * add trans_factory arg to make_server * rename rbuf_size in buffered transport to buf_size. * rename isOpen to is_open, readFrame to read_frame.
Version 0.1.10¶
Released on September 4, 2014.
Version 0.1.9¶
Released on September 1, 2014.
- refine cython binary protocol, add cython buffered transport, via #32.
- param name change, rename transport_factory to trans_factory in rpc.
Version 0.1.8¶
Released on August 28, 2014.
Version 0.1.7¶
Released on August 19, 2014.
Version 0.1.6¶
Released on August 14, 2014.
- json protocol, via #21.
- more standard module for loaded sdk, now generated TPayload objects can be pickled when module_name provided, via #22.
- gunicorn_thrift integration pingpong example, via #24.
- token cache now only checks python’s major and minor version.
- bugfix for exception handling in void api in RPC request.
- bugfix for negative number value not recognized.
- bugfix for cybin protocol to allow None value in struct.
- bugfix for double free or corruption in cybin protocol, via #26.
Version 0.1.5¶
Released on July 25, 2014.
Version 0.1.4¶
Released on July 17, 2014.
Version 0.1.3¶
Released on June 19, 2014.
- support for union, binary fields, support for empty structs, support for Apache Storm thrift file, via #14.
- bugfix for import hook
- bugfix for skip function in binary protocols
Version 0.1.2¶
Released on June 7, 2014.
- disabled the magic import hook by default. and add install/remove function to switch the hook on and off.
- reworked benchmark suit and add benchmark results.
- new __init__ function code generator. get a noticable speed boost.
- bug fixes
Version 0.1.1¶
First public release.