diff --git a/README.md b/README.md index b27e2b231..aac1266ac 100644 --- a/README.md +++ b/README.md @@ -105,99 +105,31 @@ There also are some unit tests, you can run those with cargo: $ cargo test --all ``` -# Using another standard library +# Using a standard library -As of now the standard library is under construction. +As of now the standard library is under construction. You can +use a standard library by setting the RUSTPYTHONPATH environment +variable. + +To do this, follow this method: + +```shell +$ export RUSTPYTHONPATH=~/GIT/RustPython/Lib +$ cargo run -- -c 'import xdrlib' +``` You can play around with other standard libraries for python. For example, the [ouroboros library](https://github.com/pybee/ouroboros). -To do this, follow this method: - -```shell -$ cd ~/GIT -$ git clone git@github.com:pybee/ouroboros.git -$ export RUSTPYTHONPATH=~/GIT/ouroboros/ouroboros -$ cd RustPython -$ cargo run -- -c 'import statistics' -``` - # Compiling to WebAssembly -At this stage RustPython only has preliminary support for web assembly. The instructions here are intended for developers or those wishing to run a toy example. - -## Setup - -To get started, install [wasm-pack](https://rustwasm.github.io/wasm-pack/installer/) and `npm`. ([wasm-bindgen](https://rustwasm.github.io/wasm-bindgen/whirlwind-tour/basic-usage.html) should be installed by `wasm-pack`. if not, install it yourself) - - - -## Build - -Move into the `wasm` directory. This directory contains a library crate for interop -with python to rust to js and back in `wasm/lib`, the demo website found at -https://rustpython.github.io/demo in `wasm/demo`, and an example of how to use -the crate as a library in one's own JS app in `wasm/example`. - -```sh -cd wasm -``` - -Go to the demo directory. This is the best way of seeing the changes made to either -the library or the JS demo, as the `rustpython_wasm` module is set to the global -JS variable `rp` on the website. - -```sh -cd demo -``` - -Now, start the webpack development server. It'll compile the crate and then -the demo app. This will likely take a long time, both the wasm-pack portion and -the webpack portion (from after it says "Your crate has been correctly compiled"), -so be patient. - -```sh -npm run dev -``` - -You can now open the webpage on https://localhost:8080 and Python code in either -the text box or browser devtools with: - -```js -rp.pyEval( - ` -print(js_vars['a'] * 9) -`, - { - vars: { - a: 9 - } - } -); -``` - -Alternatively, you can run `npm run build` to build the app once, without watching -for changes, or `npm run dist` to build the app in release mode, both for the -crate and webpack. +[See this doc](wasm/README.md) # Code style The code style used is the default [rustfmt](https://github.com/rust-lang/rustfmt) codestyle. Please format your code accordingly. +We also use [clippy](https://github.com/rust-lang/rust-clippy) to detect rust code issues. # Community diff --git a/docs/builtins.md b/docs/builtins.md deleted file mode 100644 index 2daa2b275..000000000 --- a/docs/builtins.md +++ /dev/null @@ -1,30 +0,0 @@ -Byterun - -* Builtins are exposed to frame.f_builtins -* f_builtins is assigned during frame creation, - self.f_builtins = f_locals['__builtins__'] - if hasattr(self.f_builtins, '__dict__'): - self.f_builtins = self.f_builtins.__dict__ -* f_locals has a __`____builtins___` field which is directly the `__builtins__` module - - -Jaspy - -* The `module()` function creates either a NativeModule or PythonModule -* The objects in the module are PyType.native -* The function call is abstracted as a `call` function, which handles different - -* IMPORT_NAME depends on `__import__()` in builtins - -TODO: - -* Implement a new type NativeFunction -* Wrap a function pointer in NativeFunction -* Refactor the CALL_FUNCTION case so it can call both python function and native function -* During frame creation, force push a native function `print` into the namespace -* Modify LOAD_* so they can search for names in builtins - -* Create a module type -* In VM initialization, load the builtins module into locals -* During frame creation, create a field that contains the builtins dict - diff --git a/docs/compat-test.md b/docs/compat-test.md deleted file mode 100644 index 35c84fdc0..000000000 --- a/docs/compat-test.md +++ /dev/null @@ -1,17 +0,0 @@ -https://wiki.python.org/moin/SimplePrograms -Simple HTTP Server -bottle.py -http://pypi-ranking.info/alltime - - simplejson - - pep8 - - httplib2 - - argparse - -http://learning-python.com/books/lp4e-examples.html - -https://docs.python.org/3/tutorial/introduction.html#numbers - -http://doc.pypy.org/en/latest/getting-started-dev.html#running-pypy-s-unit-tests - -CPython Regression suite -https://github.com/python/cpython/tree/master/Lib/test diff --git a/docs/frameTODO.txt b/docs/frameTODO.txt deleted file mode 100644 index 58b661ade..000000000 --- a/docs/frameTODO.txt +++ /dev/null @@ -1,13 +0,0 @@ -Create a frame class -change the environment to locals -turn run into run_code -> run_frame - -TEST - -Create a function class -implement MAKE_FUNCTION -implement CALL_FUNCTION -implement RETURN_VALUE - -TEST function - diff --git a/docs/insidepythonpresentation-120724001527-phpapp02.pdf b/docs/insidepythonpresentation-120724001527-phpapp02.pdf deleted file mode 100644 index 8e17cb246..000000000 Binary files a/docs/insidepythonpresentation-120724001527-phpapp02.pdf and /dev/null differ diff --git a/docs/notes.md b/docs/notes.md deleted file mode 100644 index 53f543e57..000000000 --- a/docs/notes.md +++ /dev/null @@ -1,13 +0,0 @@ -# TODO: other notes should be put into here - -# getattr() -- Required by this opcode [LOAD_ATTR](https://docs.python.org/3/library/dis.html#opcode-LOAD_ATTR) -- The builtin function: https://docs.python.org/3/library/functions.html?highlight=getattr#getattr -- - -# Memory management -- https://docs.python.org/3.6/c-api/memory.html - -# Bootstraping -- http://doc.pypy.org/en/latest/coding-guide.html#our-runtime-interpreter-is-rpython -- http://www.aosabook.org/en/pypy.html diff --git a/docs/reference.md b/docs/reference.md deleted file mode 100644 index 393d52948..000000000 --- a/docs/reference.md +++ /dev/null @@ -1,5 +0,0 @@ -Python bytecode: https://docs.python.org/3.4/library/dis.html -Python VM in python: https://github.com/nedbat/byterun -Python VM in JS: https://github.com/koehlma/jaspy - -http://www.skulpt.org/ diff --git a/docs/sharing.md b/docs/sharing.md deleted file mode 100644 index 19a4970fd..000000000 --- a/docs/sharing.md +++ /dev/null @@ -1,43 +0,0 @@ -# Sharing plan - -## Topics - -* Python architecture - * Compiler -> VM -* Flavors - * Cpython - * Pypy - * Byterun - * JSapy (?) -* Why Rust - * Security - * Learning -* Implementation plan - * VM first - * Compiler second - -* Tools for study - * dis doc - * byterun doc - * bytrun code - * cpython source - ---- -* Python VM - * Stack machine - -* Load add print - * dis - * Interpreter loop - * Python Types - -* Control flow - * Jump - * If - * Loop - ---- -* Function call - * Frame -* Builtins - diff --git a/docs/slides/201708_COSCUP/coscup-2017-rustpython_slide.html b/docs/slides/201708_COSCUP/coscup-2017-rustpython_slide.html deleted file mode 100644 index d453add7b..000000000 --- a/docs/slides/201708_COSCUP/coscup-2017-rustpython_slide.html +++ /dev/null @@ -1,529 +0,0 @@ - - - - Flavors (powered by MDSlides) - - - - - - - - - - - diff --git a/docs/slides/201708_COSCUP/pic/car_cutaway.jpg b/docs/slides/201708_COSCUP/pic/car_cutaway.jpg deleted file mode 100644 index 9c09878e2..000000000 Binary files a/docs/slides/201708_COSCUP/pic/car_cutaway.jpg and /dev/null differ diff --git a/docs/slides/201708_COSCUP/pic/electronic_parts.jpg b/docs/slides/201708_COSCUP/pic/electronic_parts.jpg deleted file mode 100644 index d04c95f91..000000000 Binary files a/docs/slides/201708_COSCUP/pic/electronic_parts.jpg and /dev/null differ diff --git a/docs/slides/201708_COSCUP/pic/ice-cream.jpg b/docs/slides/201708_COSCUP/pic/ice-cream.jpg deleted file mode 100644 index bc224b521..000000000 Binary files a/docs/slides/201708_COSCUP/pic/ice-cream.jpg and /dev/null differ diff --git a/docs/slides/201708_COSCUP/pic/python-logo.png b/docs/slides/201708_COSCUP/pic/python-logo.png deleted file mode 100644 index 738f6ed41..000000000 Binary files a/docs/slides/201708_COSCUP/pic/python-logo.png and /dev/null differ diff --git a/docs/slides/201708_COSCUP/pic/repo_QR.png b/docs/slides/201708_COSCUP/pic/repo_QR.png deleted file mode 100644 index c103af99e..000000000 Binary files a/docs/slides/201708_COSCUP/pic/repo_QR.png and /dev/null differ diff --git a/docs/slides/201708_COSCUP/pic/rust-servo.png b/docs/slides/201708_COSCUP/pic/rust-servo.png deleted file mode 100644 index e7f077930..000000000 Binary files a/docs/slides/201708_COSCUP/pic/rust-servo.png and /dev/null differ diff --git a/docs/slides/201708_COSCUP/slide.md b/docs/slides/201708_COSCUP/slide.md deleted file mode 100644 index fec3019cb..000000000 --- a/docs/slides/201708_COSCUP/slide.md +++ /dev/null @@ -1,286 +0,0 @@ -class: center, middle -##Python Interpreter in Rust -#### 2017/8/6, COSCUP -#### Shing Lyu - - -??? -top, middle, bottom -left, center, right - ---- -### About Me -* 呂行 Shing Lyu -* Mozilla engineer -* Servo team - -![rust_and_servo](pic/rust-servo.png) ---- -### Python's architecture -* Interpreted -* Garbage Collected -* Compiler => bytecode => VM - -![python](pic/python-logo.png) - ---- -background-image: url('pic/ice-cream.jpg') -class: center, middle, bleed, text-bg -# Flavors - - ---- -### Python Flavors -* CPython (THE python) -* Jython (JVM) -* IronPython (.NET) -* Pypy -* Educational - * Byterun - * Jsapy (JS) - * Brython (Python in browser) - ---- -class: center, middle - -# Why & How? ---- -### Why rewriting Python in Rust? -* Memory safety (?) -* Learn about Python's internal -* Learn to write Rust from scratch -* FUN! - ---- -### Implementation strategy -* Mostly follow CPython 3.6 -* Focus on the VM first, then the compiler - * Use the Python built-in compiler to generate bytecode -* Focus on learning rather than usability - ---- -### Milestones -* Basic arithmetics -* Variables -* Control flows (require JUMP) -* Function call (require call stack) -* Built-in functions (require native code) -* Run Python tutorial example code <= We're here -* Exceptions -* GC -* Run popular libraries - - ---- -class: center, middle, bleed, text-bg -background-image: url('pic/car_cutaway.jpg') -# Python Internals - ---- -### How Python VM works -* Stack machine - * Call stack and frames - * Has a NAMES list and CONSTS list - * Has a STACK as workspace -* Accepts Python bytecode -* `python -m dis source.py` - ---- - -### A simple Python code - -``` -#!/usr/bin/env python3 -print(1+1) -``` - -Running `python3 -m dis source.py` gives us - -``` - 1 0 LOAD_NAME 0 (print) - 3 LOAD_CONST 2 (2) - 6 CALL_FUNCTION 1 (1 positional, 0 keyword pair) - 9 POP_TOP - 10 LOAD_CONST 1 (None) - 13 RETURN_VALUE - -``` - ---- - -### LOAD_NAME "print" -* NAMES = ["print"] -* CONSTS = [None, 2] -* STACK: - -``` - | | - | print (native code)| - +--------------------+ -``` ---- -### LOAD_CONST 2 -* NAMES = ["print"] -* CONSTS = [None, 2] -* STACK: - -``` - | | - | 2 | - | print (native code)| - +--------------------+ -``` - ---- - -### CALL_FUNCTION 1 -1. `argument = stack.pop()` (argument == 2) -2. `function = stack.top()` (function == print) -3. call `print(2)` - -* NAMES = ["print"] -* CONSTS = [None, 2] -* STACK: - -``` - | | - | print (native code)| - +--------------------+ -``` ---- -### POP_TOP -* NAMES = ["print"] -* CONSTS = [None, 2] -* STACK: - -``` - | | - | (empty) | - +--------------------+ -``` - ---- -### LOAD_CONST None -* NAMES = ["print"] -* CONSTS = [None, 2] -* STACK: - -``` - | | - | None | - +--------------------+ -``` - ---- -### RETURN_VALUE - -(returns top of stack == None) - ---- - -class: center, middle, bleed, text-bg -background-image: url('pic/electronic_parts.jpg') -# Technical Detail - ---- - -### Bytecode format -* `dis` output format is for human reader -* Implementing a `dis` format parser is a waste of time -* Emit JSON bytecode using the [bytecode](https://pypi.python.org/pypi/bytecode/0.5) module - -``` - -code = compile(f,...) # Python built-in, return a Code object - -bytecode.Bytecode() - .from_code(code) - .to_concrete_bytecode() -``` -* Load into Rust using `serde_json` ---- - -### Types -* Everything is a `PyObject` in CPython -* We'll need that class hierarchy eventually -* Use a Rust `enum` for now - -``` -pub enum NativeType{ - NoneType, - Boolean(bool), - Int(i32), - Str(String), - Tuple(Vec), - ... -} -``` - ---- - -### Testing -* `assert` is essential to for unittests -* `assert` raises `AssertionError` -* Use `panic!()` before we implement exception - -``` -assert 1 == 1 -``` -``` - 1 0 LOAD_CONST 0 (1) - 3 LOAD_CONST 0 (1) - 6 COMPARE_OP 2 (==) - 9 POP_JUMP_IF_TRUE 18 - 12 LOAD_GLOBAL 0 (AssertionError) - 15 RAISE_VARARGS 1 - >> 18 LOAD_CONST 1 (None) - 21 RETURN_VALUE -``` - ---- -### Native Function - -* e.g. `print()` - -``` -pub enum NativeType { - NativeFunction(fn(Vec) -> NativeType), - ... -} - -match stack.pop() { - NativeFunction(func) => return_val = func(), - _ => ... -} - -``` - ---- - -### Next steps -* Exceptions -* Make it run a small but popular tool/library -* Implement the parser -* Figure out garbage collection -* Performance benchmarking - ---- -### Contribute - -## https://github.com/shinglyu/RustPython - -![qr_code](pic/repo_QR.png) - ---- -class: middle, center - -# Thank you - ---- - -### References -* [`dis` documentation](https://docs.python.org/3.4/library/dis.html) -* [byterun](http://www.aosabook.org/en/500L/a-python-interpreter-written-in-python.html) -* [byterun (GitHub)](https://github.com/nedbat/byterun/) -* [cpython source code](https://github.com/python/cpython) - diff --git a/docs/slides/intro/pic/ice-cream.jpg b/docs/slides/intro/pic/ice-cream.jpg deleted file mode 100644 index bc224b521..000000000 Binary files a/docs/slides/intro/pic/ice-cream.jpg and /dev/null differ diff --git a/docs/slides/intro/slide.md b/docs/slides/intro/slide.md deleted file mode 100644 index 76839286f..000000000 --- a/docs/slides/intro/slide.md +++ /dev/null @@ -1,180 +0,0 @@ -class: center, middle -##Python Interpreter in Rust -###Introduction -#### 2017/3/28 -#### Shing Lyu - - -??? -top, middle, bottom -left, center, right - ---- -name: toc -###Agenda -1. Category -1. Category -1. Category -1. Category -1. Category -1. Category -1. Category - -??? -This is a template ---- - -### Python's architecture -* Interpreted -* Garbage Collected -* Compiler => bytecode => VM - ---- -background-image: url('pic/ice-cream.jpg') -class: bleed -# Flavors - - ---- -### Python Flavors -* CPython (THE python) -* Jython (JVM) -* IronPython (.NET) -* Pypy -* Educational - * Byterun - * Jsapy (JS) - * Brython (Python in browser) - ---- -### Why rewriting Python in Rust? -* Memory safety - -* Learn about Python internal -* Learn real world Rust - ---- -### Implementation strategy -* Focus on the VM first, then the compiler - * Reuse the Python built-in compiler to generate bytecode -* Basic arithmetics -* Control flows (require JUMP) -* Function call (require call stack) -* Built-in functions (require native code) -* Run popular libraries - - ---- -### References -* [`dis` documentation](https://docs.python.org/3.4/library/dis.html) -* [byterun](http://www.aosabook.org/en/500L/a-python-interpreter-written-in-python.html) -* [byterun (GitHub)](https://github.com/nedbat/byterun/) -* [cpython source code](https://github.com/python/cpython) - ---- -### How Python VM works -* Stack machine -* Accepts Python bytecode -* `python -m dis source.py` - ---- - -### A simple Python code - -``` -#!/usr/bin/env python3 -print(1+1) -``` - -We run `python3 -m dis source.py` - ---- - -### The bytecode - -``` - 1 0 LOAD_NAME 0 (print) - 3 LOAD_CONST 2 (2) - 6 CALL_FUNCTION 1 (1 positional, 0 keyword pair) - 9 POP_TOP - 10 LOAD_CONST 1 (None) - 13 RETURN_VALUE - -``` - ---- - -### LOAD_NAME "print" -* NAMES = ["print"] -* CONSTS = [None, 2] -* STACK: - -``` - | | - | print (native code)| - +--------------------+ -``` ---- -### LOAD_CONST 2 -* NAMES = ["print"] -* CONSTS = [None, 2] -* STACK: - -``` - | | - | 2 | - | print (native code)| - +--------------------+ -``` - ---- - -### CALL_FUNCTION 1 -1. argument = stack.pop() (argument == 2) -2. function = stack.top() (function == print) -3. call print(2) - -* NAMES = ["print"] -* CONSTS = [None, 2] -* STACK: - -``` - | | - | print (native code)| - +--------------------+ -``` ---- -### POP_TOP -* NAMES = ["print"] -* CONSTS = [None, 2] -* STACK: - -``` - | | - | (empty) | - +--------------------+ -``` - ---- -### LOAD_CONST 1 -* NAMES = ["print"] -* CONSTS = [None, 2] -* STACK: - -``` - | | - | None | - +--------------------+ -``` - ---- -### RETURN_VALUE - -(returns top of stack == None) - ---- - -### Next step -* Make it run a small but popular tool/library -* Implement the parser -* Performance benchmarking diff --git a/docs/study.md b/docs/study.md deleted file mode 100644 index aff15aaf9..000000000 --- a/docs/study.md +++ /dev/null @@ -1,6 +0,0 @@ -Topic to study -================== -* How to save a Rust Iterator on the stack? -* Study how blocks are handled in CPython. - * The `why` var? - * Why unwind everything when a `break` happened? diff --git a/tests/Pipfile b/tests/Pipfile index 4bbae44ef..8ccbddc4d 100644 --- a/tests/Pipfile +++ b/tests/Pipfile @@ -10,4 +10,4 @@ pytest = "*" [dev-packages] [requires] -python_version = "3.6" +python_version = "3" diff --git a/wasm/README.md b/wasm/README.md new file mode 100644 index 000000000..2dd390b70 --- /dev/null +++ b/wasm/README.md @@ -0,0 +1,72 @@ + +# Compiling to webassembly + +At this stage RustPython only has preliminary support for web assembly. The instructions here are intended for developers or those wishing to run a toy example. + +## Setup + +To get started, install [wasm-pack](https://rustwasm.github.io/wasm-pack/installer/) and `npm`. ([wasm-bindgen](https://rustwasm.github.io/wasm-bindgen/whirlwind-tour/basic-usage.html) should be installed by `wasm-pack`. if not, install it yourself) + + + +## Build + +Move into the `wasm` directory. This directory contains a library crate for interop +with python to rust to js and back in `wasm/lib`, the demo website found at +https://rustpython.github.io/demo in `wasm/demo`, and an example of how to use +the crate as a library in one's own JS app in `wasm/example`. + +```sh +cd wasm +``` + +Go to the demo directory. This is the best way of seeing the changes made to either +the library or the JS demo, as the `rustpython_wasm` module is set to the global +JS variable `rp` on the website. + +```sh +cd demo +``` + +Now, start the webpack development server. It'll compile the crate and then +the demo app. This will likely take a long time, both the wasm-pack portion and +the webpack portion (from after it says "Your crate has been correctly compiled"), +so be patient. + +```sh +npm run dev +``` + +You can now open the webpage on https://localhost:8080 and Python code in either +the text box or browser devtools with: + +```js +rp.pyEval( + ` +print(js_vars['a'] * 9) +`, + { + vars: { + a: 9 + } + } +); +``` + +Alternatively, you can run `npm run build` to build the app once, without watching +for changes, or `npm run dist` to build the app in release mode, both for the +crate and webpack.