mirrors/ruby
0
0 Fork 0
mirror of https://github.com/ruby/ruby.git synced 2026-10-29 17:06:21 +08:00
Code Activity
89,983 Commits 33 Branches 1,200 Tags
ruby_3_4
Commit Graph
267 Commits
Author SHA1 Message Date
Takashi Kokubun 5ba052675f merge revision(s) fb18bb183c: [Backport #20989] 
[Bug #20989] Ripper: Pass `compile_error`

	For the universal parser, `rb_parser_reg_fragment_check` function is
	shared between the parser and ripper.  However `parser_params` struct
	is partially different, and `compile_error` function depends on that
	part indirectly.
 2025-02-13 17:50:05 -08:00
Nobuyoshi Nakada 7b2ae8df90 [Bug #20969] Pass assignable from ripper 
For the universal parser, `rb_reg_named_capture_assign_iter_impl`
function is shared between the parser and ripper.  However
`parser_params` struct is partially different, and `assignable`
function depends on that part indirectly.
 2024-12-19 23:20:09 +09:00
tompng 6743e6285a [Bug #20784] Fix incomplete character syntax followed by EOF 2024-10-05 15:59:01 +09:00
Peter Zhu 584559d86a Fix leak of token_info when Ripper#warn jumps 
For example, the following code leaks:

    class MyRipper < Ripper
      def initialize(src, &blk)
        super(src)
        @blk = blk
      end

      def warn(msg, *args) = @blk.call(msg)
    end

    $VERBOSE = true
    def call_parse = MyRipper.new("if true\n  end\n") { |msg| return msg }.parse

    10.times do
      500_000.times do
        call_parse
      end

      puts `ps -o rss= -p #{$$}`
    end

Before:

    37536
    53744
    70064
    86448
    102576
    119120
    135248
    151216
    167744
    183824

After:

    19280
    19696
    19728
    20336
    20448
    21408
    21616
    21616
    21824
    21840
 2024-08-07 09:14:14 -04:00
Peter Zhu ced35800d4 Fix leak in warning of duplicate keys when Ripper#warn jumps 
For example, the following code leaks:

    class MyRipper < Ripper
      def initialize(src, &blk)
        super(src)
        @blk = blk
      end

      def warn(msg, *args) = @blk.call(msg)
    end

    $VERBOSE = true
    def call_parse = MyRipper.new("if true\n  end\n") { |msg| return msg }.parse

    10.times do
      500_000.times do
        call_parse
      end

      puts `ps -o rss= -p #{$$}`
    end

Before:

    34832
    51952
    69760
    88048
    105344
    123040
    141152
    159152
    176656
    194272

After:

    18400
    20256
    20272
    20272
    20272
    20304
    20368
    20368
    20368
    20400
 2024-08-06 10:19:50 -04:00
Peter Zhu 6358397490 Fix leak of AST when Ripper#compile_error jumps 
For example, the following script leaks:

    class MyRipper < Ripper
      def initialize(src, &blk)
        super(src)
        @blk = blk
      end

      def compile_error(msg) = @blk.call(msg)
    end

    def call_parse = MyRipper.new("/") { |msg| return msg }.parse

    10.times do
      100_000.times do
        call_parse
      end

      puts `ps -o rss= -p #{$$}`
    end

Before:

    93952
    169040
    244224
    318784
    394432
    468224
    544048
    618560
    693776
    768384

After:

    19776
    19776
    20352
    20880
    20912
    21408
    21328
    21152
    21472
    20944
 2024-07-31 14:47:44 -04:00
Nobuyoshi Nakada 97449338d6 [Bug #20649] Allow nil as 2nd argument of assign_error 
Fallback to the last token element in that case, for the backward
compatibilities.
 2024-07-24 22:18:36 +09:00
Nobuyoshi Nakada a1f72a563b [Bug #20579] ripper: Dispatch spaces at END-OF-INPUT without newline 2024-06-14 17:54:02 +09:00
Nobuyoshi Nakada 2e59cf00cc [Bug #20578] ripper: Fix dispatching part at invalid escapes 2024-06-14 15:02:15 +09:00
Nobuyoshi Nakada 9e28354705 ripper: Fix excess compile_error at simple backref op_asgn 
Fix up 89cfc15207.
 2024-06-07 11:28:38 +09:00
Nobuyoshi Nakada 27321290d9 [Bug #20521] ripper: Clean up strterm 2024-06-06 20:43:56 +09:00
Nobuyoshi Nakada 05553cf22d [Bug #20517] Make a multibyte character one token at meta escape 2024-06-01 19:33:12 +09:00
Kevin Newton 47f0965269 Update duplicated when clause warning message 2024-05-24 12:36:54 -04:00
Nobuyoshi Nakada 5695c5df95 ripper: Fix opassign when assignment to backref variables 2024-05-12 15:38:22 +09:00
Nobuyoshi Nakada a1fb6cc978 ripper: Use $& instead of quoting charaters in tests 2024-05-12 15:36:42 +09:00
yui-knk 7e604a0263 Fix SEGV when ripper hits backref_error on command_asgn or arg 2024-05-11 20:47:15 +09:00
Peter Zhu 7ef8bb129f Fix memory leak in Ripper.sexp 
rb_ast_dispose does not free the rb_ast_t causing it to be leaked. This
commit changes it to use rb_ast_free instead.

For example:

    require "ripper"

    10.times do
      100_000.times do
        Ripper.sexp("")
      end

      puts `ps -o rss= -p #{$$}`
    end

Before:

    27648
    32512
    37376
    42240
    47232
    52224
    57344
    62208
    67072
    71936

After:

    22784
    22784
    22784
    22784
    22912
    22912
    22912
    22912
    22912
    22912
 2024-05-01 11:09:54 -04:00
yui-knk 515e52a0b1 Emit warn event for duplicated hash keys on ripper 
Need to use `rb_warn` macro instead of calling `rb_compile_warn`
directly to emit `warn` event on ripper.
 2024-04-15 06:29:25 +09:00
yui-knk 39be11a17a Fix segv when parsing command by ripper 
89cfc15207 made this event dispatch to pass `Qundef`
to user defined callback method by mistake.
This commit fix it to be `nil`.
 2024-04-11 10:28:58 +09:00
yui-knk 7767db2379 Fix ripper to dispatch warning event for duplicated when clause 
Need to separate `check_literal_when` function for parser and
ripper otherwise warning event is not dispatched because
parser `rb_warning1` is used in ripper.
 2024-04-07 11:15:09 +09:00
yui-knk 89cfc15207 [Feature #20257] Rearchitect Ripper 
Introduce another semantic value stack for Ripper so that
Ripper can manage both Node and Ruby Object separately.
This rearchitectutre of Ripper solves these issues.
Therefore adding test cases for them.

* [Bug 10436] https://bugs.ruby-lang.org/issues/10436
* [Bug 18988] https://bugs.ruby-lang.org/issues/18988
* [Bug 20055] https://bugs.ruby-lang.org/issues/20055

Checked the differences of `Ripper.sexp` for files under `/test/ruby`
are only on test_pattern_matching.rb.
The differences comes from the differences between
`new_hash_pattern_tail` functions between parser and Ripper.
Ripper `new_hash_pattern_tail` didn’t call `assignable` then
`kw_rest_arg` wasn’t marked as local variable.
This is also fixed by this commit.

```
--- a/./tmp/before/test_pattern_matching.rb
+++ b/./tmp/after/test_pattern_matching.rb
@@ -3607,7 +3607,7 @@
                  [:in,
                   [:hshptn, nil, [], [:var_field, [:@ident, “a”, [984, 13]]]],
                   [[:binary,
-                    [:vcall, [:@ident, “a”, [985, 10]]],
+                    [:var_ref, [:@ident, “a”, [985, 10]]],
                     :==,
                     [:hash, nil]]],
                   nil]]],
@@ -3662,7 +3662,7 @@
                  [:in,
                   [:hshptn, nil, [], [:var_field, [:@ident, “a”, [993, 13]]]],
                   [[:binary,
-                    [:vcall, [:@ident, “a”, [994, 10]]],
+                    [:var_ref, [:@ident, “a”, [994, 10]]],
                     :==,
                     [:hash,
                      [:assoclist_from_args,
@@ -3813,7 +3813,7 @@
                    [:command,
                     [:@ident, “raise”, [1022, 10]],
                     [:args_add_block,
-                     [[:vcall, [:@ident, “b”, [1022, 16]]]],
+                     [[:var_ref, [:@ident, “b”, [1022, 16]]]],
                      false]]],
                   [:else, [[:var_ref, [:@kw, “true”, [1024, 10]]]]]]]],
                nil,
@@ -3876,7 +3876,7 @@
                      [:@int, “0”, [1033, 15]]],
                     :“&&“,
                     [:binary,
-                     [:vcall, [:@ident, “b”, [1033, 20]]],
+                     [:var_ref, [:@ident, “b”, [1033, 20]]],
                      :==,
                      [:hash, nil]]]],
                   nil]]],
@@ -3946,7 +3946,7 @@
                      [:@int, “0”, [1042, 15]]],
                     :“&&“,
                     [:binary,
-                     [:vcall, [:@ident, “b”, [1042, 20]]],
+                     [:var_ref, [:@ident, “b”, [1042, 20]]],
                      :==,
                      [:hash,
                       [:assoclist_from_args,
@@ -5206,7 +5206,7 @@
                      [[:assoc_new,
                        [:@label, “c:“, [1352, 22]],
                        [:@int, “0”, [1352, 25]]]]]],
-                   [:vcall, [:@ident, “r”, [1352, 29]]]],
+                   [:var_ref, [:@ident, “r”, [1352, 29]]]],
                   false]]],
                [:binary,
                 [:call,
@@ -5299,7 +5299,7 @@
                       [:assoc_new,
                        [:@label, “c:“, [1367, 34]],
                        [:@int, “0”, [1367, 37]]]]]],
-                   [:vcall, [:@ident, “r”, [1367, 41]]]],
+                   [:var_ref, [:@ident, “r”, [1367, 41]]]],
                   false]]],
                [:binary,
                 [:call,
@@ -5931,7 +5931,7 @@
              [:in,
               [:hshptn, nil, [], [:var_field, [:@ident, “r”, [1533, 11]]]],
               [[:binary,
-                [:vcall, [:@ident, “r”, [1534, 8]]],
+                [:var_ref, [:@ident, “r”, [1534, 8]]],
                 :==,
                 [:hash,
                  [:assoclist_from_args,
```
 2024-02-20 17:33:58 +09:00
Yusuke Endoh 25d74b9527 Do not include a backtick in error messages and backtraces 
[Feature #16495]
 2024-02-15 18:42:31 +09:00
Nobuyoshi Nakada e5e1f9813e [Bug #19838] Flush delayed token nonconsecutive with the next token 2023-12-02 02:41:39 +09:00
Nobuyoshi Nakada e36b9760fd Dispatch invalid hex escape content too 2023-12-01 15:04:30 +09:00
Nobuyoshi Nakada d503e1b95a [Bug #20030] dispatch invalid escaped character without ignoring it 2023-12-01 15:04:30 +09:00
Hiroshi SHIBATA 7149543f24 Added TestRipper namespace to test_call_colon2 2023-11-02 15:13:40 +09:00
lukeg 1925c6d555 test/ripper: nest helper classes under TestRipper module 
Generic names like Node and NodeList should be namespaced properly.
 2023-11-02 15:10:50 +09:00
Nobuyoshi Nakada 40efbc7e40 [Bug #19851] Ripper: Hide internal block argument ID 2023-08-26 02:08:53 +09:00
Peter Zhu 0b8f15575a Fix memory leak for incomplete lambdas 
[Bug #19836]

The parser does not free the chain of `struct vtable`, which causes
memory leaks.

The following script reproduces this issue:

```
10.times do
  100_000.times do
    Ripper.parse("-> {")
  end

  puts `ps -o rss= -p #{$$}`
end
```
 2023-08-09 14:06:58 -04:00
Peter Zhu 5bc8fceca8 Fix memory leak in parser for incomplete tokens 
[Bug #19835]

The parser does not free the `tbl` of the `struct vtable` when there are
leftover `lvtbl` in the parser. This causes a memory leak.

The following script reproduces this issue:

```
10.times do
  100_000.times do
    Ripper.parse("class Foo")
  end

  puts `ps -o rss= -p #{$$}`
end
```
 2023-08-09 14:06:58 -04:00
Peter Zhu 4b6c584023 Remove --disable-gems for assert_separately 
assert_separately adds --disable=gems so we don't need to add
--disable-gems when calling assert_separately.
 2023-08-03 09:11:08 +09:00
Nobuyoshi Nakada 382678d411 [Bug #19788] Use the result of tCOLON2 event 2023-08-01 19:00:31 +09:00
Nobuyoshi Nakada 5c77402d88 Fix null pointer access in Ripper#initialize 
In `rb_ruby_ripper_parser_allocate`, `r->p` is NULL between creating
`self` and `parser_params` assignment.  As GC can happen there, the
typed-data functions for it need to consider the case.
 2023-07-16 15:41:10 +09:00
Peter Zhu 27d3fa2af0 Increase memory leak test timeout 
The test times out on some platforms, so increase the timeout.
 2023-06-28 13:47:48 -04:00
Peter Zhu a500eb9f8c Fix memory leak in Ripper 
The following script leaks memory in Ripper:

```ruby
require "ripper"

20.times do
  100_000.times do
    Ripper.parse("")
  end

  puts `ps -o rss= -p #{$$}`
end
```
 2023-06-28 09:50:51 -04:00
Nobuyoshi Nakada 6be402e172 [Bug #19736] Recover after unterminated interpolation 2023-06-20 20:10:46 +09:00
Nobuyoshi Nakada ac8a16237c [Bug #19563] Yield words separators per lines 
So that newlines across a here-doc terminator will be separated
tokens.

Cf. https://github.com/ruby/irb/pull/558
 2023-04-07 23:13:56 +09:00
Kazuki Tsujimoto 4ac8d11724 * in an array pattern should not be parsed as nil in ripper 
After 6c0925ba70, it was impossible
to distinguish between the presence or absence of `*`.

    # Before the commit
    Ripper.sexp('0 in []')[1][0][2][1]  #=> [:aryptn, nil, nil, nil, nil]
    Ripper.sexp('0 in [*]')[1][0][2][1] #=> [:aryptn, nil, nil, [:var_field, nil], nil]

    # After the commit
    Ripper.sexp('0 in []')[1][0][2][1]  #=> [:aryptn, nil, nil, nil, nil]
    Ripper.sexp('0 in [*]')[1][0][2][1] #=> [:aryptn, nil, nil, nil, nil]

This commit reverts it.
 2023-04-01 16:35:24 +09:00
Nobuyoshi Nakada 040fab3782 Ripper: Add keyword options to assert_parse_files 2023-03-28 17:14:34 +09:00
Kazuki Tsujimoto d51529244f [Bug #19175] p_kw without a sub pattern should be `assignable' 2023-03-26 18:57:34 +09:00
Kazuki Tsujimoto 6c0925ba70 [Bug #19175] p_rest should be `assignable' 
It should also check for duplicate names.
 2023-03-26 18:56:21 +09:00
Nobuyoshi Nakada fad48fefe1 [Bug #19399] Parsing invalid heredoc inside block parameter 
Although this is of course invalid as Ruby code, allow to just parse
and tokenize.
 2023-02-02 12:20:10 +09:00
Nobuyoshi Nakada 3becc4a105 [Bug #19291] Rewind to the previous line 
When rewinding looking ahead after newline token, also reset the last
line string, the pointers to it, and the location, not only the line
number.
 2023-01-02 16:12:08 +09:00
Nobuyoshi Nakada 1912bf5461 Adjut indent [ci skip] 2023-01-02 16:12:08 +09:00
Nobuyoshi Nakada ace2eee544 [Bug #18963] Separate string contents by here document terminator 2022-08-28 09:29:24 +09:00
Nobuyoshi Nakadaandtompng 844a0edbae [Bug #18962] Do not read again once reached EOF 
`Ripper::Lexer#parse` re-parses the source code with syntax errors
when `raise_errors: false`.

Co-Authored-By: tompng <tomoyapenguin@gmail.com>
 2022-08-12 15:58:18 +09:00
Nobuyoshi Nakada 1a70973f75 ripper: Check if anonymous parameters defined [Bug #18828] 2022-06-14 20:41:41 +09:00
Nobuyoshi Nakada 6daec46014 ripper: refine test to show expected states 2021-12-14 20:06:51 +09:00
Shugo Maeda 7686776c05 Hash values should be omitted in Ripper results 2021-09-11 22:03:10 +09:00
Nobuyoshi Nakada 2aa6826e81 Extract the wrapped value when yydebug [Bug #18075] 2021-08-15 11:40:06 +09:00