`FileTest.send(command, ...)` allows to call not only FileTest-related
methods but also any method that belongs to Kernel, Object, etc.
patched by <mame@ruby-lang.org>
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/branches/ruby_2_4@67820 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
WEBrick: prevent response splitting and header injection
This is a follow up to d9d4a28.
The commit prevented CRLR, but did not address an isolated CR or an
isolated LF.
Co-Authored-By: NARUSE, Yui <naruse@airemix.jp>
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/branches/ruby_2_4@67819 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
Loop with String#scan without creating substrings
Create the substrings necessary parts only, instead of cutting the
rest of the buffer. Also removed a useless, probable typo, regexp.
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/branches/ruby_2_4@67818 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
tmpdir.rb: permission of user given directory
* lib/tmpdir.rb (Dir.mktmpdir): check if the permission of the
parent directory only when using the default temporary
directory, and no check against user given directory. the
security is the user's responsibility in that case.
[ruby-core:91216] [Bug #15555]
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/branches/ruby_2_4@67148 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
Move autoload to toplevel
So that classes which uses Net::HTTP with https can use OpenSSL
namespace for example exception classes like OpenSSL::SSL::SSLError.
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/branches/ruby_2_4@66964 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
time.rb: fix parsing time zone in iso8601
* lib/time.rb (Time.xmlschema): a colon in time zone designator
can be omitted. [ruby-core:87277] [Bug #14790]
time.rb: fix parsing time zone in iso8601
* lib/time.rb (Time.xmlschema): the minute in time zone designator
can be omitted together with the preceding colon.
[ruby-core:87277] [Bug #14790]
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/branches/ruby_2_4@64148 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
lib/drb/extservm.rb (service): do not return `false'
invoke_service_command may set entries in @servers to `false',
making it incompatible with the intended use of the
safe navigation operator.
This caused occasional DRb test failures, but they were hidden
with automatic retry.
[ruby-core:87524] [Bug #14856]
Fixes: r53111 ("use safe navigation operator")
commit 059c9c1cf3 [GH-1142]
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/branches/ruby_2_4@64128 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
webrick: support Proc objects as body responses
* lib/webrick/httpresponse.rb (send_body): call send_body_proc
(send_body_proc): new method
(class ChunkedWrapper): new class
* test/webrick/test_httpresponse.rb (test_send_body_proc): new test
(test_send_body_proc_chunked): ditto
[Feature #855]
webrick/httpresponse: IO.copy_stream for regular files
Remove the redundant _send_file method since its functionality
is unnecessary with IO.copy_stream. IO.copy_stream also allows
the use of sendfile under some OSes to speed up copies to
non-TLS sockets.
Testing with "curl >/dev/null" and "ruby -run -e httpd" to
read a 1G file over Linux loopback reveals a reduction from
around ~0.770 to ~0.490 seconds on the client side.
* lib/webrick/httpresponse.rb (send_body_io): use IO.copy_stream
(_send_file): remove
[Feature #14237]
webrick: use IO.copy_stream for single range response
This is also compatible with range responses generated
by Rack::File (tested with rack 2.0.3).
* lib/webrick/httpresponse.rb (send_body_io): use Content-Range
* lib/webrick/httpservlet/filehandler.rb (make_partial_content):
use File object for the single range case
* test/webrick/test_filehandler.rb (get_res_body): use send_body
to test result
test/webrick/test_filehandler.rb: stricter multipart range test
We need to ensure we generate compatibile output in
the face of future changes
* test/webrick/test_filehandler.rb (test_make_partial_content):
check response body
webrick: quiet warning for multi-part ranges
Content-Length is ignored by WEBrick::HTTPResponse even if we
calculate it, so instead we chunk responses to HTTP/1.1 clients
and terminate HTTP/1.0 connections.
* lib/webrick/httpservlet/filehandler.rb (make_partial_content):
quiet warning
webrick/httpresponse: make ChunkedWrapper copy_stream-compatible
The .write method needs to return the number of bytes written
to avoid confusing IO.copy_stream.
* lib/webrick/httpresponse.rb (ChunkedWrapper#write): return bytes written
(ChunkedWrapper#<<): return self
webrick: use IO.copy_stream for multipart response
Use the new Proc response body feature to generate a multipart
range response dynamically. We use a flat array to minimize
object overhead as much as possible; as many ranges may fit
into an HTTP request header.
* lib/webrick/httpservlet/filehandler.rb (multipart_body): new method
(make_partial_content): use multipart_body
get rid of test error/failure on Windows introduced at r62955
* lib/webrick/httpresponse.rb (send_body_io): use seek if NotImplementedError
is raised in IO.copy_stream with offset.
* lib/webrick/httpservlet/filehandler.rb (multipart_body): ditto.
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/branches/ruby_2_4@63012 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
webrick/httprequest: limit request headers size
We use the same 112 KB limit started (AFAIK) by Mongrel, Thin,
and Puma to prevent malicious users from using up all the memory
with a single request. This also limits the damage done by
excessive ranges in multipart Range: requests.
Due to the way we rely on IO#gets and the desire to keep
the code simple, the actual maximum header may be 4093 bytes
larger than 112 KB, but we're splitting hairs at that point.
* lib/webrick/httprequest.rb: define MAX_HEADER_LENGTH
(read_header): raise when headers exceed max length
webrick/httpservlet/cgihandler: reduce memory use
WEBrick::HTTPRequest#body can be passed a block to process the
body in chunks. Use this feature to avoid building a giant
string in memory.
* lib/webrick/httpservlet/cgihandler.rb (do_GET):
avoid reading entire request body into memory
(do_POST is aliased to do_GET, so it handles bodies)
webrick/httprequest: raise correct exception
"BadRequest" alone does not resolve correctly, it is in the
HTTPStatus namespace.
* lib/webrick/httprequest.rb (read_chunked): use correct exception
* test/webrick/test_httpserver.rb (test_eof_in_chunk): new test
webrick/httprequest: use InputBufferSize for chunked requests
While WEBrick::HTTPRequest#body provides a Proc interface
for streaming large request bodies, clients must not force
the server to use an excessively large chunk size.
* lib/webrick/httprequest.rb (read_chunk_size): limit each
read and block.call to :InputBufferSize in config.
* test/webrick/test_httpserver.rb (test_big_chunks): new test
webrick: add test for Digest auth-int
No changes to the actual code, this is a new test for
a feature for which no tests existed. I don't understand
the Digest authentication code well at all, but this is
necessary for the subsequent change.
* test/webrick/test_httpauth.rb (test_digest_auth_int): new test
(credentials_for_request): support bodies with POST
webrick/httpauth/digestauth: stream req.body
WARNING! WARNING! WARNING! LIKELY BROKEN CHANGE
Pass a proc to WEBrick::HTTPRequest#body to avoid reading a
potentially large request body into memory during
authentication.
WARNING! this will break apps completely which want to do
something with the body besides calculating the MD5 digest
of it.
Also, keep in mind that probably nobody uses "auth-int".
Servers such as Apache, lighttpd, nginx don't seem to
support it; nor does curl when using POST/PUT bodies;
and we didn't have tests for it until now...
* lib/webrick/httpauth/digestauth.rb (_authenticate): stream req.body
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/branches/ruby_2_4@63004 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
webrick: prevent response splitting and header injection
Original patch by tenderlove (with minor style adjustments).
* lib/webrick/httpresponse.rb (send_header): call check_header
(check_header): raise on embedded CRLF in header value
* test/webrick/test_httpresponse.rb
(test_prevent_response_splitting_headers): new test
* (test_prevent_response_splitting_cookie_headers): ditto
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/branches/ruby_2_4@63002 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
webrick: favor .write over << method
This will make the next change to use IO.copy_stream
easier-to-read. When we can drop Ruby 2.4 support in a few
years, this will allow us to use writev(2) with multiple
arguments for headers and chunked responses.
* lib/webrick/cgi.rb (write): new wrapper method
lib/webrick/httpresponse.rb: (send_header): use socket.write
(send_body_io): ditto
(send_body_string): ditto
(send_body_proc): ditto
(_write_data): ditto
(ChunkedWrapper#write): ditto
(_send_file): ditto
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/branches/ruby_2_4@62988 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
resolv.rb: close socket
* lib/resolv.rb (UnconnectedUDP#lazy_initialize): store new
sockets before binding, so the sockets get closed when the
requester is closing.
* lib/resolv.rb (ConnectedUDP#lazy_initialize): ditto.
* lib/resolv.rb (UnconnectedUDP#close): synchronize to get rid of
race condition.
* lib/resolv.rb (ConnectedUDP#close): ditto.
[ruby-core:85901] [Bug #14571]
From: quixoten (Devin Christensen) <quixoten@gmail.com>
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/branches/ruby_2_4@62915 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
net/ftp: fix FrozenError in BufferedSocket
I noticed this bug while working on something else with
RUBYOPT=-d on, existing test cases all passed with it.
Note: I use String.new because it is the local style, here,
I prefer +'' (or ''.b, for a future commit)
* lib/net/ftp.rb (BufferedSocket#read): use String.new
* test/net/ftp/test_buffered_socket.rb (test_read_nil): new test
[ruby-core:84675] [Bug #14323]
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/branches/ruby_2_4@62867 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
Refactored using Method#owner.
proc.c: super_method of included method
* proc.c (method_super_method): search the next super method along
the included ancestor chain. [ruby-core:83114] [Bug #13973]
* vm_method.c (rb_callable_method_entry_without_refinements):
return the defined class.
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/branches/ruby_2_4@62865 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
raise error if value contains CR/LF in iniheader of initialize_http_header
like r59693, initialize_http_header also should raise error. [Bug #14208]
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/branches/ruby_2_4@62848 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
Merge RubyGems 2.7.6 from upstream.
It fixed some security vulnerabilities.
http://blog.rubygems.org/2018/02/15/2.7.6-released.html
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/branches/ruby_2_4@62434 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
irb.rb: preserve ARGV on binding.irb
This is not perfectly good solution (at least we don't want to have ARGV
as default value of `argv` argument), but unfortunately IRB.setup and
IRB.parse_opts are public methods and we can't make breaking change to
those methods.
We may deprecate using them and then make them private in the future,
but the removal should not be in Ruby 2.5. So I kept their interface for
now.
[Bug #14162] [close GH-1770]
* properties.
irb/init.rb: make sure ARGV refers to toplevel one
irb/test_init.rb: add test to ensure $0
is not changed.
At first `ARGV.unshift('something')` was suggested for r61149,
but it wasn't sufficient because it modifies $0.
Not only to preserve ARGV, but also r61149 intends to preserve $0.
This test prevents future breakage of the behavior.
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/branches/ruby_2_4@62185 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
lib/net/imap.rb: Ignore trailing space for Microsoft Exchange Server
Based on the patch by keysen (Jeremy Carlier).
[ruby-core:81641] [Bug #13649]
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/branches/ruby_2_4@61369 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
lib/ostruct.rb: [DOC] revise docs for OpenStruct
* update paragraph on implementation:
define_singleton_method is used, not define_method
* add call-seq with return values for each_pair
* adopt description of dig from Array and Hash
* fix description of the hash method
* :nodoc: initialize_copy, respond_to_missing?
* other small improvements, e.g. use the term `attribute' in the docs
(instead of `member'), which is clearer for users of the class
* improve code examples: e.g. use more consistent style (always use
double quotes, drop `p' and `puts', ...), update inspect output,
use example data that is not prone to change (like population)
* add more code examples
* fix some small errors and grammar
[ruby-core:79265] [Bug #13159]
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/branches/ruby_2_4@61364 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
Fix a command injection vulnerability in Net::FTP.
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/branches/ruby_2_4@61245 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
webrick: avoid unnecessary IO#sync= call
Sockets and pipes are always created with FMODE_SYNC flag
already set (otherwise many things would be broken).
* lib/webrick/server.rb (accept_client): remove unnecessary
IO#sync= call
webrick: do not hang acceptor on slow TLS connections
OpenSSL::SSL::SSLSocket#accept may block indefinitely on clients
which negotiate the TCP connection, but fail (or are slow) to
negotiate the subsequent TLS handshake. This prevents the
multi-threaded WEBrick server from accepting other connections.
Since the TLS handshake (via OpenSSL::SSL::SSLSocket#accept)
consists of normal read/write traffic over TCP, handle it in the
per-client thread, instead.
Furthermore, using non-blocking accept() is useful for non-TLS
sockets anyways because spurious wakeups are possible from
select(2).
* lib/webrick/server.rb (accept_client): use TCPServer#accept_nonblock
and remove OpenSSL::SSL::SSLSocket#accept call
* lib/webrick/server.rb (start_thread): call OpenSSL::SSL::SSLSocket#accept
* test/webrick/test_ssl_server.rb (test_slow_connect): new test
[ruby-core:83221] [Bug #14005]
webrick: fix up r60172
By making the socket non-blocking in r60172, TLS/SSL negotiation
via the SSL_accept function must handle non-blocking sockets
properly and retry on SSL_ERROR_WANT_READ/SSL_ERROR_WANT_WRITE.
OpenSSL::SSL::SSLSocket#accept cannot do that properly with a
non-blocking socket, so it must use non-blocking logic of
OpenSSL::SSL::SSLSocket#accept_nonblock.
Thanks to MSP-Greg (Greg L) for finding this.
* lib/webrick/server.rb (start_thread): use SSL_accept properly
with non-blocking socket.
[Bug #14013] [Bug #14005]
webrick: fix up r60172 and revert r60189
Thanks to MSP-Greg (Greg L) for helping with this.
* lib/webrick/server.rb (start_thread): ignore ECONNRESET, ECONNABORTED,
EPROTO, and EINVAL on TLS negotiation errors the same way they
were ignored before r60172 in the accept_client method of the
main acceptor thread.
[Bug #14013] [Bug #14005]
webrick: fix up r60172 and r60208
Thanks to MSP-Greg (Greg L) for helping with this.
* lib/webrick/server.rb (start_thread): fix non-local return
introduced in r60208
webrick: fix up r60172 and r60210
Thanks to MSP-Greg (Greg L) for helping with this.
* lib/webrick/server.rb (start_thread): properly fix non-local return
introduced in r60208 and r60210
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/branches/ruby_2_4@61239 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
webrick: compile RE correctly for beginning and end match
Using ^ and $ in regexps means we can accidentally get fooled
by "%0a" in HTTP request paths being decoded to newline
characters. Use \A and \z to match beginning and end-of-string
respectively, instead.
Thanks to mame and hsbt for reporting.
* lib/webrick/httpserver.rb (MountTable#compile):
use \A and \z instead of ^ and $
* lib/webrick/httpserver.rb (MountTable#normalize): use \z instead of $
* test/webrick/test_httpserver.rb (test_cntrl_in_path): new test
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/branches/ruby_2_4@61238 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
lib/webrick/log.rb: sanitize any type of logs
It had failed to sanitize some type of exception messages. Reported and
patched by Yusuke Endoh (mame) at https://hackerone.com/reports/223363
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/branches/ruby_2_4@59898 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
A HTTP Header value must not contain CR or LF.
to_str -> to_s
* lib/net/http/header.rb (set_field): `val` can not have `to_str`.
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/branches/ruby_2_4@59783 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
REXML: Fix a bug that unexpected methods can be called as a XPath function
[HackerOne:249295]
Reported by Andrea Jegher. Thanks!!!
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/branches/ruby_2_4@59782 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
escape.c: check argument
* ext/cgi/escape/escape.c (optimized_unescape): check
accept_charset encoding argument.
share `@@accept_charset`
* lib/cgi/{core,util}.rb: include CGI::Util not only extending, to
share `@@accept_charset` class variable, so that it is always
accessible. [ruby-core:80986] [Bug #13539]
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/branches/ruby_2_4@59511 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
ostruct.rb: fix OpenStruct.allocate
* lib/ostruct.rb (OpenStruct.allocate): initialize an instance
variable directly, without calling `intialize` method which may
be overridden in a subclass. [ruby-core:80292] [Bug #13358]
ostruct.rb: improve fix for OpenStruct.allocate + #respond_to?
* lib/ostruct.rb (OpenStruct#respond_to_missing?): this makes
OpenStruct#respond_to? works on any OpenStruct instance,
just like Kernel#respond_to? does, without workarounds.
[ruby-core:80292] [Bug #13358]
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/branches/ruby_2_4@59407 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
rexml: add close tag check on end of document to StreamParser
[ruby-core:81593] [Bug #13636]
Reported by Anton Sivakov. Thanks!!!
* properties.
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/branches/ruby_2_4@59399 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
rss: Accept empty text element as valid element
Parser has been accepted it but XML serializer wasn't accepted.
Reported by stefano frabetti. Thanks!!!
[ruby-core:80965] [Bug #13531]
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/branches/ruby_2_4@59396 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
optparse.rb: get rid of eval
* lib/optparse.rb: try Float() and Integer() instead of eval,
which does too much things.
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/branches/ruby_2_4@59316 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
erb.rb: Allow explicit trimming carriage return
when trim_mode is "-", for Windows environments.
[ruby-core:39625] [Bug #5339]
erb.rb: Allow trimming CR in all trim_modes
to unify a behavior with r58823 and r58825.
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/branches/ruby_2_4@59301 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
erb.rb: Allow trimming carriage return
when trim_mode is "<>", for Windows environments.
[Bug #11464]
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/branches/ruby_2_4@59300 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
fileutils.rb: do not make root
* lib/fileutils.rb (FileUtils#mkdir_p): no need to make root
directory which should be exist and cannot be made with mkdir
recent Cygwin can make a directory contains a colon.
[Bug #13214]
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/branches/ruby_2_4@58631 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
uri/generic.rb: fix exception on non-IP format
* lib/uri/generic.rb (URI::Generic#find_proxy): match IP address
no_proxy against resolved self IP address. [Fix GH-1513]
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/branches/ruby_2_4@58624 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
Fix bug of Tempfile#size if nothing is written [Bug #13198]
* lib/tempfile.rb (Tempfile#size): Fix its behavior when nothing
is written. Tempfile#size should return 0 in this case.
The patch is from nobu <nobu@ruby-lang.org>.
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/branches/ruby_2_4@58013 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
Raise error if spec is nil
With parallel test-all, the spec is sometimes nil.
To debug it raise more detailed error.
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/branches/ruby_2_4@58002 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
usa